Hello! In general the cost function of registration optimization could have two part. One for similarity measurement and the other to control rigidity (registration penalty). In itk I just can change similarity metric. Is there any feature in itk to change registration penalty?
What you are referring to is a regularization term (not specific to rigid), which is not available. The similarity metrics implemented in the ITK registration framework only use the image similarity and do not include a term for transformation parameter regularization.
That being said, the standalone Demons family of algorithms does include regularization via smoothing (elastic or viscous).