I am doing B-Spline registration, I use Sum of Squared Differences (SSD) as the cost and found images were deformed too much. I would like to add some regularization in the cost function to constrain image deformation. Can I use self-defined cost function for optimization in SimpleITK? Do I need to make the changes in the C++ ITK library then compile them? (e.g. write a class for customized cost in C++ …). Many thanks.
Unfortunately, you cannot add cost functions directly into SimpleITK. The process is as you described, add into ITK C++ and then wrap into SimpleITK. The best way to do this is to use the closest existing functionality. If you are not familiar with ITK development this may be a significant effort.
Possibly consider using other tools that provide the functionality you want, take a look at the ANTs or Elastix family of tools .