I am a very very bloody ITK/SimpleITK beginner trying to understand the basic principles of ITK/SimpleITK. My aim is to register to 2 point clouds: the first one is from a stereoscopic imaging modality (disparity map converted to a set of points). The second point cloud is from a known mesh (stl) where I have already extracted the points. I have everything stored in Python in numpy arrays.
After having a look at the manual and going through the examples, some things aren’t clear to me and I am stuck for days, so I decided to write this post. I will try to collect the most important questions. Please correct my statements if needed
1.) There are ITK/SimpleITK images that offer large flexibility (non equidistant pixels, etc.). Do I have to convert my point clouds into such a data type to apply the registration? I only found the function sitk.GetArrayFromImage but it is not clear to me, how to treat a point cloud Or is there a datatype like PointSet/Mesh for such tasks? In this case, is there a function like sitk.GetArrayFromImage for Meshes?
2.) I think, I would need later (now I am playing around with very simple examples) some kind of non-rigid registration. I found ITK/SimpleITK since this is one of the few libraries that offer this. What would you recommend to choose for a Python implementation: ITK or SimpleITK or is this basically the same? Do I need to consider something special about the datatypes?
3.) After thinking about the datatypes, and how I can understand my problem as a 2D problem (most of the examples are 2D only) it came to my mind that a registration of the depth maps could also be a way to go. I have the disparity map (which is basically a gray scale image) from the stereo vision and I could create something like this from the mesh by taking the distance between surface and a predefined plane and put these distances into an image. Is this possible? Are there any recommended functions?
Ok, I think, that is all until now. Especially the first questions is the most important one because this would allow me to continue working with (Simple)ITK. I am very grateful for your answers in advance!
First the “bad” news, SimpleITK does not provide support for point set to point set registration (for point data, only the rigid/affine paired point case is currently supported).
Just to clarify, ITK does supports this, just the relevant classes are not Python-wrapped in SimpleITK. @zivy do you know if the classes are not wrapped in ITK Python either?
Anyway, if not, then you can use ITK’s sibling project, VTK and its vtkIterativeClosestPointTransform class. This class registers point set to surface (so it might be even better than simple point-cloud-to-point-cloud registration) and it is Python-wrapped.
We used it for patient registration (Intel RealSense depth map to patient skin surface obtained from CT) and it worked surprisingly well. On phantom scans it gave perfect results. Depending on the shape of the visible surface patch you want to register, ICP may get stuck into a local optimum, so you may need to start from a reasonable initial pose, but other than that the method is very simple and accurate.
You can find here a Slicer Project Week page that describes what we did and contains link to the Python scripts that were used for the real-time depth map acquisition, visualization, conversion to point cloud, point-cloud-to-model registration, and integration with surgical navigation systems using SlicerIGT.
As far as I remember the PointSetToPointSetRegistrationMethod hasn’t been wrapped in ITK Python. That’s why I pointed to it but also to PCL.
Also, not to oversell registration accuracy the combination of soft tissue and gravity will introduce larger errors (not to mention my eyebrows and hair that aren’t visible in MR):
