How to pass C++ itk::Image to Python

We have a C++ application-specific class that uses an ITK image under the hood. We also expose a Python interface to our class from which we want to provide access to its itk::Image member.
Best guess at how this might be done at the end. Help is needed.

In C++ we provide access to the underlying ITK Image for advanced manipulation, filtering, etc; and this works fine.

In Python we want to provide the same kind of access to the underlying ITK image. It’s nice to use in Jupyter notebooks with itkwidgets to interactively view an image, as well as for advanced manipulation, filtering, etc.

Here’s a typical (desired) notebook:

myimg = MyImage("/path/to/foo.nrrd")
myimg.doSomething().doSomethingElse()

# We want to use itkwidgets for 3d interactive visualization of result
from itkwidgets import view
myitkimg = myimg.toITKImage()
view(myitkimg)
# ...but it doesnt' work. We can't seem to get the itkImage

# reading directly into an itk image works fine
readimg = itk.imread("/path/to/foo.nrrd")
# readimg is <itk.itkImagePython.itkImageUS3; proxy of <Swig Object of type 'itkImageUS3 *' at 0x7fab00326a20> >
view(readimg) # opens a nice interactive 3d viewer

# creating an itk image by hand using the Python template
ImageType = itk.Image[itk.F,3]
image = ImageType.New()
# image is <itk.itkImagePython.itkImageF3; proxy of <Swig Object of type 'itkImageF3 *' at 0x7fd8d92f0ae0> >

We’ve tried lots of ways to get the underlying itk::Image from MyImage as described next, but so far no success. :frowning:

This is our C++ class:

class MyImage {
  using PixelType = float;
  using ImageType = itk::Image<PixelType, 3>;

public:
  MyImage(ImageType::Pointer imagePtr) : image(imagePtr) { if (!image) throw std::invalid_argument("null imagePtr"); }

  // return this as an ITK image
  operator ImageType::Pointer() { return image; }
  ImageType& getITKImage() { return *image; }

  // ... application specific interface
  
private:
  ImageType::Pointer image;
};


We use pybind11 to expose our c++ interface, but we can’t seem to provide access to our class’s itk::Image member. We’ve tried the following to no avail:

  // pybind11 interface exposure
  py::class_<MyImage>(m, "MyImage")
  .def(py::init<const std::string&>())   // constructor shown above that load a file

  // never gets called from Python (when passed an itkImagePython)
  .def(py::init<MyImage::ImageType::Pointer>())

  // Various attempts to give the Python direct access to the underlying ITK image:

  // Python error: cannot convert to itk::SmartPointer<itk::Image...
  .def("toITKImage",            &MyImage::operator MyImage::ImageType::Pointer)
  
  // Python error: can't convert return value (itk::Image<float, 3u>) to a Python type 
  .def("toITKImage",            &MyImage::getITKImage)
  
  // Ugly template c++ compile errors 
  .def("toITKImage",            [](MyImage& I) { return *(static_cast<MyImage::ImageType::Pointer>(I); }) 
  .def("toITKImage",            [](MyImage& I) { return *(I.operator MyImage::ImageType::Pointer()); })
  
  // And for bonus, get/set the underlying ITK image using an `instance.image` property (fails).
  .def_property("image", &MyImage::getITKImage, &MyImage::setITKImage)  

The most likely solution I can currently imagine is to…

Create and return a SWIG pointer, the type mentioned in Python code above, from the class’s itk::Image member, something like this:

#include <itk_swig_something_or_other.h>

// concrete version for an itk::Image<float, 3>
itkImageF3* getPythonImage(itk::Image<float, 3>& img)
{
  return itkImageF3*(img);
}

// ...or more generally:
template <typename T, typename N>
PyObject_itkImagePython* getPythonImage(itk::Image<T,N>& img)
{
  return PyObject_itkImagePython(<whatever needs to be done to get this>);
}

…then in our binding code, just call this function in a lambda:

  .def("toITKImage", [](MyImage& I) { return getPythonImage(I); })

This might need to be

ImageType& getITKImage() { return image.GetPointer(); }
  • This fails to compile, nor does returning ImageType::Pointer&, both with the this error:
    Non-const lvalue reference to type ‘shapeworks::Image::ImageType’ (aka ‘Image<float, 3>’) cannot bind to a temporary of type ‘itk::SmartPointer<itk::Image<float, 3> >::ObjectType *’ (aka ‘itk::Image<float, 3> *’)

  • Returning ImageType::Pointer compiles, but essentially the same error as before in Python:
    TypeError: Unable to convert function return value to a Python type! The signature was
    (self: shapeworks.Image) -> itk::SmartPointer<itk::Image<float, 3u> >

Again, the C++ type (in this case an itk::SmartPointer instead of an itk::Image) is not able to be used to construct its associated Python type.

@matt.mccormick and/or @phcerdan might want to pitch in.

Your pybind11 wrappings are of different type than the ITK swig wrappings, even for the same type: itkSmartPointer<itk::Image<D, type>>. You cannot mix them like that.
I would recommend you use numpy as a bridge between your image type and the ITK type.

See https://github.com/phcerdan/SGEXT/blob/886c7b29201272196bc842985b91ada50d82bf6b/wrap/itk/declare_itk_image_py.h#L321-L400 for an implementation of from_pyarray, to_pyarray.
Then you can just:

import itk
itk_img = itk.GetImageFromArray(MyImage.to_pyarray())
view(itk_img)
1 Like

Using swig, how would we pass an itk::Image back and forth between Python and C++?

This is helpful, but prevents sharing of data between Python and C++ without copying. It would be better to be able to expose the internal data member of our class as a reference so that it can be modified as needed.