How to pass C++ itk::Image to Python

We have a C++ application-specific class that uses an ITK image under the hood. We also expose a Python interface to our class from which we want to provide access to its itk::Image member.
Best guess at how this might be done at the end. Help is needed.

In C++ we provide access to the underlying ITK Image for advanced manipulation, filtering, etc; and this works fine.

In Python we want to provide the same kind of access to the underlying ITK image. It’s nice to use in Jupyter notebooks with itkwidgets to interactively view an image, as well as for advanced manipulation, filtering, etc.

Here’s a typical (desired) notebook:

myimg = MyImage("/path/to/foo.nrrd")

# We want to use itkwidgets for 3d interactive visualization of result
from itkwidgets import view
myitkimg = myimg.toITKImage()
# ...but it doesnt' work. We can't seem to get the itkImage

# reading directly into an itk image works fine
readimg = itk.imread("/path/to/foo.nrrd")
# readimg is <itk.itkImagePython.itkImageUS3; proxy of <Swig Object of type 'itkImageUS3 *' at 0x7fab00326a20> >
view(readimg) # opens a nice interactive 3d viewer

# creating an itk image by hand using the Python template
ImageType = itk.Image[itk.F,3]
image = ImageType.New()
# image is <itk.itkImagePython.itkImageF3; proxy of <Swig Object of type 'itkImageF3 *' at 0x7fd8d92f0ae0> >

We’ve tried lots of ways to get the underlying itk::Image from MyImage as described next, but so far no success. :frowning:

This is our C++ class:

class MyImage {
  using PixelType = float;
  using ImageType = itk::Image<PixelType, 3>;

  MyImage(ImageType::Pointer imagePtr) : image(imagePtr) { if (!image) throw std::invalid_argument("null imagePtr"); }

  // return this as an ITK image
  operator ImageType::Pointer() { return image; }
  ImageType& getITKImage() { return *image; }

  // ... application specific interface
  ImageType::Pointer image;

We use pybind11 to expose our c++ interface, but we can’t seem to provide access to our class’s itk::Image member. We’ve tried the following to no avail:

  // pybind11 interface exposure
  py::class_<MyImage>(m, "MyImage")
  .def(py::init<const std::string&>())   // constructor shown above that load a file

  // never gets called from Python (when passed an itkImagePython)

  // Various attempts to give the Python direct access to the underlying ITK image:

  // Python error: cannot convert to itk::SmartPointer<itk::Image...
  .def("toITKImage",            &MyImage::operator MyImage::ImageType::Pointer)
  // Python error: can't convert return value (itk::Image<float, 3u>) to a Python type 
  .def("toITKImage",            &MyImage::getITKImage)
  // Ugly template c++ compile errors 
  .def("toITKImage",            [](MyImage& I) { return *(static_cast<MyImage::ImageType::Pointer>(I); }) 
  .def("toITKImage",            [](MyImage& I) { return *(I.operator MyImage::ImageType::Pointer()); })
  // And for bonus, get/set the underlying ITK image using an `instance.image` property (fails).
  .def_property("image", &MyImage::getITKImage, &MyImage::setITKImage)  

The most likely solution I can currently imagine is to…

Create and return a SWIG pointer, the type mentioned in Python code above, from the class’s itk::Image member, something like this:

#include <itk_swig_something_or_other.h>

// concrete version for an itk::Image<float, 3>
itkImageF3* getPythonImage(itk::Image<float, 3>& img)
  return itkImageF3*(img);

// ...or more generally:
template <typename T, typename N>
PyObject_itkImagePython* getPythonImage(itk::Image<T,N>& img)
  return PyObject_itkImagePython(<whatever needs to be done to get this>);

…then in our binding code, just call this function in a lambda:

  .def("toITKImage", [](MyImage& I) { return getPythonImage(I); })

This might need to be

ImageType& getITKImage() { return image.GetPointer(); }
  • This fails to compile, nor does returning ImageType::Pointer&, both with the this error:
    Non-const lvalue reference to type ‘shapeworks::Image::ImageType’ (aka ‘Image<float, 3>’) cannot bind to a temporary of type ‘itk::SmartPointer<itk::Image<float, 3> >::ObjectType *’ (aka ‘itk::Image<float, 3> *’)

  • Returning ImageType::Pointer compiles, but essentially the same error as before in Python:
    TypeError: Unable to convert function return value to a Python type! The signature was
    (self: shapeworks.Image) -> itk::SmartPointer<itk::Image<float, 3u> >

Again, the C++ type (in this case an itk::SmartPointer instead of an itk::Image) is not able to be used to construct its associated Python type.

@matt.mccormick and/or @phcerdan might want to pitch in.

Your pybind11 wrappings are of different type than the ITK swig wrappings, even for the same type: itkSmartPointer<itk::Image<D, type>>. You cannot mix them like that.
I would recommend you use numpy as a bridge between your image type and the ITK type.

See How to pass C++ itk::Image to Python for an implementation of from_pyarray, to_pyarray.
Then you can just:

import itk
itk_img = itk.GetImageFromArray(MyImage.to_pyarray())
1 Like

Using swig, how would we pass an itk::Image back and forth between Python and C++?

This is helpful, but prevents sharing of data between Python and C++ without copying. It would be better to be able to expose the internal data member of our class as a reference so that it can be modified as needed.

Hi Pablo, I noticed the link you shared is no longer valid. Would you mind sharing the updated implementation of those functions?

Additionally, I had a conversation (in the Topology Toolkit live session after its presentation “here” at the IEEE VIS 2020 conference) about manually incrementing/decrementing reference counts for numpy arrays in order to pass them around without requiring a copy. This can be used to accomplish sharing data without copying, and with care might facilitate the image manipulation I was hoping to enable. Like numpy, it could probably utilize copying when necessary and share otherwise. Kinda dodgy, but possible. We’ll see first how much of this type of processing is desired by our users.

But we’ll just start with your handy [to|from]_python functions if you’d be kind enough to share an updated link. Thanks!

        [](const TImagePointer &img,
           const std::string &contiguous) {
            const auto size =
            const auto shape =
                    (contiguous == "F")
                            ? std::vector<size_t>{size[2], size[1],
                            : std::vector<size_t>{size[0], size[1],
            return py::array(
                    py::dtype::of<typename TImagePointer::
                    shape, img->GetBufferPointer());
        py::arg("contig") = "F")
// TODO: Create a view (non-copy) of the data
// Problems will arise with the contig differences between
// numpy(fortran) and c.
        [](const TImagePointer & /* img */,
           const std::string & /* contiguous */) {
            throw std::runtime_error(
                    "not implemented, use to_pyarray");
        py::arg("contig") = "F")

        [](TImagePointer &img,
                   typename TImagePointer::ObjectType::PixelType>
           const std::string &contiguous) {
            using PixelType =
                    typename TImagePointer::ObjectType::PixelType;
            using Image = typename TImagePointer::ObjectType;
            using ImporterType =
            auto info = np_array.request();

            auto importer = ImporterType::New();
            auto region = img->GetLargestPossibleRegion();
            auto size = region.GetSize();

            if (contiguous == "F") {
                std::copy(info.shape.rbegin(), info.shape.rend(),
            } else if (contiguous == "C") {
                std::copy(info.shape.begin(), info.shape.end(),
            } else {
                throw std::runtime_error(
                        "Unknown parameter contig: " + contiguous +
                        ". Valid: F or C.");
            // Note that region index is kept from the staring img.
            // Metadata is ignored (defaulted)
            // importer->SetOrigin(img->GetOrigin());
            // importer->SetSpacing(img->GetSpacing());
            // importer->SetDirection(img->GetDirection());
            // img owns the buffer, not the import filter
            const bool importImageFilterWillOwnTheBuffer = false;
            const auto data =
                    static_cast<typename TImagePointer::ObjectType::
                                        PixelType *>(info.ptr);
            const auto numberOfPixels = np_array.size();
                    data, numberOfPixels,
            img = importer->GetOutput();
        py::arg("input"), py::arg("contig") = "F")

1 Like

For a reference on how to share the image buffer with NumPy without copying, i.e. a NumPy array view, see:

This is the basis of the itk.array_view_from_image function.

1 Like