# How to pass C++ itk::Image to Python

**URL:** https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547
**Category:** Community
**Tags:** dataobject, python
**Created:** [October 15, 2020, 6:56pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547 "2020-10-15T18:56:23Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![cchriste](https://discourse.itk.org/letter_avatar_proxy/v4/letter/c/5e9695/32.png) [@cchriste](https://discourse.itk.org/u/cchriste)
#### Post date: [October 15, 2020, 6:56pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/1 "2020-10-15T18:56:23Z")

</div>

We have a C++ application-specific class that uses an ITK image under the hood. We also expose a Python interface to our class from which we want to provide access to its `itk::Image` member.  
Best guess at how this might be done at the end. Help is needed.  
…  
In C++ we provide access to the underlying ITK Image for advanced manipulation, filtering, etc; and this works fine.

In Python we want to provide the same kind of access to the underlying ITK image. It’s nice to use in Jupyter notebooks with itkwidgets to interactively view an image, as well as for advanced manipulation, filtering, etc.

Here’s a typical (desired) notebook:

```python
myimg = MyImage("/path/to/foo.nrrd")
myimg.doSomething().doSomethingElse()

# We want to use itkwidgets for 3d interactive visualization of result
from itkwidgets import view
myitkimg = myimg.toITKImage()
view(myitkimg)
# ...but it doesnt' work. We can't seem to get the itkImage

# reading directly into an itk image works fine
readimg = itk.imread("/path/to/foo.nrrd")
# readimg is <itk.itkImagePython.itkImageUS3; proxy of <Swig Object of type 'itkImageUS3 *' at 0x7fab00326a20> >
view(readimg) # opens a nice interactive 3d viewer

# creating an itk image by hand using the Python template
ImageType = itk.Image[itk.F,3]
image = ImageType.New()
# image is <itk.itkImagePython.itkImageF3; proxy of <Swig Object of type 'itkImageF3 *' at 0x7fd8d92f0ae0> >

```

### We’ve tried lots of ways to get the underlying itk::Image from MyImage as described next, but so far no success. ☹

This is our C++ class:

```auto
class MyImage {
  using PixelType = float;
  using ImageType = itk::Image<PixelType, 3>;

public:
  MyImage(ImageType::Pointer imagePtr) : image(imagePtr) { if (!image) throw std::invalid_argument("null imagePtr"); }

  // return this as an ITK image
  operator ImageType::Pointer() { return image; }
  ImageType& getITKImage() { return *image; }

  // ... application specific interface
  
private:
  ImageType::Pointer image;
};

```

…  
We use pybind11 to expose our c++ interface, but we can’t seem to provide access to our class’s itk::Image member. We’ve tried the following to no avail:

```auto
  // pybind11 interface exposure
  py::class_<MyImage>(m, "MyImage")
  .def(py::init<const std::string&>()) // constructor shown above that load a file

  // never gets called from Python (when passed an itkImagePython)
  .def(py::init<MyImage::ImageType::Pointer>())

  // Various attempts to give the Python direct access to the underlying ITK image:

  // Python error: cannot convert to itk::SmartPointer<itk::Image...
  .def("toITKImage", &MyImage::operator MyImage::ImageType::Pointer)
  
  // Python error: can't convert return value (itk::Image<float, 3u>) to a Python type 
  .def("toITKImage", &MyImage::getITKImage)
  
  // Ugly template c++ compile errors 
  .def("toITKImage", [](MyImage& I) { return *(static_cast<MyImage::ImageType::Pointer>(I); }) 
  .def("toITKImage", [](MyImage& I) { return *(I.operator MyImage::ImageType::Pointer()); })
  
  // And for bonus, get/set the underlying ITK image using an `instance.image` property (fails).
  .def_property("image", &MyImage::getITKImage, &MyImage::setITKImage)  

```

### The most likely solution I can currently imagine is to…

Create and return a SWIG pointer, the type mentioned in Python code above, from the class’s itk::Image member, something like this:

```auto
#include <itk_swig_something_or_other.h>

// concrete version for an itk::Image<float, 3>
itkImageF3* getPythonImage(itk::Image<float, 3>& img)
{
  return itkImageF3*(img);
}

// ...or more generally:
template <typename T, typename N>
PyObject_itkImagePython* getPythonImage(itk::Image<T,N>& img)
{
  return PyObject_itkImagePython(<whatever needs to be done to get this>);
}

```

…then in our binding code, just call this function in a lambda:

```auto
  .def("toITKImage", [](MyImage& I) { return getPythonImage(I); })

```

---

<div class="post-metadata">

### Author: ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)
#### Post date: [October 15, 2020, 10:32pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/2 "2020-10-15T22:32:18Z")

</div>

> [@cchriste](#):
>
> `ImageType& getITKImage() { return *image; }`

This might need to be

```cpp
ImageType& getITKImage() { return image.GetPointer(); }

```

---

<div class="post-metadata">

### Author: ![cchriste](https://discourse.itk.org/letter_avatar_proxy/v4/letter/c/5e9695/32.png) [@cchriste](https://discourse.itk.org/u/cchriste)
#### Post date: [October 16, 2020, 10:33pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/3 "2020-10-16T22:33:50Z")

</div>

> [@dzenanz](#):
>
> `ImageType& getITKImage() { return image.GetPointer(); }`

- This fails to compile, nor does returning `ImageType::Pointer&`, both with the this error:  
_Non-const lvalue reference to type ‘shapeworks::Image::ImageType’ (aka ‘Image\<float, 3\>’) cannot bind to a temporary of type ‘itk::SmartPointer\<itk::Image\<float, 3\> \>::ObjectType \*’ (aka ‘itk::Image\<float, 3\> \*’)_

- Returning `ImageType::Pointer` compiles, but essentially the same error as before in Python:  
_TypeError: Unable to convert function return value to a Python type! The signature was  
(self: shapeworks.Image) → itk::SmartPointer\<itk::Image\<float, 3u\> \>_

Again, the C++ type (in this case an itk::SmartPointer instead of an itk::Image) is not able to be used to construct its associated Python type.

---

<div class="post-metadata">

### Author: ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)
#### Post date: [October 16, 2020, 11:04pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/4 "2020-10-16T23:04:01Z")

</div>

@matt.mccormick and/or @phcerdan might want to pitch in.

---

<div class="post-metadata">

### Author: ![phcerdan](https://discourse.itk.org/user_avatar/discourse.itk.org/phcerdan/32/286_2.png) [@phcerdan](https://discourse.itk.org/u/phcerdan)
#### Post date: [October 19, 2020, 7:08pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/5 "2020-10-19T19:08:53Z")

</div>

Your pybind11 wrappings are of different type than the ITK swig wrappings, even for the same type: `itkSmartPointer<itk::Image<D, type>>`. You cannot mix them like that.  
I would recommend you use numpy as a bridge between your image type and the ITK type.

See [How to pass C++ itk::Image to Python](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/8) for an implementation of `from_pyarray`, `to_pyarray`.  
Then you can just:

```python
import itk
itk_img = itk.GetImageFromArray(MyImage.to_pyarray())
view(itk_img)

```

---

<div class="post-metadata">

### Author: ![cchriste](https://discourse.itk.org/letter_avatar_proxy/v4/letter/c/5e9695/32.png) [@cchriste](https://discourse.itk.org/u/cchriste)
#### Post date: [October 20, 2020, 12:04am UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/6 "2020-10-20T00:04:33Z")

</div>

> [@phcerdan](#):
>
> Your pybind11 wrappings are of different type than the ITK swig wrappings, even for the same type: `itkSmartPointer<itk::Image<D, type>>` . You cannot mix them like that.

Using swig, how would we pass an itk::Image back and forth between Python and C++?

> [@phcerdan](#):
>
> I would recommend you use numpy as a bridge between your image type and the ITK type.

This is helpful, but prevents sharing of data between Python and C++ without copying. It would be better to be able to expose the internal data member of our class as a reference so that it can be modified as needed.

---

<div class="post-metadata">

### Author: ![cchriste](https://discourse.itk.org/letter_avatar_proxy/v4/letter/c/5e9695/32.png) [@cchriste](https://discourse.itk.org/u/cchriste)
#### Post date: [October 28, 2020, 5:08pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/7 "2020-10-28T17:08:27Z")

</div>

Hi Pablo, I noticed the link you shared is no longer valid. Would you mind sharing the updated implementation of those functions?

Additionally, I had a conversation (in the Topology Toolkit live session after its presentation “here” at the [IEEE VIS 2020 conference](https://virtual.ieeevis.org/index.html)) about manually incrementing/decrementing reference counts for numpy arrays in order to pass them around without requiring a copy. This can be used to accomplish sharing data without copying, and with care might facilitate the image manipulation I was hoping to enable. Like numpy, it could probably utilize copying when necessary and share otherwise. Kinda dodgy, but possible. We’ll see first how much of this type of processing is desired by our users.

But we’ll just start with your handy `[to|from]_python` functions if you’d be kind enough to share an updated link. Thanks!

---

<div class="post-metadata">

### Author: ![phcerdan](https://discourse.itk.org/user_avatar/discourse.itk.org/phcerdan/32/286_2.png) [@phcerdan](https://discourse.itk.org/u/phcerdan)
#### Post date: [October 28, 2020, 10:37pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/8 "2020-10-28T22:37:20Z")

</div>

```cpp
.def(
        "to_pyarray",
        [](const TImagePointer &img,
           const std::string &contiguous) {
            const auto size =
                    img->GetLargestPossibleRegion().GetSize();
            const auto shape =
                    (contiguous == "F")
                            ? std::vector<size_t>{size[2], size[1],
                                                  size[0]}
                            : std::vector<size_t>{size[0], size[1],
                                                  size[2]};
            return py::array(
                    py::dtype::of<typename TImagePointer::
                                          ObjectType::PixelType>(),
                    shape, img->GetBufferPointer());
        },
        py::arg("contig") = "F")
// TODO: Create a view (non-copy) of the data
// Problems will arise with the contig differences between
// numpy(fortran) and c.
.def(
        "as_pyarray",
        [](const TImagePointer & /* img */,
           const std::string & /* contiguous */) {
            throw std::runtime_error(
                    "not implemented, use to_pyarray");
        },
        py::arg("contig") = "F")

.def(
        "from_pyarray",
        [](TImagePointer &img,
           py::array_t<
                   typename TImagePointer::ObjectType::PixelType>
                   np_array,
           const std::string &contiguous) {
            using PixelType =
                    typename TImagePointer::ObjectType::PixelType;
            using Image = typename TImagePointer::ObjectType;
            using ImporterType =
                    itk::ImportImageFilter<PixelType,
                                           Image::ImageDimension>;
            auto info = np_array.request();

            auto importer = ImporterType::New();
            auto region = img->GetLargestPossibleRegion();
            auto size = region.GetSize();

            if (contiguous == "F") {
                std::copy(info.shape.rbegin(), info.shape.rend(),
                          size.begin());
            } else if (contiguous == "C") {
                std::copy(info.shape.begin(), info.shape.end(),
                          size.begin());
            } else {
                throw std::runtime_error(
                        "Unknown parameter contig: " + contiguous +
                        ". Valid: F or C.");
            }
            region.SetSize(size);
            // Note that region index is kept from the staring img.
            importer->SetRegion(region);
            // Metadata is ignored (defaulted)
            // importer->SetOrigin(img->GetOrigin());
            // importer->SetSpacing(img->GetSpacing());
            // importer->SetDirection(img->GetDirection());
            // img owns the buffer, not the import filter
            const bool importImageFilterWillOwnTheBuffer = false;
            const auto data =
                    static_cast<typename TImagePointer::ObjectType::
                                        PixelType *>(info.ptr);
            const auto numberOfPixels = np_array.size();
            importer->SetImportPointer(
                    data, numberOfPixels,
                    importImageFilterWillOwnTheBuffer);
            importer->Update();
            img = importer->GetOutput();
        },
        py::arg("input"), py::arg("contig") = "F")

```

---

<div class="post-metadata">

### Author: ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)
#### Post date: [November 24, 2020, 4:07pm UTC](https://discourse.itk.org/t/how-to-pass-c-itk-image-to-python/3547/9 "2020-11-24T16:07:11Z")

</div>

> [@cchriste](#):
>
> This is helpful, but prevents sharing of data between Python and C++ without copying.

For a reference on how to share the image buffer with NumPy without copying, i.e. a NumPy array view, see:

> <https://github.com/InsightSoftwareConsortium/ITK/tree/master/Modules/Bridge/NumPy>
>
> //github.com/InsightSoftwareConsortium/ITK/tree/master/Modules/Bridge/NumPy

This is the basis of the `itk.array_view_from_image` function.