Images in physical space in Python

Yes, lessons from the past emphasize the need for explicit, specific definitions.

To take another pass, and borrow from the ITK Software Guide:

  • origin: N-dimensional sequence of floating-point values that specifies the location of the center of the 0-indexed pixel.
  • spacing: N-dimensional sequence of floating-point values that specifies the physical distance between pixel centers in each direction.
  • direction: NxN matrix of floating-point values that specifies the orientation of the image. Columns of the matrix are orthonormal direction cosines. For medical images, direction corresponds to an LPS (DICOM) coordinate system, i.e. the right-to-left, anterior-to-posterior, inferior-to-superior directions of the imaged person.

For a given index \vec{I} in 3D, the physical location \vec{P} is calculated as:

\begin{equation} \begin{pmatrix} P_1\\\ P_2\\\ P_3 \end{pmatrix} = \begin{pmatrix} O_1\\\ O_2\\\ O_3 \end{pmatrix} + \begin{pmatrix} D_{1,1} & D_{1,2} & D_{1,3}\\\ D_{2,1} & D_{2,2} & D_{2,3}\\\ D_{3,1} & D_{3,2} & D_{3,3} \end{pmatrix} * \begin{pmatrix} S_1 & 0 & 0 \\\ 0 & S_2 & 0 \\\ 0 & 0 & S_3 \end{pmatrix} * \begin{pmatrix} I_1\\\ I_2\\\ I_3 \end{pmatrix} \end{equation}

Where:

  • \vec{I}: pixel space index.
  • \vec{P}: resulting physical space position of the image index \vec{I}.
  • \vec{O}, origin: physical space origin of the first image index.
  • \mathcal{D}, direction: direction cosines matrix (orthonormal). It represents the orientation relationship between the image and the physical space coordinate system.
  • \vec{S}, spacing: physical spacing between pixels of the same axis.
4 Likes

Good point. We can create a reference, pure NumPy-based implementation, create tests for the attributes and methods. We can also provide those tests in a way that any implementation that wants to be spatial-image-array-like duck can test if its interface quacks like a spatial-image-array-like duck. :duck: :microphone:

Perhaps we should create an independent package for now, e.g. spatialimage. This package could only depend on numpy, so it is lightweight dependency. We can move rapidly – after it matures, it could move to a different home.

1 Like

@matt.mccormick That is looking like a good start.

There was a recent discussion of how to handle time. Similarly there are a couple different ways to order color. i.e. CXYZ, XYZTC etc. Simularly tensor images too. How can non-physical dimensions be handled? ITK uses a templated pixel type to separate a physical dimension form “pixel components”

There is also the definition and order of an “index”. We mean (i,j,k) where i is the fasted changing index, but then ndarrays are indexed via [k,j,i].

I expect others from different domains will have additional concerns and comments to improve the specification too. It may be worthwhile to go through a more formal proposal and comment process to aid in getting more eyes on the spec, and interest.

Would the LPS direction imply that 2-D it just LP with the origin being in the top right?

2 Likes

Fully agree.

In most medical imaging applications, 2D images are still 3D images (just happen to have a single slice), with their position, spacing (pixel size and slice thickness), and axis directions defined in 3D space. In these images, axes would be still LPS.

If the 2D image is not defined in 3D space (microscopy, optical camera, etc.) then I don’t think LPS coordinate system would be applicable.

3 Likes

It is reasonable to apply the conventions of scikit-image for indexing, space, channel, and time order. These are compatible with how data is natively organized in ITK and NumPy array interfaces to itk.Image. That is, [k,j,i] indexing,

We can add documentation to a repository and use pull requests and line-based comments for additional discussion.

1 Like

To reiterate, I really think it might be better to build a spec on top of xarray rather than numpy. They exactly solve the problem of “arrays with annotations”, and we can leverage their extensive efforts to provide compatibility with dask, distributed computing, etc. Matt Rocklin specifically pulled me aside at SciPy to implore me to look long and hard at xarray, which he considers to be ready for production, before developing our own metadata handling at scikit-image.

6 Likes

Hi @matt.mccormick @lassoan @blowekamp @matthew.brett and all! Did any of this end up happening?

I’ve been trying to dig up any conventions about how to put physical space info into zarr but have only found discussions but no conclusions…

Nothing too definitive I’m afraid but:

  • xarray are now working on a more minimal NamedArray object that might be suitable.
  • if you only need (a subset of) the named axes tczyx and scale and translation information, ome-zarr/ome-ngff v0.4 allows you to store this metadata with a zarr array, and there is good support for reading that metadata in many languages.
  • in the context of ome-zarr, if you need more sophisticated transforms, you will have to wait a couple of versions I’m afraid — see my RFC-3 and the currently stalled but imho close-to-final transformations specification PR.
1 Like

Thanks for the update and links @jnl :+1:

Here are few examples! :sun_with_face:

They,

  1. Generate correct scales and translations, aka as spacing and origin in ITK lingo, which are included in v0.4 of the OME-Zarr specification. This includes correct values for the multiple scales.
  2. Generate the multiscale representation for the first and second library
  3. Can be used in memory in Python or to convert files from ITK and non-ITK-based IO libraries, tifffile, imageio, or pyimagej, preserving or adding spatial metadata as needed.

ngff-zarr

:sparkles: Features

  • Minimal dependencies
  • Work with arbitrary Zarr store types
  • Lazy, parallel, and web ready – no local filesystem required
  • Process extremely large datasets
  • Multiple downscaling methods
  • Supports Python>=3.8
  • Implements version 0.4 of the
    OME-Zarr NGFF specification

Installation

To install the command line interface (CLI):

pip install 'ngff-zarr[cli]'

CLI example:

ngff-zarr -i ./MR-head.nrrd -o ./MR-head.ome.zarr

Bidirectional type conversion that preserves spatial metadata is available with
itk_image_to_ngff_image and ngff_image_to_itk_image.

Once represented as an NgffImage, a multiscale representation can be generated
with to_multiscales. And an OME-Zarr can be generated from the multiscales
with to_ngff_zarr.

ITK Python

An example with
ITK Python:

>>> import itk
>>> import ngff_zarr as nz
>>>
>>> itk_image = itk.imread('cthead1.png')
>>>
>>> ngff_image = nz.itk_image_to_ngff_image(itk_image)
>>>
>>> # Back again
>>> itk_image = nz.ngff_image_to_itk_image(ngff_image)

ITK-Wasm Python

An example with ITK-Wasm. ITK-Wasm’s Image is a simple
Python dataclass like NgffImage.

>>> from itkwasm_image_io import imread
>>> import ngff_zarr as nz
>>>
>>> itk_wasm_image = imread('cthead1.png')
>>>
>>> ngff_image = nz.itk_image_to_ngff_image(itk_wasm_image)
>>>
>>> # Back again
>>> itk_wasm_image = nz.ngff_image_to_itk_image(ngff_image, wasm=True)

Python Array API

NGFF-Zarr supports conversion of any NumPy array-like object that follows the
Python Array API Standard into
OME-Zarr. This includes such objects an NumPy ndarray’s, Dask Arrays, PyTorch
Tensors, CuPy arrays, Zarr array, etc.

Array to NGFF Image

Convert the array to an NgffImage, which is a standard
Python dataclass that
represents an OME-Zarr image for a single scale.

When creating the image from the array, you can specify

  • names of the dims from {‘t’, ‘z’, ‘y’, ‘x’, ‘c’}
  • the scale, the pixel spacing for the spatial dims
  • the translation, the origin or offset of the center of the first pixel
  • a name for the image
  • and axes_units with
    UDUNITS-2 identifiers
>>> # Load an image as a NumPy array
>>> from imageio.v3 import imread
>>> data = imread('cthead1.png')
>>> print(type(data))
<class 'numpy.ndarray'>

Specify optional additional metadata with to_ngff_zarr.

>>> import ngff_zarr as nz
>>> image = nz.to_ngff_image(data,
                             dims=['y', 'x'],
                             scale={'y': 1.0, 'x': 1.0},
                             translation={'y': 0.0, 'x': 0.0})
>>> print(image)
NgffImage(
    data=dask.array<array, shape=(256, 256),
    dtype=uint8,
chunksize=(256, 256), chunktype=numpy.ndarray>,
    dims=['y', 'x'],
    scale={'y': 1.0, 'x': 1.0},
    translation={'y': 0.0, 'x': 0.0},
    name='image',
    axes_units=None,
    computed_callbacks=[]
)

The image data is nested in a lazy dask.array and chucked.

If dims, scale, or translation are not specified, NumPy-compatible
defaults are used.

Generate multiscales

OME-Zarr represents images in a chunked, multiscale data structure. Use
to_multiscales to build a task graph that will produce a chunked, multiscale
image pyramid. to_multiscales has optional scale_factors and chunks
parameters. An antialiasing method can also be prescribed.

>>> multiscales = nz.to_multiscales(image,
                                    scale_factors=[2,4],
                                    chunks=64)
>>> print(multiscales)
Multiscales(
    images=[
        NgffImage(
            data=dask.array<rechunk-merge, shape=(256, 256), dtype=uint8,chunksize=(64, 64), chunktype=numpy.ndarray>,
            dims=['y', 'x'],
            scale={'y': 1.0, 'x': 1.0},
            translation={'y': 0.0, 'x': 0.0},
            name='image',
            axes_units=None,
            computed_callbacks=[]
        ),
        NgffImage(
            data=dask.array<rechunk-merge, shape=(128, 128), dtype=uint8,
chunksize=(64, 64), chunktype=numpy.ndarray>,
            dims=['y', 'x'],
            scale={'x': 2.0, 'y': 2.0},
            translation={'x': 0.5, 'y': 0.5},
            name='image',
            axes_units=None,
            computed_callbacks=[]
        ),
        NgffImage(
            data=dask.array<rechunk-merge, shape=(64, 64), dtype=uint8,
chunksize=(64, 64), chunktype=numpy.ndarray>,
            dims=['y', 'x'],
            scale={'x': 4.0, 'y': 4.0},
            translation={'x': 1.5, 'y': 1.5},
            name='image',
            axes_units=None,
            computed_callbacks=[]
        )
    ],
    metadata=Metadata(
        axes=[
            Axis(name='y', type='space', unit=None),
            Axis(name='x', type='space', unit=None)
        ],
        datasets=[
            Dataset(
                path='scale0/image',
                coordinateTransformations=[
                    Scale(scale=[1.0, 1.0], type='scale'),
                    Translation(
                        translation=[0.0, 0.0],
                        type='translation'
                    )
                ]
            ),
            Dataset(
                path='scale1/image',
                coordinateTransformations=[
                    Scale(scale=[2.0, 2.0], type='scale'),
                    Translation(
                        translation=[0.5, 0.5],
                        type='translation'
                    )
                ]
            ),
            Dataset(
                path='scale2/image',
                coordinateTransformations=[
                    Scale(scale=[4.0, 4.0], type='scale'),
                    Translation(
                        translation=[1.5, 1.5],
                        type='translation'
                    )
                ]
            )
        ],
        coordinateTransformations=None,
        name='image',
        version='0.4'
    ),
    scale_factors=[2, 4],
    method=<Methods.ITKWASM_GAUSSIAN: 'itkwasm_gaussian'>,
    chunks={'y': 64, 'x': 64}
)

The Multiscales dataclass stores all the images and their metadata for each scale according the OME-Zarr data model. Note that the correct scale and translation for each scale are automatically computed.

Write to Zarr

To write the multiscales to Zarr, use to_ngff_zarr.

nz.to_ngff_zarr('cthead1.ome.zarr', multiscales)

Use the .ome.zarr extension for local directory stores by convention.

Any other Zarr store type can also be used.

The multiscales will be computed and written out-of-core, limiting memory usage.


multiscale-spatial-image

enerate a multiscale, chunked, multi-dimensional spatial image data structure
that can serialized to OME-NGFF.

Each scale is a scientific Python Xarray spatial-image Dataset, organized
into nodes of an Xarray Datatree.

Installation

pip install multiscale_spatial_image

Usage

import numpy as np
from spatial_image import to_spatial_image
from multiscale_spatial_image import to_multiscale
import zarr

# Image pixels
array = np.random.randint(0, 256, size=(128,128), dtype=np.uint8)

image = to_spatial_image(array)
print(image)

An Xarray spatial-image DataArray. Spatial metadata can also be passed
during construction.

<xarray.SpatialImage 'image' (y: 128, x: 128)>
array([[114,  47, 215, ..., 245,  14, 175],
       [ 94, 186, 112, ...,  42,  96,  30],
       [133, 170, 193, ..., 176,  47,   8],
       ...,
       [202, 218, 237, ...,  19, 108, 135],
       [ 99,  94, 207, ..., 233,  83, 112],
       [157, 110, 186, ..., 142, 153,  42]], dtype=uint8)
Coordinates:
  * y        (y) float64 0.0 1.0 2.0 3.0 4.0 ... 123.0 124.0 125.0 126.0 127.0
  * x        (x) float64 0.0 1.0 2.0 3.0 4.0 ... 123.0 124.0 125.0 126.0 127.0
# Create multiscale pyramid, downscaling by a factor of 2, then 4
multiscale = to_multiscale(image, [2, 4])
print(multiscale)

A chunked Dask Array MultiscaleSpatialImage Xarray Datatree.

DataTree('multiscales', parent=None)
├── DataTree('scale0')
│   Dimensions:  (y: 128, x: 128)
│   Coordinates:
│     * y        (y) float64 0.0 1.0 2.0 3.0 4.0 ... 123.0 124.0 125.0 126.0 127.0
│     * x        (x) float64 0.0 1.0 2.0 3.0 4.0 ... 123.0 124.0 125.0 126.0 127.0
│   Data variables:
│       image    (y, x) uint8 dask.array<chunksize=(128, 128), meta=np.ndarray>
├── DataTree('scale1')
│   Dimensions:  (y: 64, x: 64)
│   Coordinates:
│     * y        (y) float64 0.5 2.5 4.5 6.5 8.5 ... 118.5 120.5 122.5 124.5 126.5
│     * x        (x) float64 0.5 2.5 4.5 6.5 8.5 ... 118.5 120.5 122.5 124.5 126.5
│   Data variables:
│       image    (y, x) uint8 dask.array<chunksize=(64, 64), meta=np.ndarray>
└── DataTree('scale2')
    Dimensions:  (y: 16, x: 16)
    Coordinates:
      * y        (y) float64 3.5 11.5 19.5 27.5 35.5 ... 91.5 99.5 107.5 115.5 123.5
      * x        (x) float64 3.5 11.5 19.5 27.5 35.5 ... 91.5 99.5 107.5 115.5 123.5
    Data variables:
        image    (y, x) uint8 dask.array<chunksize=(16, 16), meta=np.ndarray>

Store as an Open Microscopy Environment-Next Generation File Format (OME-NGFF)
/ netCDF Zarr store.

It is highly recommended to use dimension_separator='/' in the construction of
the Zarr stores.

store = zarr.storage.DirectoryStore('multiscale.zarr', dimension_separator='/')
multiscale.to_zarr(store)
  • A more detailed ITK-based example: Binder

ITKIOOMEZarrNGFF

This is a wrapper around a pure C++ core.

Installation

pip install itk-ioomezarrngff

Example

import itk

image = itk.imread('cthead1.png')
itk.imwrite(image, 'ctheaed1.ome.zarr')

in the context of ome-zarr, if you need more sophisticated transforms, you will have to wait a couple of versions I’m afraid — see my RFC-3 1 and the currently stalled but imho close-to-final transformations specification PR.

We will continue to hack on OME-Zarr Coordinate Transformations this weekend – please join us!

3 Likes

Thanks @matt.mccormick - these are great references!

I’ve spent some time investigating them and find the whole ecosystem quite impressive :+1:

But here’s the rub:

What about directions? I.e. the concept represented by ImageOrientationPatient in DICOM or space directions in nrrd?

Support for directions in ITK was a huge step forward back in the day, and I think it’s equally important to have it as a first class property of the zarr/ngff world if it’s going to be workable for medical imaging in addition to microscopy.

I won’t be able to join the hackathon, but look forward to catching up on this next month - have fun hacking!

1 Like

@pieper yes you are absolutely right – directions are quite important, especially for medical imaging datasets.

Those tools I referenced do have some ad-hoc support for direction information, but it is not yet in the standard and not supported by the broader ecosystem, which is quite large at this point. And the hackathon aims to make progress in this area. Bummer that you will not be able to make the hackathon, but the community effort will certainly continue past the event, and I look forward to your participation! :raised_hands:

2 Likes

It is perfectly fine if most of the tools only know about voxel space. It is even understandable if some processing algorithms only need spacing information (e.g., to do some non-linear spatial filtering).

However, adding a physical image translation without specifying physical axis direction information just does not make any sense. Wherever support for translation is added, at the same place it has to be easy to use the direction information as well.

Adding support for origin and spacing without directions would make the situation much worse than not having any support for origin and spacing at all, because it makes it much harder to add direction support later, when there is already a half-baked solution for physical mapping and we need to worry about backward compatibility.

Is it too late to step in and either prevent adding support for scales and translation, or (preferably) add support for directions as well? I might be able to join the hackathon if that helps. When/where is it?

Translation and Scales are already in the specification:
https://ngff.openmicroscopy.org/latest/index.html#trafo-md

Support for pixels, then spacing, then an origin, then direction is a common path to the functionality that is needed in many cases. While we enjoy the support ITK has had for decades for all of these things, we should recognize that ITK did not immediately have direction support either (it was only in 2005 that support was added :slight_smile: ). We should recognize that:

  • Support across open source tools, algorithms and standards is not going to be instantaneous, it will be a process.
  • Support for origin and spacing without direction is better than no spatial metadata support.

I strongly disagree. Support for origin and spacing does not make it harder to add direction support. And the addition of direction support, when there was not direction support, is not a backwards compatibility issue. If direction information is needed, then it should have been pre-applied before inputting to a system that does not have direction support.

@lassoan your participation is welcome. The details are here.

1 Like

I would prefer not to have incomplete/incorrect physical mapping implementation, because it would make it much harder for me to later find all the places where fixes are needed than implementing mapping everywhere with having image directions in mind. But maybe this is just a personal preference.

Anyway, hopefully implementations are still not that far ahead so that we can make corrections. We don’t need to add direction support at every levels (xarray, dask, …), but if we want NGFF to be directly usable for clinical imaging (without adapter libraries) then it must be able to specify linear mapping between image and physical space.

I’ll try to join the hackathon.

1 Like