Streaming data to ITK through local disk

Beginner Questions
1

we have a data analysis library that we have been building out over the last few years. In a previous version we were able to hand ITK a wrapped pointer to our data for ITK to process. In our new version this is not possible since the data may reside on local disk (Out-of-Core). Is there an “adapter” class in ITK that we can subclass to build a bridge between ITK and our data that would rely on our own iterator (or an STL compliant iterator) to pull data from our data array?

Thank You for any help or ideas.

Mike Jackson

2

You could look at this discussion, e.g. consider FileFreeImageIO and MRMLIDImageIO to base your class on. MetaImage and HDF5 IOs have streaming support, so you could also look there for some inspiration.

3

A possible option is to use OME-Zarr via, for instance,

1 Like
4

I forgot that basic streaming is working for ZarrIO. I remembered there is an open issue about streaming.

5

Are you saying the HDF5 IO classes that are already built into ITK have Streaming support? And is there an example (I’m sure there is… )

6

Yes, the HDF5ImageIO class is based on the “StreamingImageIOBase” class and support reading arbitrary regions. I have not used it much.

What type of example are you looking for with streaming?

This most basic would be to use `ImageFileReader::SetRequestedRegion" method to only read part of the image.

Another would be to connect a ImageFileReader->ExtractImageFilter to extract an image.

Another would be some of the classes derived from the ImageSink class like LabelStatisticsImageFilter or MinimumMaximumImageFilter. These filter have a SetNumberOfStreamDivisions method to set the number of chunks to process the data in.

Hope that helps.

7

Here is an example:

With writer->SetNumberOfStreamDivisions() being the key method to invoke.

8

We have our own data handling classes that are roughly equivalent to ITKArray which have all the basic API of an “Array” class. The subclasses from that class will typically hold a “Storage” class. Currently the Storage class that we have implemented uses the typical C++ new/delete with some C++ unique pointer to ensure ownership.

This is currently how we get our own buffer over to ITK for processing:

We should maybe take a look at the StreamingImageIOBase class and see if we can derive something from that to form an “adapter” as a go between our DataArray class and an ItkImage.

1 Like