C# or F# or dotnet core bindings for the "main" ITK library instead of SimpleITK?

vincenzoml · October 29, 2019, 2:40pm

Hi all. In our research on declarative image analysis (see www.voxlogica.org) we have used with quite some success SimpleITK from fsharp. I now went to check whether ITK 5 is available in SimpleITK and the answer (gotten from checking the build directory) seems “NO”.

[EDIT: the answer is “YES”, see the discussion below]

Now since I’ve seen that the python interface to ITK (the main library) has been made “more functional”, and on the other hand I see there are wrappers for Java for the main ITK library, I wonder whether it may be easier to bind ITK 5 to C# without having to pass by SimpleITK. I’ll add I’m using dotnet core; I’m seeking for pointers to understand the wrapping architecture for java, and for python.

matt.mccormick · October 29, 2019, 11:24am

Hi Vincenzo,

ITK, like SimpleITK, uses SWIG for generating wrapper interfaces. SWIG supports C# and Java wrapper generation. In ITK/Wrapping/Generators, we have infrastructure for Java – a similar configuration could be added for CSharp.

Technically, there are not any hurdles – a champion is required with the time and resources to bring support for these languages up to the level of Python.

A good overview of the wrapping infrastructure can be found in the ITK Software Guide.

Your contributions are welcome!

blowekamp · October 29, 2019, 11:39am

Hello

SimpleITK is ITKv5 ready. We nightly build and test SimpleITK compiled against ITKv5. If you build it yourself with the SimpleITK Superbuild all you have to do is add the CMake definition “ITK_GIT_TAG:STRING= v5.1b01”.

We will be releasing SimpleITK 2.0 build against ITK 5.0 in a couple months. Perhaps we should package a SimpleITK 2.0 Alpha compiled against ITK 5.1 when it’s tagged?

vincenzoml · October 29, 2019, 1:09pm

Hi @blowekamp & @matt.mccormick,

thanks for your quick reply. Here are some considerations:

it’s great to see that SimpleITK is already about to release itk5 support. I am now building with itk5.1b1 as you suggested. I believe that for the current development of VoxLogicA that may suffice, and I can certainly wait the release of SimpleITK 2.0. It would also help a lot to distribute nuget binaries for the main operating systems (currently I need to recompile the csharp wrappers when I want to update them). I think we already discussed this some months ago, I will try to understand how to do this as I myself have no idea about that.

However I have some questions about SimpleITK (before even considering to allocate the effort required for #2 below).

First of all, do you think it could be possible to take advantage of the iterators in the experimental namespace of ITK5? In particular the neighbours iterators may be very helpful.

Second, we are planning to investigate using the GPU quite soon, as some (non-ITK-based) operations that we do are expensive and really parallel. But is there support for the GPU also in ITK? And in SimpleITK? There’s no occurrence of the word GPU in the software guide.

Third, I’ve had quite some difficulties in using ConnectedThresholdImageFilter in SimpleITK because converting the array of seed points from dotnet to the SWIG type is terribly slow. In my use case I found an alternative method, but that left me wondering (but this is likely a problem also for #2 below). Providing low-level access to an uninitialised array could be a solution (similar to GetBufferAsXXX, but for arrays). BTW I’ve had to externally lock the images as GetBufferAsXXX is not thread-safe.

I am also happy to get information about the wrappers architecture in ITK. I checked the java examples a bit closer. It would be very useful in my opinion to have access to the full ITK from dotnet, however, I am unsure about some technicalities (that’s also because I never used ITK myself, since I’ve been using SimpleITK.

First, the mangling of types goes against the fact that fsharp (and probably also csharp) have generics, so it would be useful to “demangle” again these types by auto-generating a higher-level fsharp library. Or just don’t mangle them in the beginning and generate sensible fsharp generic types. But this is something that would require some help from members of the fsharp community I believe. I don’t have the manpower to do such a reasoning on my own.

Second, what about pipelining? And parallel processing? VoxLogicA in essence builds a (large,sophisticated) processing pipeline and runs it in parallel on the available cpu cores. Using the pipelining functions in ITK could be a major reason to switch from SimpleITK to ITK, however I think it may be very difficult to get these right in languages different from C++ or am I wrong? And what about inserting also user-defined filters in the pipeline? Is it possible to write such filters outside of C++/python? Also: is it possible to access the ITK thread pool “externally” so that the thread pool of the dotnet virtual machine and the one that ITK uses don’t fight for processors?

Thanks for reading so far
Vincenzo

blowekamp · October 29, 2019, 1:30pm

It would be great to get SimpleITK into nuget. Help is quite welcomed on this topic. The SimpleITK build infrastructure is quite automated now so this is a great time to start on this topic.

Iterators do not perform well in any wrapped interface as there are too many layers causing significant performance issues. The new neighborhood iterators are only going to be a direct benefit when using C++, otherwise the benefit will come through improved performance in the filters which use them. Are more are coming!

The ITK GPU classes are not currently used much and it has been a while since I used them last. The architecture allows for auto dispatching filters to a GPU implementation if enabled. So if the ITK version SimpleITK is using has GPU enabled, and the filter is implemented on the GPU, SimpleITK may use the GPU implementation. It has been quite a while since I have done this so their may be other hurdles.

You should open a SimpleITK issue on Github for this. In short, you can a global threshold then a connected components, then select you desired components to perform the equivalent operation more efficient in parallel.

Please open and SimpleITK issue on GitHub. This is likely an issue with how you are particularly using it and SimpleITK’s lazy copying, you may just need to call “MakeUnique” before the GetBuffer method.

Hope this helps.

p.s. This is a lot of questions to be asking all at once, and I only got half of them

matt.mccormick · October 29, 2019, 1:50pm

As @blowekamp mentioned, there is some OpenCL-based GPU support, but there is interest in expanding this support and supporting other types of GPU support, e.g. CUDA, ArrayFire, SYCL, PyTorch.

ITK processing pipelines work well in wrapped languages, e.g. Python.

You can create your own ITK module with C++ code that can be wrapped as described in the ITKModuleTemplate and the ITK Software Guide.

In Python, you can also create your own processing pipeline filters:

github.com

InsightSoftwareConsortium/ITK/blob/master/Wrapping/Generators/Python/PyUtils/itkPyImageFilter.h

/*=========================================================================
 *
 *  Copyright Insight Software Consortium
 *
 *  Licensed under the Apache License, Version 2.0 (the "License");
 *  you may not use this file except in compliance with the License.
 *  You may obtain a copy of the License at
 *
 *         http://www.apache.org/licenses/LICENSE-2.0.txt
 *
 *  Unless required by applicable law or agreed to in writing, software
 *  distributed under the License is distributed on an "AS IS" BASIS,
 *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *  See the License for the specific language governing permissions and
 *  limitations under the License.
 *
 *=========================================================================*/

#ifndef itkPyImageFilter_h
#define itkPyImageFilter_h

This file has been truncated. show original

github.com

InsightSoftwareConsortium/ITK/blob/e620a946a8ca19ab89883cd5160b9f0c5641da7e/Wrapping/Generators/Python/itkExtras.py#L944-L1084


class pipeline:


    """A convenient class to store the reference to the filters of a pipeline


    With this class, a method can create a pipeline of several filters and
    return it without losing the references to the filters in this pipeline.
    The pipeline object act almost like a filter (it has a GetOutput() method)
    and thus can be simply integrated in another pipeline.
    """


    def __init__(self, *args, **kargs):
        self.clear()
        self.input = None
        set_inputs(self, args, kargs)


    def connect(self, filter):
        """Connect a new filter to the pipeline


        The output of the first filter will be used as the input of this
        one and the filter passed as parameter will be added to the list

This file has been truncated. show original

This functionality could be added to other languages.

Since the native ITK libraries and the dotnet virtual machine have different runtime environments, the threads cannot be shared. So, limiting the threads on both sides may be helpful to reduce oversubscription.

vincenzoml · October 29, 2019, 2:21pm

Hi again @matt.mccormick @blowekamp,

thanks for your detailed replies. In the meantime I’ve checked, and there is no issue at all running itk5 instead of itk4 in current SimpleITK. This is great! Although for the specific filters I use (connected components, boolean operations, thresholds, boolean morphology) I did not get performance improvements at all.

I think I can safely stick to SimpleITK for the time being. There are a lot of topics to explore starting from your answers, so I will say no more right now, except on the GPU aspects, which would be a quantum leap in application of VoxLogicA, and are going to be a priority for us in the next months. A good GPU architecture for our use case would be one where there is a type for “GPU blobs” denoting an image which is residing in GPU, and the possibility to submit an arbitrary (OpenCL) kernel to an ITK function to be evaluated on a given GPU blob, and obviously converting back and forth between GPU blobs and normal images, which would indeed be expensive.

matt.mccormick · October 29, 2019, 2:24pm

This architecture is already available Here is more information: https://itk.org/Doxygen/html/group__ITKGPUCommon.html

blowekamp · October 29, 2019, 3:40pm

You may want to look into using ITK’s grayscale morphology for binary operations as they generally perform better.

vincenzoml · October 29, 2019, 2:30pm

So one can actually just not load OpenCL at all and use ITK, this looks great. Where can I find more info e.g. on how to compile ITK for GPU, how to configure drivers, etc?

dzenanz · October 29, 2019, 2:34pm

When configuring ITK, turning on ITK_USE_GPU should be enough to get you started. You will need OpenCL development headers and library from some source (e.g. this).

vincenzoml · October 29, 2019, 2:53pm

Thanks! But at this point, do filters (which ones?) use the GPU automatically? I guess for instance the connectedComponents filter doesn’t (or if it does, I’m very interested). Otherwise, how to proceed? Where is a list of filters that can operate on GPUImage? (sorry if this is obvious, not to me!)

dzenanz · October 29, 2019, 2:59pm

Directories with GPU prefix are modules containing GPU filters. Connected component isn’t one of them. As it has single-threaded sections, it is not easy to translate it into GPU formulation.

vincenzoml · November 5, 2019, 1:35pm

Hi, for connected components, I know quite well, that is also going to be part of my research plan for next year; if we get to a sensible result we may consider proposing patches to ITK. But also binary erosion is not implemented in GPU yet, or am I just wrong in interpreting that directory? [EDIT} Also what about boolean operations (and, not, or)?

dzenanz · November 5, 2019, 1:58pm

There are two additional GPU-related folders, GPUCommon and GPUFiniteDifference. But yes, only a very small subset of ITK filters have GPU variants. I don’t think binary erosion is among them.

vincenzoml · November 13, 2019, 5:51pm

@blowekamp I’m now starting to experiment with the GPU. Specifically, I want to be able to use the ITK classes for managing memory from CPU to GPU and back, and also to be able to write my own custom kernels that would then be called on instances of the said ITK class. But as mentioned earlier, I’m using simpleitk.

Presumably, looking at

https://itk.org/SimpleITKDoxygen120/html/namespaceitk.html

I think that the “itk” namespace of SimpleITK already contains the needed functionality, but I don’t have it in the .NET dll (at least it doesn’t autocomplete). I suppose I need to enable GPU in cmake, but when I run ccmake in my SimpleITK-build directory, I see no option related to GPU. Is there something I should do, or maybe GPU support is not in SimpleITK yet?