Remote ITKModule build on top of another

python
cmake
wrapping

(Thomas Janvier) #1

Hi again,

I did not found any tool for 3D object thickness measurement in ITK…
So I used a composition of ITKBinaryThinning3D and EuclideanDistance to compute the distance from medial axis to the outside as the object thickness.
the ultimate goal is to retro-propagate the medial axis thickness to adjacent voxels, but hey walk before you run

I’m trying to build an ITK composite filter with that could wrap in a single instruction such behavior

I started a module : ITKThickness3D but how to point another “remote” module in its project cmake conf ?

I added ITKBinaryThinning3D and EuclideanDistance in the itk-module.cmake, but ofc it does not compile since it is not part of ITK “core”.

Thanks


(Matt McCormick) #2

Hi there Thomas,

:cool:

Yes, in the module dependencies, (itk-module.cmake) add BinaryThinning3D (note there should be no ITK prefix because we are referencing the module name.

Build ITKBinaryThinning3D first against your ITK build tree. Then, build ITKThickness3D.

For an example, see this module’s build configuration.

HTH,
Matt


(Thomas Janvier) #3

Thanks for explanations and links, I managed to do it I think
But I still have issues with the compilation and I don’t know where they come from…

  • CircleCI / Linux : fails to test (complains about the TestDriver that can’t be found Could not find executable /ITK-build/bin/Thickness3DTestDriver) and packaging does not find the dependency …
  • TravisCI / OSX : flags errors about itkConceptChecking …
  • Appveyor / Windows : runs smoothly but no artifacts produced…

kinda lost here

also a crelated issue: can you compile such project (ITKModule with wrapping) within a conda environnement ?
to avoid hundreads of commits on github and appveyor/circle/travis overload/latency…


(Matt McCormick) #4

The errors / warnings on the CI are pointing to issues that we can resolve. I created a few issues on the repository with some pointers and suggestions. E.g. the concept check errors are likely related to wrapping for 2D images algorithms that are only defined for 3D images.

A conda environment unfortunately does not help us here due to details related to the build system, CI resources, and binary compatibility.

That said, a migration to Azure Pipelines should help greatly. I will look into this over the next month and include you for review.


(Thomas Janvier) #5

Thanks a lot for your help,
contributing to ITK is delightful thanks to your support

Just had a look at azure pipelines, that seems like a real nice thing to use !


(Thomas Janvier) #6

Hi again,

I finally manage to get rid of most errors and pass tests&build !
Travis CI manage to build one wheel for python 3.6, how to provide 2.7, 3.5 and 3.7 wheels ?

But CircleCI provides me an include error at wrapping step:

[9/16] Generating ../../../../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml
FAILED: ../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml 
cd /work/_skbuild/linux-x86_64-2.7/cmake-build/Wrapping/Modules/Thickness3D && /work/_skbuild/linux-x86_64-2.7/cmake-build/Wrapping/Generators/CastXML/castxml/bin/castxml -o /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml --castxml-gccxml --target=x86_64-linux-gnu --castxml-start _wrapping_ --castxml-cc-gnu "(" /opt/rh/devtoolset-2/root/usr/bin/g++ -Wno-array-bounds -Wno-array-bounds -std=c++11 ")" -w -c @/work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/Thickness3D.castxml.inc /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.cxx
In file included from /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.cxx:16:
/work/include/itkMedialThicknessImageFilter3D.h:22:10: fatal error: 'itkBinaryThinningImageFilter3D.h' file not found
#include "itkBinaryThinningImageFilter3D.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

Appveyor runs smoothly but does not produce artifacts … :confused:


(Matt McCormick) #7

Well done! :clap: :taco: :sunny:

In the TravisCI configuration, change:

to:

/Users/Kitware/Dashboards/ITK/ITKPythonPackage/scripts/macpython-build-module-wheels.sh 3.6

Note that some projects restricted the Travis CI buildsd ot Python 3.6 due to timeout issues. This will not be an issue after moving to Azure Pipelines.

I think changing

to:

./ITKPythonPackage/scripts/dockcross-manylinux-build-module-wheels.sh

should do the trick (the previous command overwrite the existing build with ITKBinaryThinning3D).

There is the output:

   package init file '_skbuild\win-amd64-3.7\cmake-install\itk\__init__.py' not found (or not a regular file)

which may be an issue with scikit-build or the ITK builds – the ITK 5.0 RC 1 is coming out soon. Let’s try again building against it.

CC: @jcfr


(Thomas Janvier) #8

Ok,

I’ll try to build against different python versions with Travis
Thanks for CircleCI … i didn’t know where it could comes from…
For AppVeyor, I’ll wait to build against ITKv5.

Meanwhile I’ll try to build the wheels against ITKv4
hope I wont get the same issue than with ITKBinaryThinning3D…


(Thomas Janvier) #9

still no luck, ikeep ending with the same error

[8/16] Generating ../../../../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml[9/16] Generating ../../../../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml
FAILED: ../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml 
cd /work/_skbuild/linux-x86_64-2.7/cmake-build/Wrapping/Modules/Thickness3D && /work/_skbuild/linux-x86_64-2.7/cmake-build/Wrapping/Generators/CastXML/castxml/bin/castxml -o /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml --castxml-gccxml --target=x86_64-linux-gnu --castxml-start _wrapping_ --castxml-cc-gnu "(" /opt/rh/devtoolset-2/root/usr/bin/g++ -Wno-array-bounds -Wno-array-bounds -std=c++11 ")" -w -c @/work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/Thickness3D.castxml.inc /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.cxx
In file included from /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.cxx:16:
/work/include/itkMedialThicknessImageFilter3D.h:22:10: fatal error: 'itkBinaryThinningImageFilter3D.h' file not found
#include "itkBinaryThinningImageFilter3D.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

(Thomas Janvier) #10

Ok so I’ve been trying several things but I’m kinda stuck with this :

Travis is fine :pray::champagne:

- /Users/Kitware/Dashboards/ITK/ITKPythonPackage/scripts/macpython-build-module-wheels.sh 2.7 3.5 3.6 3.7

Everything is OK, build runs smoothly and wheels are produced

Appveyor is ok, but…

The error about skbuild was not the cause (it also appears in the build log of the dependency that still produces wheels).

package init file '_skbuild\win-amd64-3.7\cmake-install\itk\__init__.py' not found (or not a regular file)

The error came from a wrong path, it has been fixed

But how to provide python 2 wheel ?

CircleCI does not find dependencies’ sources :persevere:

I copied the config from the ITKUltrasound module, but still CircleCI does not found the dependency headers…

  • Build Module Dependencies: OK
  • Build Python packages: Fails
  • Changed the config.yml to package using ./ITKPythonPackage/scripts/dockcross-manylinux-build-module-wheels.sh didn’t change anything…
/work/include/itkMedialThicknessImageFilter3D.h:22:10: fatal error: 'itkBinaryThinningImageFilter3D.h' file not found

(Matt McCormick) #11

Well done! :sun_with_face: :taco: :champagne:

Nice catch. :eagle: :eye:

Since ITK 5 requires C++11, we cannot provide a Python 2 wheel on Windows because binary compatibility with the Python 2 distribution requires Visual Studio 2008.

I created a pull request to work around this issue, but I will continue to investigate. There may be interaction with the remote modules and their use in the build. Would you like to submit ITKBinaryThinning3D as a remote module in ITK? This means adding a file to ITK/Modules/Remote similar to the other files there by submitting a pull request. The community would also benefit from improved awareness and access to your module. :evergreen_tree: :drum:


(Thomas Janvier) #12

CircleCI :champagne::exploding_head::raised_hands::champagne:

Thanks a lot !

Sure I would, I’ll look into it since I never did such thing … (might disturb you again through the process :wink:).
Maybe I should work a bit on the itk::LocalThicknessImageFilter3D<TInputImage, TOutputImage> prior to remote module ?

[edit]

  • if you have a starting point / doc / example(s) for such contributions …
  • should I do it also for ITKBinaryThinning3D since it does not appear in the list (however published in the Insight Journal) ? (or merge both modules…)

(Thomas Janvier) #13

If I read correctly the documentation I should submit an IJ paper prior to the remote module. But then I might wait a bit longer to add the LocalThicknessImageFilter3D

For ITKBInaryThinning3D, It could be directed to the original since I only provided the wrapping.

Also all the remote modules listed are hosted inside github.com/InsightSoftwareConsortium. I have no problem transfering the repo to kitware, but how does this work for maintenance (e.g. build/deploy new wheels as for ITKBoneMorphometry not available for python 3.7)?


(Dženan Zukić) #14

Since there is already an IJ article for the thinning filter, you might not need to do a lot of work to adjust the current repo to the model. You could go through the steps described in the module template, examine the differences with the current repo and apply them.


(Thomas Janvier) #15

Thanks @dzenanz,

I think that both ITKBinaryThinning3D and ITKThickness3D repo match the model since they were created using coockiecutter and the ITKModuleTemplate (following this tuto).

My concern is more about the maintenance of the pull request on ITK including the remote.cmake file.

  • Should I do point to my own T4mmi repo or should I transfer it to InsightSoftwareConsortium ?
  • In the second case who will maintain PyPI up-to-date ?
  • Is it better to decompose a module in separate filters (i.e. one for thinning, one for medial thickness, one for local thickness transform) or group them (got second thoughts using someone else implementation in a module with my name on it) ?

(Dženan Zukić) #16

The module template has been updated over time. So some update of modules created some time ago might be needed.

It is better to transfer the repositories. After transferring the repo to InsightSoftwareConsortium, we will give you write permissions on it (in case you don’t retain them). And you will be the maintainer of that repo. Its address will have changed and more people will have write access, but that should be it. So you should maintain PyPI up to date.

I think it is better to group the filters - that’s what modules are for. Especially given there is some overhead for each module.


(Thomas Janvier) #17

Ok then,
I’ll merge ITKBinaryThinning3D into ITKThickness3D and check the module before transfering them :wink: