Remote ITKModule build on top of another

T4mmi · January 25, 2019, 1:49pm

Hi again,

I did not found any tool for 3D object thickness measurement in ITK…
So I used a composition of ITKBinaryThinning3D and EuclideanDistance to compute the distance from medial axis to the outside as the object thickness.
the ultimate goal is to retro-propagate the medial axis thickness to adjacent voxels, but hey walk before you run

I’m trying to build an ITK composite filter with that could wrap in a single instruction such behavior

I started a module : ITKThickness3D but how to point another “remote” module in its project cmake conf ?

I added ITKBinaryThinning3D and EuclideanDistance in the itk-module.cmake, but ofc it does not compile since it is not part of ITK “core”.

Thanks

matt.mccormick · January 25, 2019, 2:24pm

Hi there Thomas,

Yes, in the module dependencies, (itk-module.cmake) add BinaryThinning3D (note there should be no ITK prefix because we are referencing the module name.

Build ITKBinaryThinning3D first against your ITK build tree. Then, build ITKThickness3D.

For an example, see this module’s build configuration.

HTH,
Matt

T4mmi · January 28, 2019, 3:39pm

Thanks for explanations and links, I managed to do it I think …
But I still have issues with the compilation and I don’t know where they come from…

CircleCI / Linux : fails to test (complains about the TestDriver that can’t be found Could not find executable /ITK-build/bin/Thickness3DTestDriver) and packaging does not find the dependency …
TravisCI / OSX : flags errors about itkConceptChecking …
Appveyor / Windows : runs smoothly but no artifacts produced…

kinda lost here

also a crelated issue: can you compile such project (ITKModule with wrapping) within a conda environnement ?
to avoid hundreads of commits on github and appveyor/circle/travis overload/latency…

matt.mccormick · January 28, 2019, 4:08pm

The errors / warnings on the CI are pointing to issues that we can resolve. I created a few issues on the repository with some pointers and suggestions. E.g. the concept check errors are likely related to wrapping for 2D images algorithms that are only defined for 3D images.

A conda environment unfortunately does not help us here due to details related to the build system, CI resources, and binary compatibility.

That said, a migration to Azure Pipelines should help greatly. I will look into this over the next month and include you for review.

T4mmi · January 28, 2019, 4:42pm

Thanks a lot for your help,
contributing to ITK is delightful thanks to your support

Just had a look at azure pipelines, that seems like a real nice thing to use !

T4mmi · January 30, 2019, 4:00pm

Hi again,

I finally manage to get rid of most errors and pass tests&build !
Travis CI manage to build one wheel for python 3.6, how to provide 2.7, 3.5 and 3.7 wheels ?

But CircleCI provides me an include error at wrapping step:

[9/16] Generating ../../../../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml
FAILED: ../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml 
cd /work/_skbuild/linux-x86_64-2.7/cmake-build/Wrapping/Modules/Thickness3D && /work/_skbuild/linux-x86_64-2.7/cmake-build/Wrapping/Generators/CastXML/castxml/bin/castxml -o /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml --castxml-gccxml --target=x86_64-linux-gnu --castxml-start _wrapping_ --castxml-cc-gnu "(" /opt/rh/devtoolset-2/root/usr/bin/g++ -Wno-array-bounds -Wno-array-bounds -std=c++11 ")" -w -c @/work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/Thickness3D.castxml.inc /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.cxx
In file included from /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.cxx:16:
/work/include/itkMedialThicknessImageFilter3D.h:22:10: fatal error: 'itkBinaryThinningImageFilter3D.h' file not found
#include "itkBinaryThinningImageFilter3D.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

Appveyor runs smoothly but does not produce artifacts …

matt.mccormick · January 30, 2019, 4:27pm

Well done!

In the TravisCI configuration, change:

github.com

T4mmi/ITKThickness3D/blob/056b7615f73a4973f024e0ab1645583822991559/.travis.yml#L18


directories:
  - "$HOME/Library/Caches/Homebrew"
script:
- pushd ..
- git clone https://github.com/T4mmi/ITKBinaryThinning3D
- pushd ITKBinaryThinning3D
- curl -L https://raw.githubusercontent.com/InsightSoftwareConsortium/ITKPythonPackage/master/scripts/macpython-download-cache-and-build-module-wheels.sh -O
- chmod u+x macpython-download-cache-and-build-module-wheels.sh
- ./macpython-download-cache-and-build-module-wheels.sh
- popd; popd
- /Users/Kitware/Dashboards/ITK/ITKPythonPackage/scripts/macpython-build-module-wheels.sh 3.6
- tar -zcvf dist.tar.gz dist/
- curl --upload-file dist.tar.gz https://transfer.sh/dist.tar.gz

to:

/Users/Kitware/Dashboards/ITK/ITKPythonPackage/scripts/macpython-build-module-wheels.sh 3.6

Note that some projects restricted the Travis CI buildsd ot Python 3.6 due to timeout issues. This will not be an issue after moving to Azure Pipelines.

I think changing

github.com

T4mmi/ITKThickness3D/blob/056b7615f73a4973f024e0ab1645583822991559/.circleci/config.yml#L72


        git clone https://github.com/T4mmi/ITKBinaryThinning3D
        cd ITKBinaryThinning3D
        cp ~/ITKThickness3D/dockcross-manylinux-download-cache-and-build-module-wheels.sh .
        ./dockcross-manylinux-download-cache-and-build-module-wheels.sh
  - run:
      name: Build Python packages
      no_output_timeout: 1.0h
      command: |
        mv ~/ITKBinaryThinning3D/ITKPythonPackage .
        mv ~/ITKBinaryThinning3D/tools .
        ./dockcross-manylinux-download-cache-and-build-module-wheels.sh
  - store_artifacts:
      path: dist
      destination: dist


workflows:
version: 2
build-test-package:
  jobs:
    - build-and-test
    - package

to:

./ITKPythonPackage/scripts/dockcross-manylinux-build-module-wheels.sh

should do the trick (the previous command overwrite the existing build with ITKBinaryThinning3D).

There is the output:

   package init file '_skbuild\win-amd64-3.7\cmake-install\itk\__init__.py' not found (or not a regular file)

which may be an issue with scikit-build or the ITK builds – the ITK 5.0 RC 1 is coming out soon. Let’s try again building against it.

CC: @jcfr

T4mmi · January 30, 2019, 4:38pm

Ok,

I’ll try to build against different python versions with Travis
Thanks for CircleCI … i didn’t know where it could comes from…
For AppVeyor, I’ll wait to build against ITKv5.

Meanwhile I’ll try to build the wheels against ITKv4
hope I wont get the same issue than with ITKBinaryThinning3D…

T4mmi · January 31, 2019, 11:45am

still no luck, ikeep ending with the same error

[8/16] Generating ../../../../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml[9/16] Generating ../../../../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml
FAILED: ../../../ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml 
cd /work/_skbuild/linux-x86_64-2.7/cmake-build/Wrapping/Modules/Thickness3D && /work/_skbuild/linux-x86_64-2.7/cmake-build/Wrapping/Generators/CastXML/castxml/bin/castxml -o /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.xml --castxml-gccxml --target=x86_64-linux-gnu --castxml-start _wrapping_ --castxml-cc-gnu "(" /opt/rh/devtoolset-2/root/usr/bin/g++ -Wno-array-bounds -Wno-array-bounds -std=c++11 ")" -w -c @/work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/Thickness3D.castxml.inc /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.cxx
In file included from /work/ITK-cp27-cp27m-manylinux1_x64/Wrapping/itkThickness3D.cxx:16:
/work/include/itkMedialThicknessImageFilter3D.h:22:10: fatal error: 'itkBinaryThinningImageFilter3D.h' file not found
#include "itkBinaryThinningImageFilter3D.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

T4mmi · February 5, 2019, 3:37pm

Ok so I’ve been trying several things but I’m kinda stuck with this :

Travis is fine

Added multiple python versions in travis.yml:

- /Users/Kitware/Dashboards/ITK/ITKPythonPackage/scripts/macpython-build-module-wheels.sh 2.7 3.5 3.6 3.7

Everything is OK, build runs smoothly and wheels are produced

Appveyor is ok, but…

The error about skbuild was not the cause (it also appears in the build log of the dependency that still produces wheels).

package init file '_skbuild\win-amd64-3.7\cmake-install\itk\__init__.py' not found (or not a regular file)

The error came from a wrong path, it has been fixed…

But how to provide python 2 wheel ?

CircleCI does not find dependencies’ sources

I copied the config from the ITKUltrasound module, but still CircleCI does not found the dependency headers…

Build Module Dependencies: OK
Build Python packages: Fails
Changed the config.yml to package using ./ITKPythonPackage/scripts/dockcross-manylinux-build-module-wheels.sh didn’t change anything…

/work/include/itkMedialThicknessImageFilter3D.h:22:10: fatal error: 'itkBinaryThinningImageFilter3D.h' file not found

matt.mccormick · February 6, 2019, 1:04am

T4mmi:

Travis is fine

Added multiple python versions in travis.yml :
- /Users/Kitware/Dashboards/ITK/ITKPythonPackage/scripts/macpython-build-module-wheels.sh 2.7 3.5 3.6 3.7
Everything is OK, build runs smoothly and wheels are produced

Well done!

T4mmi:

The error about skbuild was not the cause (it also appears in the build log of the dependency that still produces wheels) .
package init file '_skbuild\win-amd64-3.7\cmake-install\itk\__init__.py' not found (or not a regular file)
The error came from a wrong path, it has been fixed…

Nice catch.

Since ITK 5 requires C++11, we cannot provide a Python 2 wheel on Windows because binary compatibility with the Python 2 distribution requires Visual Studio 2008.

T4mmi:

CircleCI does not find dependencies’ sources

I copied the config from the ITKUltrasound module, but still CircleCI does not found the dependency headers…

Build Module Dependencies : OK

Build Python packages : Fails

Changed the config.yml to package using ./ITKPythonPackage/scripts/dockcross-manylinux-build-module-wheels.sh didn’t change anything…
/work/include/itkMedialThicknessImageFilter3D.h:22:10: fatal error: 'itkBinaryThinningImageFilter3D.h' file not found

I created a pull request to work around this issue, but I will continue to investigate. There may be interaction with the remote modules and their use in the build. Would you like to submit ITKBinaryThinning3D as a remote module in ITK? This means adding a file to ITK/Modules/Remote similar to the other files there by submitting a pull request. The community would also benefit from improved awareness and access to your module.

T4mmi · February 6, 2019, 8:53am

Thanks a lot !

Sure I would, I’ll look into it since I never did such thing … (might disturb you again through the process ).
Maybe I should work a bit on the itk::LocalThicknessImageFilter3D<TInputImage, TOutputImage> prior to remote module ?

[edit]

if you have a starting point / doc / example(s) for such contributions …

should I do it also for ITKBinaryThinning3D since it does not appear in the list (however published in the Insight Journal) ? (or merge both modules…)

T4mmi · February 6, 2019, 10:44am

If I read correctly the documentation I should submit an IJ paper prior to the remote module. But then I might wait a bit longer to add the LocalThicknessImageFilter3D…

For ITKBInaryThinning3D, It could be directed to the original since I only provided the wrapping.

gist.github.com

https://gist.github.com/T4mmi/20449a97dce99f71eec8a9bb6e8853d4

BinaryThinning3D.remote.cmake

# Contact: Thomas Janvier <thomas.p.janvier@gmail.com>
itk_fetch_module(BinaryThinning3D

"3D thinning algorithm

Implements a new itk::BinaryThinningImageFilter3D that can find all deletable surface points at every iteration and is thus very fast.

Insight Journal article:

  Homann H.

This file has been truncated. show original

Also all the remote modules listed are hosted inside github.com/InsightSoftwareConsortium. I have no problem transfering the repo to kitware, but how does this work for maintenance (e.g. build/deploy new wheels as for ITKBoneMorphometry not available for python 3.7)?

dzenanz · February 6, 2019, 2:12pm

Since there is already an IJ article for the thinning filter, you might not need to do a lot of work to adjust the current repo to the model. You could go through the steps described in the module template, examine the differences with the current repo and apply them.

T4mmi · February 6, 2019, 2:36pm

Thanks @dzenanz,

I think that both ITKBinaryThinning3D and ITKThickness3D repo match the model since they were created using coockiecutter and the ITKModuleTemplate (following this tuto).

My concern is more about the maintenance of the pull request on ITK including the remote.cmake file.

Should I do point to my own T4mmi repo or should I transfer it to InsightSoftwareConsortium ?
In the second case who will maintain PyPI up-to-date ?
Is it better to decompose a module in separate filters (i.e. one for thinning, one for medial thickness, one for local thickness transform) or group them (got second thoughts using someone else implementation in a module with my name on it) ?

dzenanz · February 6, 2019, 2:59pm

The module template has been updated over time. So some update of modules created some time ago might be needed.

It is better to transfer the repositories. After transferring the repo to InsightSoftwareConsortium, we will give you write permissions on it (in case you don’t retain them). And you will be the maintainer of that repo. Its address will have changed and more people will have write access, but that should be it. So you should maintain PyPI up to date.

I think it is better to group the filters - that’s what modules are for. Especially given there is some overhead for each module.

T4mmi · February 6, 2019, 3:05pm

Ok then,
I’ll merge ITKBinaryThinning3D into ITKThickness3D and check the module before transfering them