Build Python Module Wheel for linux/arm64

mdevl · October 25, 2022, 12:30pm

While itk itself publishes wheels for pretty much all relevant architectures, I am facing the issue that I need to build a linux/arm64 python wheel for a specific module for which a linux/arm64 wheel is not available.

The official build scripts supplied by ITKPythonPackage are aimed at linux/x86 architecture only, and as far as I could find building a python wheel for an ITK module against the officially published python packages is not well documented. I guess it is too much of an edge use case.

In consequence, manually building the wheel becomes rather non-trivial for somebody not familiar in-depth with ITK and its build process.

I realise that this is a very general request, but any pointer in the right direction regarding this issue would be greatly appreciated, and would likely save me a (very) great deal of trial and error.

Thanks!

dzenanz · October 25, 2022, 2:26pm

@Tom_Birdsong, I think you are the most up-to-date on this. There is also an associated GitHub issue.

Tom_Birdsong · October 25, 2022, 2:37pm

@jcfr and @thewtex would be the best ones to weigh in here regarding target platforms.

I believe this is related to tarball specializations in our CI:

github.com

InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction/blob/main/.github/workflows/build-test-package-python.yml#L53


      
          - name: 'Build 🐍 Python 📦 package'
            shell: bash
            run: |
              export ITK_PACKAGE_VERSION=${{ inputs.itk-wheel-tag }}
              if [ -z ${{ inputs.cmake-options }}]; then
                CMAKE_OPTIONS=""
              else
                CMAKE_OPTIONS="--cmake_options ${{ inputs.cmake-options }}"
              fi
                
              for tarball in "-manylinux_2_28" "-manylinux2014"; do
                rm -rf ITKPythonPackage
                export TARBALL_SPECIALIZATION=${tarball}
                ./dockcross-manylinux-download-cache-and-build-module-wheels.sh cp${{ matrix.python-version }} $CMAKE_OPTIONS
              done
          
          
- name: Publish Python package as GitHub Artifact
            uses: actions/upload-artifact@v1
            with:
              name: LinuxWheel${{ matrix.python-version }}
              path: dist

Different ITK build specializations are packaged here to use in building module wheels:

mdevl · November 7, 2022, 12:13pm

I wanted to briefly check back in with you, whether anybody would be willing to help out by providing some additional information/documentation on this, or ideally even an extension of the existing build-scripts. As stated before, I realize that this is probably more of a niche problem, and as such I fully understand if its not considered worth it to sink any time or effort into this.

I hope that I do not seem impatient, but it would just be really great to know in which direction this is moving, so that I can plan accordingly. Thanks!

matt.mccormick · November 24, 2022, 4:45am

Hi @mdevl ,

Scripts to support building linux/arm64 remote module wheels will be added with 5.3.0 per

simon.rit · November 25, 2022, 2:41pm

That’s great! So my understanding is that we can now build ARM64 Python packages for both Linux and MacOS but can one does that with Github actions? A colleague of mine build MacOS Python packages for another software using cirrus-ci.org, (see Wheel for M1 by tbaudier · Pull Request #85 · OpenGATE/opengate · GitHub), is this a good option in your opinion?

mdevl · November 28, 2022, 2:46pm

Hi matt, that is great news. Looking very much forward to the release of 5.3.0.

Big thanks to you and everybody else involved for investing the time and effort to officially support building arm64 module wheels.

I already tried compiling, but I think at the current time it still fails due to the submodule in question being at 5.2. So hopefully that will resolve itself after a version bump.

Just as an fyi: while ITKPythonPackage is now officially on 5.3.0, the dockcross-manylinux-download-cache-and-build-module-wheels.py is still hardcoded to ITK_PACKAGE_VERSION:=v5.2.0.post1. Just wanted to mention it, in case it is not intentional (I changed it manually when I tried to build the wheel).

matt.mccormick · December 20, 2022, 7:01pm

@mdevl thanks for the testing and the feedback!

We have the v5.3.0 released. ITK_PACKAGE_VERSION has been updated accordingly.

ARM support for building remote module Python is in the works here:

github.com/InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction

ENH: Introduce manylinux option with ARM

InsightSoftwareConsortium:main ← InsightSoftwareConsortium:manylinux-options

opened 03:47PM - 01 Dec 22 UTC

tbirdso

+27 -3

Exposes input parameter to allow the user to build for multiple Linux toolsets …and target architectures via available manylinux docker images. Depends on ARM support introduced in ITKPythonPackage: https://github.com/InsightSoftwareConsortium/ITKPythonPackage/pull/236

Testing appreciated.

Also, testing and feedback here:

github.com/InsightSoftwareConsortium/ITKPythonPackage

Comment by tbirdso to COMP: Consolidate Linux ARM build script

InsightSoftwareConsortium:master ← InsightSoftwareConsortium:amend-arm-builds

Ready for review as CI jobs are passing. However, it should be noted that Python… packaging jobs build but do not load and test wheels. Once ARM Linux wheel artifacts are available from the [ITKSplitComponents test run](https://github.com/tbirdso/ITKSplitComponents/actions/runs/3736220631) it would be good to verify that those can be used on an ARM platform. It may take me several days to get back to testing those, or someone else can test to verify. Steps to test: 1. Pull ITKSplitComponents Linux ARM wheel artifacts from https://github.com/tbirdso/ITKSplitComponents/actions/runs/3736220631 2. Obtain access to a Linux ARM machine, such as running an ARM instance on AWS EC2 3. Install the corresponding Python wheel 4. Install ITK with `pip install itk==v5.3.0` 5. Put the ITKSplitComponents wheel on the machine and install the local wheel in the Python environment 6. In a Python shell try loading a filter from the module: `import itk; itk.SplitComponentsImageFilter.GetTypes()` 7. (Optional) Try reproducing example functionality from the [ITKSplitComponents example notebook](https://github.com/InsightSoftwareConsortium/ITKSplitComponents/blob/master/examples/ITKSplitComponents.ipynb)

Also, help getting the warnings addressed for ARM builds in CI:

is welcome. This will keep ARM support strong into the future.