Extending ITK for C++ and Python: Build and Test ITK External Modules in a GitHub Action

Engineering
, , , ,
1
By: Tom Birdsong, Matt McCormick ORCID, Lucas Gandel, Simon Rit ORCID, Jean-Christophe Fillion-Robin, Stephen Aylward ORCID

Medical imaging is an ever-changing domain, with technological advances introducing exciting new data and processing techniques every year. In a rapidly evolving landscape, the open-source Insight Toolkit (ITK) continues to provide essential and cutting-edge tools for medical image processing to aid researchers and practitioners alike. ITK keeps up with rapid advances in the field through the contributions of its community members and their commitment to open science. To support these contributions, ITK provides a robust framework for the development and inclusion of external modules. Recent updates that leverage GitHub Actions workflows have made external module maintenance easier than ever. Any ITK developer can now quickly create and disseminate external modules that enhance ITK’s capabilities on multiple Mac, Linux, and Windows platforms, in C++ and for python versions 3.7-3.11 via pip. Furthermore, if an external module is deemed to add sufficient value to the ITK community, then it can be distributed as a “remote module” of ITK, allowing it to be easily enabled and built as part of the ITK C++ library and Python package.

An individual might create a new external module for any of several reasons, such as:

  • To add domain-specific tools, such as ultrasound despeckle techniques provided in the ITKUltrasound external module or tomographic reconstruction in the reconstruction toolkit (RTK);
  • To wrap an existing library for medical imaging so that it can be used in the ITK ecosystem, such as wrapping the Elastix registration library in the ITKElastix external module;
  • To add support for an emerging data formatting standard, such as support for reading from and writing to the OME-Zarr spatial array chunking standard in the ITKIOOMEZarrNGFF external module.

A notable challenge in creating an external module is the learning curve required for new contributors. In order to fully leverage the building blocks that ITK provides, it is generally necessary to have basic knowledge of C++ programming, CMake, and the ITK pipelining system, the latter of which is covered in detail in the ITK Software Guide. Fortunately, new users can make use of the ITKModuleTemplate cookiecutter discussed in the Python Packages for ITK Modules article, rather than starting from scratch. In the past, another significant barrier to entry was familiarity with good software practices, namely setting up continuous integration and continuous deployment (CI/CD) pipelines, so that modules are tested and remain compatible as ITK’s core is updated. Initial pipelines populated by the cookiecutter template had limited support for good software practices. This article introduces recent updates that take advantage of Github Action Workflows and make it easy for most ITK modules to follow CI/CD best practices and minimize maintenance requirements.

Introducing an ITK Module Reusable Workflow

The ITKRemoteModuleBuildTestPackageAction GitHub Action is a central repository to collect common procedures for building, testing, and packaging ITK open source external modules that are hosted on GitHub. Scripts leverage the CI/CD framework provided by GitHub Actions to automatically run pipelines on certain events, such as when a user opens a pull request or when a maintainer merges new changes. The repository currently offers two yaml pipeline files that lay out steps for building an external module for C++ or Python targets. These steps can be used by the majority of ITK external modules (a) to verify that the module compiles across platforms with proposed changes; (b) to run and report results from module tests; and (c) to package and deploy cross-platform Python wheels for users to easily install and run.

In most cases community members can leverage ITKRemoteModuleBuildTestPackageAction in their external module projects for several advantages:

  • Build procedures are provided as GitHub Actions (GHA) reusable workflow .yml files which integrate directly with GHA runners. These can be invoked for a given external module in its GitHub repository with minimal code.
  • Build procedure updates are centralized. Rather than replicating build step updates directly in a custom build procedure for each external module, developers may simply update workflow commit tags to include process updates in external module CI. For example, an update to a runner platform version from Ubuntu 18.04 to Ubuntu 20.04 would be introduced to ITKRemoteModuleBuildTestPackageAction once and then consumed by a variety of external modules by simply updating the ITKRemoteModuleBuildTestPackageAction commit hash accordingly.

ITK reusable workflows are best leveraged by ITK external modules that are implemented primarily in C++ and have a simple build process. If a developer wants to distribute their filters without requiring that users have prerequisites to build the module, then they can invoke the Python reusable workflow to easily package and distribute the module when new versions of the module are released. If the external module has a more complex build process with one or more external dependencies, but those dependencies can be fetched with CMake and included in the module build process, then the module will likewise be able to leverage external pipelines.

Unfortunately, at this time, external modules that require certain system dependencies in place before the build starts cannot easily leverage the reusable workflows. Pure Python external modules are also unable to be packaged with ITK reusable workflows at this time.

Usage

Developers can take advantage of ITK reusable workflows with just a few lines of code. GitHub Actions will automatically detect and run any yaml pipeline files inside a project’s .github/workflows directory. Here is an example of a basic file to run C++ and Python CI/CD pipelines from an ITK external module:

name: Build, test, package

on: [push,pull_request]

jobs:
  cxx-build-workflow:
    uses: InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction/.github/workflows/build-test-cxx.yml@d4a5ce4f219b66b78269a15392e15c95f90e7e00

  python-build-workflow:
    uses: InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction/.github/workflows/build-test-package-python.yml@d4a5ce4f219b66b78269a15392e15c95f90e7e00
    secrets:
      pypi_password: ${{ secrets.pypi_password }}

The code example above tells GitHub Actions to find and run reusable workflows in the ITKRemoteModuleBuildTestPackageAction repository. cxx-build-workflow is a job that exists to fetch and run the build-test-cxx reusable workflow, while python-build-workflow is a job that fetches and runs the build-test-package-python reusable workflow. Each workflow itself subsequently handles scheduling runners, performing environment setup, and then building the ITK external module.

The example workflow file above is broken down line by line in the ITKRemoteModuleBuildTestPackageAction README file on GitHub.

C++ Workflow

The build-test-cxx workflow provides a common process to build and test a given ITK external module. The workflow offers cross-platform testing on Windows, Ubuntu, and macOS images as provided by GitHub Actions. First, build tools and prerequisites such as CMake and Python are installed, then the Insight Toolkit is built from scratch, and finally the ITK external module is built. Any tests that are defined for CTest are run as part of the procedure and uploaded to the ITK CDash dashboard at https://open.cdash.org/index.php?project=Insight under the Experimental header.

Users can specify a number of input arguments to direct the external module build process, including:

  • The ITK C++ version to use;
  • CMake variables for the ITK build or external module build;
  • Test options and warnings to ignore in reporting results to the ITK dashboard;
  • Other ITK external modules to build first and include as build prerequisites for the current module.

More information on the C++ workflow can be found in the project README.

Python Workflow

The build-test-package-python workflow provides a common process to build, package, and deploy ITK external module Python wheels to the Python Package Index. A fixed version of ITK is used with cached Python build results in order to avoid long build times.

Several input arguments are available to direct the Python wheel build process:

  • The ITK Python build cache version to use, which roughly corresponds to ITK minor release versions;
  • CMake variables for the external module build (but not the underlying ITK build);
  • The remote reference for build scripts to use, which allows developers to build their modules with custom scripts in a development branch;
  • Linux toolsets and architectures to target, including building Python wheels for ARM platforms;
  • Other ITK external modules to build first and include as build prerequisites for the current module.

More information on the Python workflow can be found in the project README.

Examples

Reusable workflows are being actively adopted to streamline ITK external module builds. Developers can reference a number of open source projects to get started with leveraging reusable workflows in a number of ways.

  • ITKUltrasound uses ITK reusable workflows to minimize complicated build maintenance to track changes in its five prerequisite external modules;
  • ITKElastix uses reusable workflows to deploy Python wheels for ARM architectures;
  • RTK, The Reconstruction Toolkit, uses reusable workflows to build Python packages.

When the reusable workflow is not directly usable because the workflow environment must be tuned to the needs of the module, it can serve as a basis.

What’s next

ITK reusable workflows continue to expand due to user contributions. Future workflows may aim to tackle additional challenges that face the ITK community, such as packaging and testing pure Python modules that leverage ITK processing pipelines. VTK remote modules like the LookingGlassVTKModule aim to adopt similar reusable workflows. Additional planned features will address common build issues such as Windows path lengths for modules with long names, large build sizes on GitHub runners, and cross-compiled builds for ARM to improve build times. Longer term, with upgrades to our scikit-build Python package configuration, the action will further build on community tools like cibuildwheel.

Enjoy ITK!

6 Likes