Run python test by the CI

axel_garcia · July 17, 2025, 11:26am

Hello, I’m working on the CudaCommon remote module and need an efficient way to test Python packages from generated wheels in our CI pipeline. Current options have limitations:

The first approach with itk_python_add_test

Add Python tests using itk_python_add_test in the module’s CMakeLists.txt
These tests run via CTest with a specific label: ctest -L Python.
Key issue: This requires building ITK with Python wrapping enabled (-DITK_WRAP_PYTHON=ON), which is time-consuming.

Alternative approaches I’m considering:

Use pytest directly as done in ITKRemoteModuleBuildTestPackageAction for notebooks, but this disconnects the tests from CTest.
Add a python-test flag in CudaCommon , which simplify the configuration (I just have to configure CudaCommon with the flag enabled, see my PR), but I would like to avoid creating new flags.

Has anyone developed an efficient workflow for testing Python-wrapped modules without rebuilding ITK each time? What approach do you recommend?

matt.mccormick · July 17, 2025, 8:25pm

Hi @axel_garcia ,

When we build the itk python packages, we finish by:

Testing installation of the built wheels
Running a few tests with the contents of those wheels.

For example, on Linux:

github.com/InsightSoftwareConsortium/ITKPythonPackage

scripts/internal/manylinux-build-wheels.sh

4d07821a3


      
          # Install packages and test
          for PYBIN in "${PYBINARIES[@]}"; do
              ${PYBIN}/pip install --user numpy
              ${PYBIN}/pip install --upgrade pip
              ${PYBIN}/pip install itk --user --no-cache-dir --no-index -f /work/dist
              (cd $HOME && ${PYBIN}/python -c 'from itk import ITKCommon;')
              (cd $HOME && ${PYBIN}/python -c 'import itk; image = itk.Image[itk.UC, 2].New()')
              (cd $HOME && ${PYBIN}/python -c 'import itkConfig; itkConfig.LazyLoading = False; import itk;')
              (cd $HOME && ${PYBIN}/python ${script_dir}/../../docs/code/test.py )
          done

The ITKPythonPackage remote module build scripts could be extended to:

Install the built packages
Install pytest
Run pytest and have it discover any tests following pytest’s naming conventions.

simon.rit · July 18, 2025, 1:32pm

Thanks for your quick answer. If I understand correctly, ITK uses Python tests declared with itk_python_add_test, run by the CI e.g. in AzurePipelinesLinuxPython.yml but you would not recommend using this for testing the Python code of remote modules but to prefer instead using pytest?

matt.mccormick · July 18, 2025, 2:02pm

We have two types of functionality to test:

Are the wrappers we generate during development working correctly?
Are the packages we generate working correctly from an installation?

For the former, yes, itk_python_add_test makes CI and development possible. For the latter, testing after installation is desirable. It should be possible to use the same testing for both itk_python_add_test and pytest by naming the test files test_*.