Compilation of ITK from source with Python wrappers in Arch-based systems
Continuing the discussion from Extracting displacement fields from rigid registration:
Hello!
I am having long days trying to correctly compile ITK in Parabola GNU/Linux (based on Arch). The compilation seems successful, but itk.TransformToDisplacementFieldFilter is unusable (and probably other portions too). What are the steps that I need to follow to compile from sources (and have itk.TransformToDisplacementFieldFilter with a 3D registration)?
Thank you in advance 
.
How I built
In summary, all remote plugins are disabled and the compilation is for my specific architecture (-march=native -mtune=native).
To improve my odds, I tried a script (of my own to install in a custom directory) and `PKGBUILD`s. The use of -G Ninja is for better memory management, and samu (samurai) is a ninja-compatible build tool.
Script
To have a more successful and quicker compilation, I am running this from a virtual terminal (no GUI) with the least number of background processes (e.g. no network!).
#!/usr/bin/env bash
export \
CFLAGS="-march=native -mtune=native" \
CXXFLAGS="-march=native -mtune=native" \
LANG=C \
LANGUAGE=C \
LC_ALL=C \
LC_COLLATE=C \
MAKEFLAGS="-j6" \
OMPI_MCA_mpi_oversubscribe=0 \
OMPI_MCA_opal_cuda_support=0 \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/opt/paraview/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl \
PETSC_DIR=/usr \
SHELL=/bin/bash
unset LC_MEASUREMENT LC_MONETARY LC_NUMERIC LC_PAPER LC_TIME LDFLAGS ALTERNATE_EDITOR EDITOR
cmake -S ./ -B ./build \
-DCMAKE_INSTALL_PREFIX:FILEPATH=pkg_test/usr \
-DCMAKE_BUILD_TYPE:STRING=Release \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
\
-DCMAKE_CXX_FLAGS:STRING="-std=c++11 -mtune=native -march=native" \
-DCMAKE_C_FLAGS:STRING="-march=native -mtune=native" \
-DITK_CXX_OPTIMIZATION_FLAGS:STRING="-O3 -DNDEBUG -mtune=native -march=native" \
-DITK_C_OPTIMIZATION_FLAGS:STRING="-O3 -DNDEBUG -mtune=native -march=native" \
-G Ninja \
\
-DBUILD_TESTING:BOOL=OFF \
-DBUILD_EXAMPLES:BOOL=OFF \
-DITK_BUILD_DOCUMENTATION:BOOL=OFF \
\
-DITK_WRAP_PYTHON:BOOL=ON \
-DITK_USE_SYSTEM_SWIG:BOOL=ON \
-DITK_USE_SYSTEM_CASTXML:BOOL=ON \
\
-DITK_LEGACY_SILENT:BOOL=ON \
-DBUILD_SHARED_LIBS:BOOL=ON \
-DITK_USE_SYSTEM_LIBRARIES:BOOL=ON \
-DITK_USE_SYSTEM_JPEG:BOOL=ON \
-DITK_USE_SYSTEM_PNG:BOOL=ON \
-DITK_USE_SYSTEM_ZLIB:BOOL=ON \
-DITK_USE_SYSTEM_TIFF:BOOL=ON \
-DITK_USE_SYSTEM_GDCM:BOOL=ON \
-DITK_USE_SYSTEM_EXPAT:BOOL=ON \
-DITK_USE_SYSTEM_FFTW:BOOL=ON \
-DITK_USE_SYSTEM_HDF5:BOOL=ON \
-DITK_USE_64BITS_IDS:BOOL=ON \
\
-DITK_WRAP_IMAGE_DIMS:STRING="2;3;4" \
-DITK_WRAP_VECTOR_COMPONENTS:STRING="2;3;4" \
-DITK_WRAP_float:BOOL=ON \
-DITK_WRAP_vector_float:BOOL=ON \
\
-DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
\
-DCMAKE_SKIP_INSTALL_RPATH=ON \
-DITK_BUILD_DEFAULT_MODULES=ON
# Persist after error
local _f=1 _try_i=0 _max_try=20
while [[ ! "${_f}" == "0" ]] && \
[[ "${_i}" -lt "${_max_try}" ]]; do
_f=0
samu -j8 -C ./build ||
_f=$? # catch error
_try_i="$(( ${_i} + 1 ))" # counter
sleep 3; # pause (to kill manually)
done
DESTDIR=pkg_test samu -j8 -C ./build install
# Based on AUR's itk-git package
# (quick fix for
# https://github.com/InsightSoftwareConsortium/ITK/issues/2960)
_pycmd='print("%s.%s" % (sys.version_info[:2]))'
_pyver=$(python -c "import sys; ${_pycmd}")
_pkg_test="$(cd ./build/pkg_test && find . -type d -name "pkg_test" -print -quit)"
[[ -d pkg_test ]] && mv pkg_test /tmp
[[ -d "./build/pkg_test/${_pkg_test}" ]] && mv "./build/pkg_test/${_pkg_test}" .
_py_dir="$(find ./pkg_test -type d -name "python${_pyver}" -print -quit)"
mv "${_py_dir}" pkg_test/usr/lib
# python -O -m compileall "pkg_test/usr/lib"
PKGBUILD
Here is the latest version. The intention was to clear the environment as much as possible (see Blk. 3 if needed).
makepkg --config ./makepkg.conf
pacman -U itk-5.2.1-15*pkg.tar.xz
#!/hint/bash
#
# /etc/makepkg.conf
#
#########################################################################
# SOURCE ACQUISITION
#########################################################################
#
#-- The download utilities that makepkg should use to acquire sources
# Format: 'protocol::agent'
DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u'
'ftp::/usr/bin/curl -gqfC - --ftp-pasv --retry 5 --retry-delay 3 -o %o %u'
'http::/usr/bin/curl -gqb "" -fLC - --retry 5 --retry-delay 3 -o %o %u'
'https::/usr/bin/curl -gqb "" -fLC - --retry 5 --retry-delay 3 -o %o %u'
'rsync::/usr/bin/rsync --no-motd -z %u %o'
'scp::/usr/bin/scp -C %u %o')
# Other common tools:
# /usr/bin/snarf
# /usr/bin/lftpget -c
# /usr/bin/wget
#-- The package required by makepkg to download VCS sources
# Format: 'protocol::package'
VCSCLIENTS=('bzr::bzr'
'fossil::fossil'
'git::git'
'hg::mercurial'
'svn::subversion')
#########################################################################
# ARCHITECTURE, COMPILE FLAGS
#########################################################################
#
CARCH="x86_64"
CHOST="x86_64-pc-linux-gnu"
#-- Compiler and Linker Flags
# -march (or -mcpu) builds exclusively for an architecture
# -mtune optimizes for an architecture, but builds for whole processor family
CFLAGS="-march=native -mtune=native"
CXXFLAGS="-march=native -mtune=native"
#RUSTFLAGS="-C opt-level=2"
#-- Make Flags: change this for DistCC/SMP systems
MAKEFLAGS="-j6"
#-- Debugging flags
DEBUG_CFLAGS="-g -fvar-tracking-assignments"
DEBUG_CXXFLAGS="-g -fvar-tracking-assignments"
#DEBUG_RUSTFLAGS="-C debuginfo=2"
#########################################################################
# BUILD ENVIRONMENT
#########################################################################
#
# Makepkg defaults: BUILDENV=(!distcc !color !ccache check !sign)
# A negated environment option will do the opposite of the comments below.
#
#-- distcc: Use the Distributed C/C++/ObjC compiler
#-- color: Colorize output messages
#-- ccache: Use ccache to cache compilation
#-- check: Run the check() function if present in the PKGBUILD
#-- sign: Generate PGP signature file
#
BUILDENV=(!distcc color !ccache check !sign)
#
#-- If using DistCC, your MAKEFLAGS will also need modification. In addition,
#-- specify a space-delimited list of hosts running in the DistCC cluster.
#DISTCC_HOSTS=""
#
#-- Specify a directory for package building.
#BUILDDIR=/tmp/makepkg
#########################################################################
# GLOBAL PACKAGE OPTIONS
# These are default values for the options=() settings
#########################################################################
#
# Makepkg defaults: OPTIONS=(!strip docs libtool staticlibs emptydirs !zipman !purge !debug !lto)
# A negated option will do the opposite of the comments below.
#
#-- strip: Strip symbols from binaries/libraries
#-- docs: Save doc directories specified by DOC_DIRS
#-- libtool: Leave libtool (.la) files in packages
#-- staticlibs: Leave static library (.a) files in packages
#-- emptydirs: Leave empty directories in packages
#-- zipman: Compress manual (man and info) pages in MAN_DIRS with gzip
#-- purge: Remove files specified by PURGE_TARGETS
#-- upx: Compress binary executable files using UPX
#-- optipng: Optimize PNG images with optipng
#-- debug: Add debugging flags as specified in DEBUG_* variables
#-- lto: Add compile flags for building with link time optimization
#
# OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge !debug !lto)
OPTIONS=(!strip docs libtool staticlibs !emptydirs zipman !purge !lto)
#-- File integrity checks to use. Valid: md5, sha1, sha224, sha256, sha384, sha512, b2
INTEGRITY_CHECK=(sha512)
#-- Options to be used when stripping binaries. See `man strip' for details.
STRIP_BINARIES="--strip-all"
#-- Options to be used when stripping shared libraries. See `man strip' for details.
STRIP_SHARED="--strip-unneeded"
#-- Options to be used when stripping static libraries. See `man strip' for details.
STRIP_STATIC="--strip-debug"
#-- Manual (man and info) directories to compress (if zipman is specified)
MAN_DIRS=({usr{,/local}{,/share},opt/*}/{man,info})
#-- Doc directories to remove (if !docs is specified)
DOC_DIRS=(usr/{,local/}{,share/}{doc,gtk-doc} opt/*/{doc,gtk-doc})
#-- Files to be removed from all packages (if purge is specified)
PURGE_TARGETS=(usr/{,share}/info/dir .packlist *.pod)
#-- Directory to store source code in for debug packages
DBGSRCDIR="/usr/src/debug"
#########################################################################
# PACKAGE OUTPUT
#########################################################################
#
# Default: put built package and cached source in build directory
#
#-- Destination: specify a fixed directory where all packages will be placed
#PKGDEST=/home/packages
#-- Source cache: specify a fixed directory where source files will be cached
#SRCDEST=/home/sources
#-- Source packages: specify a fixed directory where all src packages will be placed
#SRCPKGDEST=/home/srcpackages
#-- Log files: specify a fixed directory where all log files will be placed
#LOGDEST=/home/makepkglogs
#-- Packager: name/email of the person or organization building packages
#PACKAGER="John Doe <john@doe.com>"
#-- Specify a key to use for package signing
#GPGKEY=""
#########################################################################
# COMPRESSION DEFAULTS
#########################################################################
#
COMPRESSGZ=(gzip -c -f -n)
COMPRESSBZ2=(bzip2 -c -f)
COMPRESSXZ=(xz -c -z --threads=4 -)
COMPRESSZST=(zstd -T4 -c -z -q -)
COMPRESSLRZ=(lrzip -q)
COMPRESSLZO=(lzop -q)
COMPRESSZ=(compress -c -f)
COMPRESSLZ4=(lz4 -q)
COMPRESSLZ=(lzip -c -f)
#########################################################################
# EXTENSION DEFAULTS
#########################################################################
#
# WARNING: Do NOT modify these variables unless you know what you are
# doing.
#
PKGEXT='.pkg.tar.xz'
SRCEXT='.src.tar.gz'
#########################################################################
# OTHER
#########################################################################
#
#-- Command used to run pacman as root, instead of trying sudo and su
#PACMAN_AUTH=()
#########################################################################
# MODULAR CONFIGURATION
#########################################################################
#
for file in /etc/makepkg.d/*.conf; do
[[ -f "$file" ]] || continue
source "$file"
done
unset file
unset LC_MEASUREMENT LC_MONETARY LC_NUMERIC LC_PAPER LC_TIME LDFLAGS ALTERNATE_EDITOR EDITOR
# Maintainer: Chris <christopher.r.mullins g-mail>
# Contributor: geosam <samuelmesa@linuxmail.org>
# Contributor: Andrzej Giniewicz <gginiu@gmail.com>
# Contributor: Thomas Dziedzic < gostrc at gmail >
# Contributor: joel schaerer <joel.schaerer@laposte.net>
_pkgbase=insight-toolkit
pkgname=itk
pkgver=5.2.1
pkgrel=15.edgar.static
_pkgdesc="Cross-platform system that provides developers"
_pkgdesc="$pkgdesc with an extensive suite of software"
_pkgdesc="$pkgdesc tools for image analysis"
pkgdesc="${_pkgdesc}"
arch=('i686' 'x86_64')
url='http://www.itk.org/'
license=('APACHE')
depends=('fftw' 'libjpeg-turbo' 'libpng' 'zlib' 'libtiff'
'gdcm' 'expat' 'hdf5' 'gtest' 'eigen')
optdepends=(
'python2: build python wrapping'
'ruby'
'tcl: build tcl wrapping (currently not supported)'
'perl: build perl wrapping (currently not supported)'
'java-runtime: build java wrapping (currently not supported)'
'swig: generate python wrappers'
'pcre: for wrapping'
'castxml-git: for wrapping and docs'
'clang: for swig'
'castxml-git: for ITK')
makedepends=('cmake' 'git' 'castxml' "samurai")
source=("${_pkgbase}-${pkgver}.tar.gz::https://github.com/InsightSoftwareConsortium/ITK/releases/download/v${pkgver}/InsightToolkit-${pkgver}.tar.gz")
sha512sums=('6786e39cdf3d0c3a31abd1e23481e30f6dc9dac189ffe372dde3db688f2f57686a8beb321778327e1ff683ed844d41f1dee937b0ba542b2365e2195dfca398c7')
provides=(python-itk=${pkgver})
conflicts=(python-itk)
options=(!strip docs libtool staticlibs !emptydirs zipman !purge)
export OMPI_MCA_opal_cuda_support=0
export OMPI_MCA_mpi_oversubscribe=0
# https://github.com/InsightSoftwareConsortium/ITK/search?q=typeinfo&type=issues
# https://insightsoftwareconsortium.atlassian.net/browse/ITK-3538?focusedCommentId=27765&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-27765
# from here: https://github.com/InsightSoftwareConsortium/ITK/issues/1701#issuecomment-598311460
export LC_ALL=C LANG=C LANGUAGE=C
prepare() {
cd "${srcdir}"
_basedir="${srcdir}"/"${_pkgbase}"-"${pkgver}"
[[ -d InsightToolkit-"${pkgver}" ]] &&
[[ ! -d "${_basedir}" ]] &&
mv "${srcdir}"/InsightToolkit-"${pkgver}" "${_basedir}"
}
build() {
cd "${srcdir}"
_build_dir="${srcdir}"/"${_pkgbase}"-"${pkgver}"/build
_confopts=(
-S "${srcdir}"/"${_pkgbase}"-"${pkgver}"
-B "${_build_dir}"
-DCMAKE_INSTALL_PREFIX:FILEPATH=/usr
-DCMAKE_BUILD_TYPE:STRING=Release
-DCMAKE_VERBOSE_MAKEFILE:BOOL=OFF
-DCMAKE_CXX_FLAGS:STRING="-std=c++11 -mtune=native -march=native"
-DCMAKE_C_FLAGS:STRING="-march=native -mtune=native"
-DITK_CXX_OPTIMIZATION_FLAGS:STRING="-O3 -DNDEBUG -mtune=native -march=native"
-DITK_C_OPTIMIZATION_FLAGS:STRING="-O3 -DNDEBUG -mtune=native -march=native"
-G Ninja
-DBUILD_TESTING:BOOL=OFF
-DBUILD_EXAMPLES:BOOL=OFF
-DITK_BUILD_DOCUMENTATION:BOOL=OFF
# Python
-DITK_WRAP_PYTHON:BOOL=ON
-DITK_USE_SYSTEM_SWIG:BOOL=ON
-DITK_USE_SYSTEM_CASTXML:BOOL=ON
-DITK_LEGACY_SILENT:BOOL=ON
-DBUILD_SHARED_LIBS:BOOL=ON
-DITK_USE_SYSTEM_LIBRARIES:BOOL=ON
-DITK_USE_SYSTEM_JPEG:BOOL=ON
-DITK_USE_SYSTEM_PNG:BOOL=ON
-DITK_USE_SYSTEM_ZLIB:BOOL=ON
-DITK_USE_SYSTEM_TIFF:BOOL=ON
-DITK_USE_SYSTEM_GDCM:BOOL=ON
-DITK_USE_SYSTEM_EXPAT:BOOL=ON
-DITK_USE_SYSTEM_FFTW:BOOL=ON
-DITK_USE_SYSTEM_HDF5:BOOL=ON
-DITK_USE_64BITS_IDS:BOOL=ON
-DITK_WRAP_IMAGE_DIMS:STRING="2;3;4"
-DITK_WRAP_VECTOR_COMPONENTS:STRING="2;3;4"
-DITK_WRAP_float:BOOL=ON
-DITK_WRAP_vector_float:BOOL=ON
# generates a compile_commands.json
-DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON
# https://github.com/hubutui/pkgbuild-for-itk-pkgs/blob/master/itk/PKGBUILD.build-from-source
-DCMAKE_SKIP_INSTALL_RPATH=ON
-DITK_BUILD_DEFAULT_MODULES=ON
)
cmake "${_confopts[@]}"
# If like me, the compilation keeps crashing, try this
local _f=1 _try_i=0 _max_try=20
while [[ ! "${_f}" == "0" ]] && \
[[ "${_i}" -lt "${_max_try}" ]]; do
_f=0
samu -j8 -C "${_build_dir}" ||
_f=$? # catch error
_try_i="$(( ${_i} + 1 ))" # counter
sleep 3; # pause (to kill manually)
done
}
package() {
_build_dir="${srcdir}"/"${_pkgbase}-${pkgver}"/build
cd "${_build_dir}"
# _get_cores_func;
DESTDIR="${pkgdir}" samu -j8 -C "${_build_dir}" install
# Based on AUR's itk-git package
# (quick fix for
# https://github.com/InsightSoftwareConsortium/ITK/issues/2960)
install -dm755 "${pkgdir}"/usr/lib
_pycmd='print("%s.%s" % (sys.version_info[:2]))'
_pyver=$(python -c "import sys; ${_pycmd}")
find "${pkgdir}" -type d -name "python${_pyver}" \
-print0 -quit | xargs -0 mv -vt "${pkgdir}/usr/lib"
python -O -m compileall "${pkgdir}/usr/lib"
}
Actual result
The code which is referenced at the top of this post works well in both cases, but itk.TransformToDisplacementFieldFilter does not load in neither. In both cases (below), I tried this:
python
import itk
import itk.TransformToDisplacementFieldFilter
PKGBUILD (system installation)
File "/usr/lib/python3.10/itk/support/lazy.py", line 125, in __getattribute__
value = types.ModuleType.__getattribute__(self, attr)
AttributeError: module 'itk' has no attribute 'TransformToDisplacementFieldFilter'
Script (custom directory)
cd pkg_test/usr/lib/python3.10/site-packages
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pkg_test/usr/lib/python3.10/site-packages/itk/support/lazy.py", line 125, in __getattribute__
value = types.ModuleType.__getattribute__(self, attr)
AttributeError: module 'itk' has no attribute 'TransformToDisplacementFieldFilter'
Expected result
I need to get the displacement field of a registration, and would like to use itk.TransformToDisplacementFieldFilter.
System description
Table 1: Python and ITK versions| Python | 3.10.1 |
| ITK | 5.2.1 |
| Numpy | 1.21.5 |
| Scipy | 1.7.3 |
| Parabola_(GNU/Linux) | 5.15.12-gnu-1 |
| Samurai | 1.9 |
| CMake | 3.22.2 |
| makepkg | 6.0.0 |