Proposal: Hard-fork vnl inside ITK and end upstream-vxl tracking

Engineering
1

TL;DR

ITK’s use of vxl is narrow, concentrated, and already well past the
point where co-maintaining a synced copy with vxl/vxl upstream
provides reciprocal value. I’m proposing we:

  1. Deprecate and then remove ITK_USE_SYSTEM_VXL.
  2. Stop tracking upstream vxl/vxl for the vendored copy.
  3. Treat ITK’s vnl as a hard fork — vendored, pruned to the closure
    of what ITK actually compiles, evolved on ITK’s schedule.
  4. Use that freedom to stage a migration of vnl call sites to Eigen
    and <numbers>/<cmath> where straightforward, and to introduce
    ABI/API improvements upstream couldn’t accept.

Why now — what the audit shows

Symbol-level audit of Modules/ (excluding the vendored copy):

  • ITK uses only core/vnl, core/vnl/algo, core/vcl (already
    zero vcl_* includes — fully migrated), and v3p/netlib.
  • Zero references to core/vbl, core/vgl, core/vul,
    core/vil, core/vsl, core/vpl, or any of contrib/.
  • Top vnl consumers by mention count: vnl_vector (405),
    vnl_matrix (202), vnl_vector_fixed (81), vnl_matrix_fixed
    (73), vnl_nonlinear_minimizer (50), vnl_math (45),
    vnl_lbfgsb (21), vnl_svd (19), vnl_lbfgs (18),
    vnl_matrix_inverse (17), vnl_quaternion (15), vnl_amoeba
    (14), vnl_cost_function (14), vnl_sparse_matrix (14),
    vnl_symmetric_eigensystem (6), vnl_fft_base (7).

We already pass VXL_BUILD_CORE_NUMERICS_ONLY=ON, so the build is
narrow — but the source tree under Modules/ThirdParty/VNL/src/vxl/
ships everything.

Why now — what the fork landscape shows

A survey of the 124 visible forks of vxl/vxl:

  • All the ITK-contributor forks (hjmjohnson, thewtex, blowekamp,
    N-Dekker, bradking, dzenanz, seanm) are 0 commits ahead of
    upstream master — pure PR-staging mirrors.
  • The only meaningfully divergent fork is one researcher’s
    computer-vision fork (rfabbri/vxl-1, ~2500 commits ahead).
  • No organization-owned downstream maintains a divergent vxl.

The active contributor base outside ITK is one other organization
working in computer-vision territory whose priorities (image
algorithms, geometry, multi-view, etc.) diverge from ITK’s
medical-imaging-numerics priorities. The cost of mutual
accommodation has stopped being symmetric.

What the freedom buys us

  • Pruning. Drop everything outside ITK’s compile closure — most
    of the vendored tree.
  • Surface reduction. Most vnl API isn’t exposed in ITK’s public
    headers. The unexposed portion can be hidden, renamed, made
    internal, or wrapped behind ITK facades without backward-compat
    worries.
  • A real migration path to Eigen + std. In rough order:
    • vnl_math → <numbers> / <cmath>
    • vnl_*_fixed<T,N,M> → Eigen::Matrix<T,N,M>
    • vnl_matrix / vnl_vector → Eigen::MatrixXd / VectorXd
    • vnl_svd / vnl_symmetric_eigensystem / vnl_sparse_matrix →
      Eigen equivalents
    • Optimizers (vnl_lbfgs, vnl_lbfgsb, vnl_amoeba,
      vnl_levenberg_marquardt, vnl_powell,
      vnl_conjugate_gradient) last, with a numerical regression
      suite, because ITK’s registration framework depends on their
      exact behavior.
  • Performance work. Alignment, SIMD, view types, move semantics,
    allocator-aware containers — changes that would be ABI/API-breaking
    upstream and so blocked there indefinitely.

What it costs

  • ITK no longer pulls upstream vxl bug fixes automatically. The
    flux into ITK’s slice has been small in recent years; we accept
    this.
  • ITK_USE_SYSTEM_VXL consumers (do any exist in production?) need
    a deprecation path — one minor release with a warning, then
    removal.
  • Round-tripping vnl fixes through upstream vxl becomes a manual
    cherry-pick when relevant. In practice this has been rare.

Open questions for the community

  1. Production ITK_USE_SYSTEM_VXL users — who are you? If you
    build ITK against a system vxl in production (packaging,
    distros, downstream SuperBuilds), please speak up. We need to
    know who needs a migration path.
  2. Optimizer numerical equivalence. What’s the appetite for
    building an optimizer regression suite as a precondition for the
    final migration phase? Anyone willing to own that effort?
  3. Scope discipline. Should we draw an explicit line that says
    “any vnl behavior not currently exercised by an ITK test is fair
    game to delete” — and if so, how do we communicate that
    externally?
  4. Naming. Once the prune lands, is Modules/ThirdParty/VNL/
    still the honest name? Some prefer to make the rebrand explicit
    (e.g., VNLForITK); others prefer to leave the directory name
    alone to minimize churn.
  5. Coordination with upstream vxl. Is a courtesy heads-up to
    vxl/vxl maintainers warranted before the deprecation lands?
    I lean yes — the relationship is professional and worth
    maintaining even as the technical paths diverge.

Discussion welcome. The GitHub issue is the canonical record for
The proposal itself; this thread is for community discussion that
benefits from Discourse’s threading and broader audience.

2

Thanks @hjmjohnson for being thoughtful and seeking feedback from ITK users before making any significant decision.

I would just provide one illustrative example. We maintain the PLUS toolkit, which includes calibration algorithms for image-guided therapy. Over the years, we have relied on optimizers and a number of smaller numerical algorithms that ITK conveniently provided through VNL. See our current VNL usage here.

We would not object to changes in ITK, as long as an LLM could reliably and automatically update our codebase without introducing regressions. How realistic do you think that expectation is?

Even if the changes are likely to introduce small problems for us (e.g., we need to update our test baselines, maybe tune some optimization parameters), they may be still acceptable, as long as there are significant direct benefits for us (such as speed would be many times faster by utilizing many CPU cores or GPU, or the changes would allow easily building pyodide packages and run our software in the browser while without the changes these would be nearly impossible).

If the changes may bring problems but there is no direct upside for us then it just feels like a punishment: we are forced to spend time (either diverting resources from funded projects or give up some of our evenings and weekends) just to keep everything working as is. If this was the case then I would prefer if the ITK changes don’t happen at all, at least until AI agents get better (so that transition becomes painless) or alternative solution is found (that provides direct benefits, which can justify the time investment).

1 Like
3

I did a quick search in all 3D Slicer source code and extensions as well. It seems that the set of non-trivial VNL features (that may not have exact Eigen replacement) is very small:

  • linear solvers: vnl_lsqr, vnl_sparse_matrix_linear_system
  • optimization: vnl_amoeba, vnl_cost_function

They are all in SlicerRT, SlicerIGSIO, SlicerANTs. Slicer core only contains trivial VNL usage.

A possible low-risk approach could be to keep VNL linear solvers and optimizers in ITK exposed the same way as they are now.

1 Like