Announcing the Continuous Registration Challenge (CRC)

Do you love developing registration algorithms but are you tired of the time it takes to properly setup experiments? What if you could just implement your method and then run experiments on many different data sets and compare your method against other registration algorithms with no extra work? What if your algorithm was packaged, maintained, easily accessible to end-users, and benchmarked weekly all in the name of open source software and reproducible science? Does this sound like a sales pitch for a registration challenge that completely automates code testing and the experimental setup? It is!

We are excited to invite you to participate in the Continuous Registration Challenge (CRC). CRC is a challenge for registration of lung- and brain images inspired by modern software development practices.

To participate, you simply implement your algorithm using the open source SuperElastix C++ API on GitHub which comes with continuous integration, experiments, and evaluation system out of the box. All submissions are automatically compiled, tested, and benchmarked weekly on eight different data sets on our own cluster.

The goal is to provide an open source, collaborative platform where researchers develop, compare, and continuously improve algorithms and parameter settings for many different body parts.

The challenge focuses on pairwise registration of lungs and brains, two problems frequently encountered in clinical settings. We have collected seven open-access data sets and one private data set (3+1 lung data sets, 4 brain data sets). The challenge results will be presented and discussed at the upcoming Workshop On Biomedical Image Registration (WBIR 2018). All participants will be invited to collaborate on a journal paper describing the results of the challenge.

Experiments will continue to run and new data sets will be added even after the challenge event at WBIR2018.

Important dates:

  • February 9th 2018: Announcement of challenge and release of SuperElastix 1.0.
  • June 14th 2018: Final commit to the SuperElastix repository for challenge participants.
  • June 28/29 2018: Results presented at the Workshop on Biomedical Image Registration 2018.
  • June 30: SuperElastix Hackathon.

Would you like to know more? Then join our introductory webinar on Monday February 12th at 16:00 CET. Sign up here. The introductory webinars will take place every Monday at 16:00 until April 23rd. We also have weekly technical Q&A webinars on Tuesdays at 16:00 from February 20th until June 15th to help you get started with SuperElastix. Sign up here. You can also request an invite to our slack channel here where we will help you implement your method and talk about everything related to the challenge.

You can read this and more at https://continuousregistration.grand-challenge.org. Hope to see you there!

Kind Regards,
Kasper Marstal (k.marstal@erasmusmc.nl)
Floris Berendsen (f.berendsen@lumc.nl)
Niels Dekker (c.e.dekker@lumc.nl)
Marius Staring (m.staring@lumc.nl)
Stefan Klein (s.klein@erasmusmc.nl)

6 Likes

We are happy to announce the following improvements to the Continuous Registration Challenge format:

  • Attribution. Users of SuperElastix are explicitly asked to cite the authors of the algorithms to avoid users simply citing SuperElastix itself. Authors can put citing information directly in code, and the citing information will be printed to the registration log everytime a registration is run. For example, using a configuration consisting of a niftyreg registration followed by ANTs registration will result in citing information for these to libraries to be printed to the log.
  • Evaluation metrics. Participants can now submit new evaluation metrics in addition to their algorithm. Results from all participants will be evaluated with the new metrics if they are accepted and merged. The evaluation metrics are implemented here: https://github.com/SuperElastix/SuperElastix/blob/SELX-58-prototype-for-continuous-challenge-infrastructure-2/ContinuousRegistration/Source/metrics.py. Simply submit a pull request to have your metric evaluated for inclusion in the challenge.

The changes will be in effect from Monday February 26th 2018. If there are any other issues that we should be aware of let us know! Reply here or send an email to k.marstal@erasmusmc.nl, or visit our mailing list at https://groups.google.com/forum/#!categories/elastix-imageregistration/continuousregistration.

Happy coding!
Kasper Marstal
Floris Berendsen
Marius Staring
Stefan Klein

2 Likes