"The Continuous Registration Challenge"

Kasper Marstal, Floris Berendsen, Niels Dekker, Marius Staring and Stefan Klein

Abstract

Introduction: Image registration is a common preprocessing step in many different medical image analysis applications. However, registration methods are implemented in a large variety of toolboxes, and the methods are rarely compared on the same datasets. The lack of standardization makes it challenging for end-users to select the right registration algorithm and hyperparameters for their application.
To standardize comparison, registration methods can enter a Grand Challenge (GC). GCs are competitions with standardized data sets, evaluation methods, and experimental setups that focus on specific research topics. The experiments are run by third parties which ensures fair, independent evaluations. However, most modern GCs are static, one-time events, that allow closed source contributions. This hampers collaboration and reproducibility.

Methods: To address these limitations, and inspired by modern software development practices, we proposed the Continuous Registration Challenge (CRC). For this challenge, we developed a fully automatic platform for benchmarking registration methods on many different data sets. The platform consists of a C++ API for running registrations, a Python framework and a Continuous Integration system for running experiments, a compute backend, and a website with public leaderboards. The platform and all submissions are open source. The C++ API, SuperElastix, was designed with a role-base architecture that allows many different registration paradigms to co-exist in the same framework. The challenge focuses on pairwise registration of lungs and brains, two problems frequently encountered in clinical settings.

Results: The system described above is used for the Continuous Registration Challenge. The system allows researchers to test their methods in a standardized way, using a fully automatic experimental setup. The experiments are running every week, and participants can follow the leaderboards, which are updated every weekend (https://continuousregistration.grand-challenge.org/leaderboard/). The leaderboards will continue to be updated and the repository will be open for contributions even after the challenge ends. The results will be presented and discussed at the Workshop On Biomedical Image Registration (WBIR 2018, https://wbir2018.nl/). All participants are invited to collaborate on a paper which we plan to submit to a leading journal in the field.

Conclusion: We present an open source framework for the continuous and automated benchmarking of image registration algorithms.

BibTeX entry