Is there any convention for storing label information (tissue name, color, index, maybe other metadata) in medical image segmentation? Does ITK support loading/saving this metadata?
How does the ML community handle such data?
I have not yet come across any convention. Internally we use a text file that assumes label=0 is “Background”, and then lists one tissue name and RGB color per line.
Also, is there any (non-proprietary) format that stores the segmented image AND the label information in a single file?
There is no widely adopted standard for storing all of the potential segmentation representations and associated meta-data.
Key points:
There are a variety of ways of representing a (fractional/overlapping) segmentation. The slicer documentation has a very nice overview.
Storing meta-data is a separate requirement and is most often not of interest in DL. In DL you usually don’t care what the labels mean, human readable description, or what color to use for display, you just want to segment them as well as possible. This is why many DL datasets store their segmentations in nii.gz (segmentation sans meta-data).
Having said this,
If you need a file format that conforms to an official standard and can also store any meta-data you want, visualization parameters, segment names etc… DICOM may be what you want. See the nice work from the DCMQI folks.
Finally, many research formats (i.e. nrrd, mha…) can store the segmentation and associated meta-data in a single file. The image’s meta-data dictionary can be retrieved from the image after reading it using ITK/SimpleITK.
In 3D Slicer segmentation files (.seg.nrrd) we use a hybrid method. We store label metadata in custom fields in a nrrd file that does not depend on or require any knowledge of DICOM. However, the fields were developed in collaboration by DCMQI people in a way that allows storing all essential metadata of a DICOM segmentation object, to make lossless DICOM → nrrd → editing → nrrd → DICOM roundtrip feasible.
The format supports overlapping segments: in case there are overlapping segments then a 4D image (3D volume list) is stored, and segment metadata specifies which volume stores the voxels with what label value.
Arbitrary number of key-value pairs can be also stored for each segment to store custom metadata. This is leveraged for storing essential DICOM data (in the “TerminologyEntry” field), which can be used for non-DICOM applications as well. The most useful concept in this terminology entry is the usage of standard coded terms. Instead of using just a string to specify name of a segment, a (coding scheme, code value, code meaning) triplet is used. Coding scheme specifies a terminology (SNOMED Clinical Terms; Terminologica Anatomica 2, etc. and it is possible to specify custom dictionaries as well), and code value is a machine-readable string (usually letters and numbers) that uniquely defines the term within the dictionary. Code meaning is a human-readable string, usually in English that can be displayed to the user. Using coded terms makes content definition very robust (reduces errors in data entry), allows translation of content to any languages (code value is language independent), and allows interoperability between different applications.
Specification of the custom fields are available here.
We also provide slicerio - a little Python package that allows convenient parsing of the seg.nrrd file header to retrieve metadata or pixel data of any segment (parsing of TerminologyEntry and overlapping segments are supported).
We try to get .seg.nrrd fields and pixel data layout to become a part of the emerging OME-Zarr file format. There is a big momentum in the medical image computing community to develop this new, Python and web friendly file format, which is more complex than current research file formats (nrrd, metaIO, nifti), but offer many more features.