Posting on discourse as I am mot sure whether to classify this as a bug or a philosophical issue.
In general ITK is agnostic to metric units, though most of us that work with medical images assume the unit of measurement is mm (DICOM standard specifies this). This is not the case if you are working with microscopy (nm spacing) or remote sensing (m, km). You can work comfortably with ITK as long as all your computations use the same metric units (mixing units or using incorrect ones will not end well).
My specific issue is with the Nifti IO. The writer implicitly sets image sizes to mm. Should the user be able to set this value through the meta-data dictionary? Or is the official unit of distance in ITK mm?
In any case, the IO is consistent in that the reader scales the units to mm too.