MetaImage: M_ReadElementsROI: data not read completely

JK_Kim · July 14, 2021, 8:54pm

Hi

I am using SimpleITK 2.0.2 and Python 3.7.9 to read a 3D CT image in the compressed mhd format (ct.mhd, ct.zraw). When I run the code below, I am getting the following message. “data not read completely”. So, I saved the read image back to file, opened in a viewer and I got the correct image. So, it seems like there is no issue for the code… Anyone has any clue why I am getting this message?

Thanks.
JK

[message]
MetaImage: M_ReadElementsROI: data not read completely
ideal = 1069833 : actual = 633067

[code]
file_reader = sitk.ImageFileReader()
file_reader.SetFileName(img_path)
file_reader.SetExtractIndex(start_index)
file_reader.SetExtractSize(extract_size)
img = file_reader.Execute()

dzenanz · July 15, 2021, 12:20pm

That might be true for an image that is mostly zero, so the last 40% of it which was not read only contained zeroes. This is practically possible with label images.

dzenanz · July 15, 2021, 12:21pm

I can’t quite imaging that being the case for a CT image though. Can you share your image?

Will_G · September 15, 2021, 12:44pm

Did you manage to find a resoluton to this issue?, I am having the same one as well.

Z_Huang · June 26, 2022, 4:18am

True for me. I have experienced the same “MetaImage: M_ReadElementsROI: data not read completely” issue and figured the 3D image is a label, where 28,295,732 pixels are 0 and 15,820 pixels are labels.

jkasten · July 11, 2023, 9:49pm

I’m experiencing similar issues with (non-binary) CT images.
As an experiment I tried reading a CT image with a purposely corrupted .raw file.
when doing so, I see the console message:

MetaImage: M_ReadElementsData: data not read completely
   ideal = 203948032 : actual = 16384

What is concerning is that with ITK 4.13, attempting to read this image with the standard itk::ImageFileReader would throw an itk::ExceptionObject. However, after updating to ITK 5.2 reading fails silently.

Stephen_Aylward · July 12, 2023, 1:05pm

This is a bug! Great catch!

The M_ReadElementsROI() function correctly returns false in this condition:

github.com

Kitware/MetaIO/blob/0b8c5284f3fe126968fd66c24639fd70b88e8788/src/metaImage.cxx#L3012


      
                  }
                }
              }
            }
          
          
  if (gc != readSize)
            {
              std::cerr << "MetaImage: M_ReadElementsROI: data not read completely" << std::endl;
              std::cerr << "   ideal = " << readSize << " : actual = " << gc << std::endl;
              delete[] currentIndex;
              return false;
            }
          
          
  delete[] currentIndex;
          }
          else // if not compressed
          {
            double tf = 0;
            MET_SizeOfType(m_ElementType, &elementSize);
          
          
  char * data = static_cast<char *>(_data);

but that return value is now being lost within the M_ReadROIStream() function

github.com

Kitware/MetaIO/blob/0b8c5284f3fe126968fd66c24639fd70b88e8788/src/metaImage.cxx#L2593


      
            quantity *= (_indexMax[i] - _indexMin[i] + 1);
          }
          
          
bool        usePath;
          std::string pathName;
          std::string fName;
          usePath = MET_GetFilePath(m_FileName, pathName);
          
          
if ("Local" == m_ElementDataFileName || "LOCAL" == m_ElementDataFileName || "local" == m_ElementDataFileName)
          {
            M_ReadElementsROI(_stream, m_ElementData, quantity, _indexMin, _indexMax, subSamplingFactor, m_Quantity);
          }
          else if ("LIST" == m_ElementDataFileName.substr(0, 4))
          {
            int     nWrds;
            char ** wrds;
            MET_StringToWordArray(m_ElementDataFileName.c_str(), &nWrds, &wrds);
            for (i = 0; i < nWrds; i++)
            {
              delete[] wrds[i];
            }

and also in lines 2661, 2767, and 2832 in that file.

I’ll create a pull request to fix these errors in the MetaIO library. I’ll need to work with the other ITK devs to determine how to update that library in ITK.

Thanks!
Stephen

dzenanz · July 12, 2023, 1:07pm

Once the PR is merged in MetaIO repository, invoking UpdateFromUpstream.sh script in ITK should update it appropriately.

Stephen_Aylward · July 12, 2023, 1:14pm

Actually, ignoring the return value is a now systemic problem in the metaImage.cxx file - happens with writing and reading. It is going to take some hunting to fix all of them. git blame…

Stephen_Aylward · July 12, 2023, 11:50pm

Here is the MetaIO pull request:

github.com/Kitware/MetaIO

BUG: Propagate return values from functions to catch errors

Kitware:master ← aylward:PropagateReturnValues

opened 11:49PM - 12 Jul 23 UTC

aylward

+163 -53

Many calls to private functions (e.g., M_ReadElementsROI) did not check or propa…gate return values, so errors were not being caught, and ITK reading was failing silently on invalid files. Also ROI reading of compressed data was not computing size of data being read correctly, so once we started catching errors, the function was reporting requested data size not matching returned data size and was now failing with checks added. Fixed the computation of gc in M_ReadElementsROI for compressed data. See discussion: https://discourse.itk.org/t/metaimage-m-readelementsroi-data-not-read-completely/4269/9

dzenanz · July 17, 2023, 2:45pm

ITK PR is now open:

github.com/InsightSoftwareConsortium/ITK

MetaIO 2023-07-16 (556864f5)

InsightSoftwareConsortium:master ← dzenanz:updateMetaIO

opened 02:44PM - 17 Jul 23 UTC

dzenanz

+163 -53

Code extracted from: https://github.com/Kitware/MetaIO.git at commit 5…56864f5a7bb2853466d0d22068b268e49137e52 (master). ## PR Checklist - [x] No [API changes](https://github.com/InsightSoftwareConsortium/ITK/blob/master/CONTRIBUTING.md#breaking-changes) were made (or the changes have been approved) - [x] No [major design changes](https://github.com/InsightSoftwareConsortium/ITK/blob/master/CONTRIBUTING.md#design-changes) were made (or the changes have been approved)

Stephen_Aylward · July 22, 2023, 1:23pm

Merged!