Big TIFF problem

Hi,

I want to be able to convert (to 8-bit) and compress large 16-bit grayscale 3D images. I’ve been sent a sample file, 10GB, created by cropping a bigger file using ImageJ. The image, which can be viewed in ImageJ, is 3219 x 1820 x 929. When I attempt to read it using itk::ImageFileReader it tells me that the image is 3219 x 1820 x 1. Here is the relevant code (using ITK 4.5.2):

#include “itkImage.h”
#include “itkImageFileReader.h”
#include “itkImageFileWriter.h”
#include “itkSize.h”

typedef itk::Image<unsigned short,3> ImageType_u16;
ImageType_u16::Pointer im_16;

typedef itk::ImageFileReader<ImageType_u16> FileReaderType_u16;
FileReaderType_u16::Pointer reader = FileReaderType_u16::New();
reader->SetFileName(argv[1]);
try
{
reader->Update();
}
catch (itk::ExceptionObject &e)
{
std::cout << e << std::endl;
return 3;
}

im_16 = reader->GetOutput();
width = im_16->GetLargestPossibleRegion().GetSize()[0];
height = im_16->GetLargestPossibleRegion().GetSize()[1];
depth = im_16->GetLargestPossibleRegion().GetSize()[2];
printf("Image dimensions: width, height, depth: %d %d %d\n",width,height,depth);

Am I doing something wrong, or is this a limitation of this ITK version? The code works on a 880MB file.

Thanks, Gib

I faced similar problems, in my case was a problem with SCIFIO reading the tiff: Differences between using component ITKImageIO and explicitly setting ITKIOTIFF

Do you have Module_SCIFIO on?
When you use the reader without specifying the format, SCIFIO gets picked over ther regular ITKIOTIFF for tiff images. Try setting it manually (see link above) and see if the image dimensions are read properly.

Explicitly:

#include "itkImageFileReader.h"
#include "itkTIFFImageIO.h"

  using ReaderType = itk::ImageFileReader< ImageType >;
  auto reader = ReaderType::New();
  reader->SetFileName( inputImage );
  // Defense against dodgy tiff images (wrong metadata)
  // Read incorrectly with SCIFIO module.
  auto tiffIO = itk::TIFFImageIO::New();
  if(tiffIO->CanReadFile(inputImage.c_str()))
    reader->SetImageIO( tiffIO );

1 Like

I don’t even know what module_SCIFIO is, I’ve never heard of it.

reader->GetImageIO() gives:

TIFFImageIO (00000000020A20C0)

RTTI typeinfo: class itk::TIFFImageIO

Reference Count: 1

Modified Time: 177

Debug: Off

Object Name:

Observers:

none

AbortGenerateData: Off

Progress: 0

FileName: \LN_structure\Cardiff\Tumour_Lectin_Staining.tiff

FileType: TypeNotApplicable

ByteOrder: OrderNotApplicable

IORegion:

ImageIORegion (00000000020A2180)

  Dimension: 3

  Index: 0 0 0

  Size: 3219 1820 1

Number of Components/Pixel: 1

Pixel Type: scalar

Component Type: unsigned_short

Dimensions: ( 3219 1820 )

Origin: ( 0 0 )

UseCompression: Off

UseStreamedReading: On

UseStreamedWriting: Off

Compression: 1

I’m afraid I don’t know how to use this code. When I paste that fragment into my program I get a multitude of errors -
1>convert_16-8bit.cpp(52): error C2873: ‘ReaderType’ : symbol cannot be used in a using-declaration
1>convert_16-8bit.cpp(52): error C2143: syntax error : missing ‘;’ before ‘=’
1>convert_16-8bit.cpp(52): error C2065: ‘ImageType’ : undeclared identifier
1>convert_16-8bit.cpp(53): error C2653: ‘ReaderType’ : is not a class or namespace name
1>convert_16-8bit.cpp(53): error C3861: ‘New’: identifier not found
1>convert_16-8bit.cpp(54): error C2227: left of ‘->SetFileName’ must point to class/struct/union/generic type
1> type is ‘‘unknown-type’’
1>convert_16-8bit.cpp(54): error C2065: ‘inputImage’ : undeclared identifier
1>convert_16-8bit.cpp(57): error C3083: ‘TIFFImageIO’: the symbol to the left of a ‘::’ must be a type

That’s c++11 syntax, it might not work in your setup, but I think your problem is not related with mine, so that code won’t solve it.

Scifio: GitHub - scifio/scifio: SCientific Image Format Input & Output: a flexible, extensible framework for image I/O. *EXPERIMENTAL* All API is subject to change, so depend at your own risk! See also @openmicroscopy/bioformats.. There is a module that can be enabled in ITK to use SCIFIO. Used to read quite different format files. ImageJ uses it as well.

The reader is already using TIFFImageIO, so as I said, your problem is different than the one I faced.

Maybe somebody else might have a clue of what’s happening, or how to debug it, I don’t, sorry!

Can you run something like ‘tiffinfo’ to dump all the tags in the file?

I have done that. I didn’t mention that reading this file gives two warning messages about unrecognized tags.
TIFFReadDirectory: Warning, Unknown field with tag 50838 (0xc696) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 50839 (0xc697) encountered.
I didn’t mention it because the messages occur when I read the 883MB image, which is read successfully.
It turns out that each of these two tags is a long list of integers. I have truncated them here - the second tag is about 6 MB.

TIFF Directory at offset 0x8 (8)
Subfile Type: (0 = 0x0)
Image Width: 3219 Image Length: 1820
Resolution: 7751.95, 7751.95 pixels/cm
Bits/Sample: 16
Compression Scheme: None
Photometric Interpretation: min-is-black
Samples/Pixel: 1
Rows/Strip: 1820
Planar Configuration: single image plane
ImageDescription: ImageJ=1.51u
images=929
slices=929
unit=cm
loop=false
min=0.0
max=4095.0

Tag 50838: 12,2436,2436,2436,2436,2436,2436,2436,2436,2436,2436,2436,2436,2436,2436,2436,2436, …
Tag 50839: 73,74,73,74,108,97,98,108,0,0,3,161,0,82,0,101,0,100,0,32,0,70,0,117,0,108,0,108,0, …

I don’t know if these are an artifact of ImageJ.

By the way I’m a bit surprised that tiffinfo does not tell us if this is a bigtiff.

Can you convert some 3D image into TIFF format using ITK (set dimension to 3) and dump its header? That should give us an idea what ITK calls the third dimension, because neither slices not images are enough.

You can check whether the TIFF is a BigTIFF by inspecting the first few bytes of the file:

od -h -j2 -N2 myImage.tif | head -n1 | sed 's/[0-9]*  *//'

A regular TIFF will be 2a00 or 002a (i.e. magic number 42). Whereas a BigTIFF will be 2b00 or 002b (magic number 43). Anything else besides those indicates the TIFF is corrupted, i.e. not a TIFF.

1 Like

Dzenan, I have run tiffinfo on the 883MB file, which ITK can read. Whereas before the output for the 10GB file stopped after the two unrecognized tags, in this case there are many identical sections following. I have truncated the tag lines, and truncated the whole output to show just a few of the sections. There is one section per slice.

TIFF Directory at offset 0x8 (8)
Subfile Type: (0 = 0x0)
Image Width: 973 Image Length: 1178
Resolution: 7751.95, 7751.95 pixels/cm
Bits/Sample: 16
Compression Scheme: None
Photometric Interpretation: min-is-black
Samples/Pixel: 1
Rows/Strip: 1178
Planar Configuration: single image plane
ImageDescription: ImageJ=1.51u
images=394
slices=394
unit=cm
loop=false
min=113.0
max=428.0

Tag 50838: 12,2446,2446,2446,2446,2446,2446,2446,2446,2446,2446,2446,2446,2446,2446,2446,2446,2446, …
Tag 50839: 73,74,73,74,108,97,98,108,0,0,1,138,0,67,0,114,0,111,0,112,0,112,0,101,0,100,0,95,0,90, …
TIFF Directory at offset 0x35e47c49 (904166473)
Subfile Type: (0 = 0x0)
Image Width: 973 Image Length: 1178
Resolution: 7751.95, 7751.95 pixels/cm
Bits/Sample: 16
Compression Scheme: None
Photometric Interpretation: min-is-black
Samples/Pixel: 1
Rows/Strip: 1178
Planar Configuration: single image plane
ImageDescription: ImageJ=1.51u
images=394
slices=394
unit=cm
loop=false
min=113.0
max=428.0

TIFF Directory at offset 0x35e47ceb (904166635)
Subfile Type: (0 = 0x0)
Image Width: 973 Image Length: 1178
Resolution: 7751.95, 7751.95 pixels/cm
Bits/Sample: 16
Compression Scheme: None
Photometric Interpretation: min-is-black
Samples/Pixel: 1
Rows/Strip: 1178
Planar Configuration: single image plane
ImageDescription: ImageJ=1.51u
images=394
slices=394
unit=cm
loop=false
min=113.0
max=428.0

TIFF Directory at offset 0x35e47d8d (904166797)
Subfile Type: (0 = 0x0)
Image Width: 973 Image Length: 1178
Resolution: 7751.95, 7751.95 pixels/cm
Bits/Sample: 16
Compression Scheme: None
Photometric Interpretation: min-is-black
Samples/Pixel: 1
Rows/Strip: 1178
Planar Configuration: single image plane
ImageDescription: ImageJ=1.51u
images=394
slices=394
unit=cm
loop=false
min=113.0
max=428.0

Hi Curtis,

I am using Windows, and Windows 7 does not have that tool. I tried using xvi32, but it is not able to load the 10GB file, presumably because it uses 32-bit IDs. Do you know of a Windows program that I could use for this?

Cheers, Gib

More info. I have learned from the source of these images that my ITK-based program works on images < 4GB, fails otherwise. I have also created a 4.4GB tiff and discovered that I can read it. In other words, a big tiff created with ITK can be read by ITK - no surprise there, I guess.

Here is the start of the output from tiffinfo with the file I created:

TIFF Directory at offset 0x298ad4 (2722516)
Subfile Type: multi-page document (2 = 0x2)
Image Width: 1650 Image Length: 1650
Resolution: 25.4, 25.4 pixels/inch
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: min-is-black
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 1
Rows/Strip: 635
Planar Configuration: single image plane
Page Number: 0-1650
Software: InsightToolkit
TIFF Directory at offset 0x53173c (5445436)
Subfile Type: multi-page document (2 = 0x2)
Image Width: 1650 Image Length: 1650
Resolution: 25.4, 25.4 pixels/inch
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: min-is-black
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 1
Rows/Strip: 635
Planar Configuration: single image plane
Page Number: 1-1650
Software: InsightToolkit
TIFF Directory at offset 0x7ca3a4 (8168356)
Subfile Type: multi-page document (2 = 0x2)
Image Width: 1650 Image Length: 1650
Resolution: 25.4, 25.4 pixels/inch
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: min-is-black
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 1
Rows/Strip: 635
Planar Configuration: single image plane
Page Number: 2-1650
Software: InsightToolkit

It goes like this for some time, then fails with this message:
TIFFFetchDirectory: zzz_unc.tif: Seek error accessing TIFF directory.
TIFFReadDirectory: Failed to read directory at offset 2148383476.
which is apparently a limitation of tiffinfo.

I think my correspondent uses ImageJ for all image processing, so it looks as if ImageJ is saving big tiffs using a format that ITK4 can’t handle.

Hi Bradley,

In response to your email suggestion (which didn’t get here), I am reluctant to build ITK 5, because that will require me to rebuild VTK, and all my tools that use ITK, and probably change the way I do things in some places with both ITK and VTK. I regard this as a “last-ditch” option.

Is there not a patch for ITK4?

Thanks for responding
Gib

More info.
I opened in ImageJ the 4.4GB tiff that I created with ITK, then saved it with a different name. The save was successful, but at the end the log file had this:

Stack is larger than 4GB. Most TIFF readers will only open the first image. Use this information to open as raw:
name=Untitled, dir=, width=1650, height=1650, nImages=1650, offset=75319, type=byte, byteOrder=big, format=0, url=, whiteIsZero=f, lutSize=256, comp=1, ranges=null, samples=1

The tiff that was written was almost the same size as the original. tiffinfo gives the same results as before: a single section for one slice, then the two unrecognized tags, each followed by a long series of integers, nothing more.

Bradley has suggested that I test with ITK 5, but I’m not keen to venture down that path.

As far as I can tell, ImageJ uses the Bio-Formats library to do image file IO. Apparently Bio-Formats uses OME_TIFF: https://docs.openmicroscopy.org/ome-model/5.6.1/ome-tiff/specification.html
" Note this is for the TIFF standard specification only; the header structure is slightly different for BigTIFF; see the BigTIFF file format specification."
For some reason the BigTIFF format used by Bio-Formats differs from that implemented in ITK4.

Good tracking @Gib, if not keen to update, you can try to use BioFormats directly in ITK adding the module SCIFIO.

With the cmake option: Module_SCIFIO = ON
You can do that from your current itk build folder with:

cmake -DModule_SCIFIO:BOOL=ON /path/to/your/ITK-source

and rebuild ITK.

Thanks Pablo.

Is that available in ITK 4?
If so, is there an example showing how to use it to read a tiff?

Cheers, Gib

When the module is ON, it will be selected and used directly from the ImageReader, you don’t need to change anything a priori. And yes, it is available for ITK4.

However, you can fine tune what ImageIO the reader uses selecting it directly with something like:

  itk::SCIFIOImageIO::Pointer scifioIO = itk::SCIFIOImageIO::New();
  if(scifioIO->CanReadFile(myFilename.c_str()))
    reader->SetImageIO( scifioIO );

This can be useful when multiple IO can read the same image type, and you choose which one to use directly. But, in my experience SCIFIO gets picked first before TIFFIO, so you can try without any modification in your code and the problematic images.

You can see what imageIO has been selected for testing/debugging purposes in your reader with something like:

reader->GetImageIO()->Print( std::cout );
1 Like

That looks good, but unfortunately when I turn Module_SCIFIO on (in cmake_GUI) in my previous build and click Configure I get an error:

CMake Error at CMake/ITKModuleRemote.cmake:12 (message):
Failed to clone repository: ‘https://github.com/scifio/scifio-imageio.git
Call Stack (most recent call first):
CMake/ITKModuleRemote.cmake:107 (_git_clone)
CMake/ITKModuleRemote.cmake:152 (_fetch_with_git)
Modules/Remote/SCIFIO.remote.cmake:1 (itk_fetch_module)
Modules/Remote/CMakeLists.txt:6 (include)

I’m building from a source tree that I downloaded InsightToolkit-4.7.2.zip.