numpy arrays with color normalization

Beginner Questions
1

… python script trying to use numpy arrays in ‘.npy’ files as input to

color normalization, ie args for ‘itk.structure_preserving_color_normalization_filter’

flist is a list of input paths

npy_inp_img = np.load(inp_flist[0])
print(f’ npy_inp_img: dtype, shape = {npy_inp_img.dtype}, {npy_inp_img.shape}’)

npy_inp_img: dtype, shape = uint8, (1024, 1024, 3)

itk_inp_img = itk.GetImageFromArray( npy_inp_img )
print(f’ itk_inp_img: dtype, shape = {itk_inp_img.dtype}, {itk_inp_img.shape}’)

itk_inp_img: dtype, shape = <class ‘numpy.uint8’>, (1024, 1024, 3)

itk_inp_img =
itk.StructurePreservingColorNormalizationFilter[itk.Image[itk.RGBPixel[itk.UC],2]].New(itk_inp_img)

but this fails with error
File “/Users/sauer/Apr_2022_dev/tst.py”, line 184, in
itk.StructurePreservingColorNormalizationFilter[itk.Image[itk.RGBPixel[itk.UC],2]].New(itk_inp_img)
File “/Users/sauer/dl4cv/lib/python3.9/site-packages/itk/itkStructurePreservingColorNormalizationFilterPython.py”, line 636, in New
template_class.New(obj, *args, **kargs)
File “/Users/sauer/dl4cv/lib/python3.9/site-packages/itk/support/template_class.py”, line 800, in New
itk.set_inputs(self, args, kargs)
File “/Users/sauer/dl4cv/lib/python3.9/site-packages/itk/support/extras.py”, line 1209, in set_inputs
new_itk_object.SetInput(args[0])
TypeError: Expecting argument of type itkImageRGBUC2 or itkImageSourceIRGBUC2.
Additional information:
Wrong number or type of arguments for overloaded function ‘itkImageToImageFilterIRGBUC2IRGBUC2_SetInput’.
Possible C/C++ prototypes are:
itkImageToImageFilterIRGBUC2IRGBUC2::SetInput(itkImageRGBUC2 const *)
itkImageToImageFilterIRGBUC2IRGBUC2::SetInput(unsigned int,itkImageRGBUC2 const *)

2

Is the issue that we need the is_vector flag as in

itk_inp_img = itk.GetImageFromArray( npy_inp_img, is_vector=True )
3

inserting ‘is_vector=True’ in the args for ‘itk.GetImageFromArray’ leads to a ‘segmentation fault 11’ when the output is used in ‘itk.StructurePreservingColorNormalizationFilter’

4

But I now see ‘itk.StructurePreservingColorNormalizationFilter’ isn’t needed.

Thanks Matt!

5

Sorry, thanks Lee!!

6

output from the following test script is:
type(image0)=<class ‘itk.itkImagePython.itkImageRGBUC2’>
type(image1)=<class ‘itk.itkImagePython.itkImageRGBUC2’>
type(image2)=<class ‘itk.itkVectorImagePython.itkVectorImageUC2’>
but when attempting color normalization, the reference image is like image1 and the input for the input images from array slices is like image2, which causes
itk.structure_preserving_color_normalization_filter
to exit while processing its args.

test script

import numpy as np
import itk

image0 = itk.imread(“Hard.png”) # as in itk color normalization demo
array = itk.GetArrayFromImage(image0)

ComponentType = itk.UC
PixelType = itk.RGBPixel[ComponentType]
ImageType = itk.Image[PixelType, 2]

image1 = itk.GetImageFromArray( array, ImageType )
print(f’ type(image0)={type(image0)}\n type(image1)={type(image1)}')

patch = array[100:200,100:200,:]
image2 = itk.GetImageFromArray( patch, ImageType )
print(f’ type(image2)={type(image2)}')

7

The color normalization will definitely want both of its inputs to be of the exact same type. What is array.shape? How about array[100:200,100:200,:].shape? Also, does it get any better if we use is_vector=True as the second argument in both itk.GetImageFromArray calls (either instead of ImageType or just before it)?

8

itk_test_script.py sent to ‘ITK discussion forum’ 220516

import numpy as np
import itk

ComponentType = itk.UC
PixelType = itk.RGBPixel[ComponentType]
ImageType = itk.Image[PixelType, 2]

image0 = itk.imread(“Hard.png”) # as in itk color normalization demo
print(f’\n image0.shape = {image0.shape}’)
array = itk.GetArrayFromImage(image0)

image1 = itk.GetImageFromArray( array, ImageType )
print(f’\n image1.shape = {image1.shape}’)
print(f’ type(image0)={type(image0)}\n type(image1)={type(image1)}’)

patch = array[100:200,100:200,:]
image2 = itk.GetImageFromArray( patch, ImageType )
print(f’\n image2.shape = {image2.shape}’)
print(f’ type(image2)={type(image2)}’)

print()

[why aren’t, how to make] lines 2 & 3 identical?

“”" # output above script
image0.shape = (1046, 1790, 3)

image1.shape = (1046, 1790, 3)
type(image0)=<class ‘itk.itkImagePython.itkImageRGBUC2’>
type(image1)=<class ‘itk.itkImagePython.itkImageRGBUC2’>

image2.shape = (100, 100, 3)
type(image2)=<class ‘itk.itkVectorImagePython.itkVectorImageUC2’>
“”" # output above script

Jon R. Sauer
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009

9

What is array.shape? How about array[100:200,100:200,:].shape?

10

Hi Lee, maybe a little better response:

itk_test_script.py sent to ‘ITK discussion forum’ 220516

import numpy as np
import itk

ComponentType = itk.UC
PixelType = itk.RGBPixel[ComponentType]
ImageType = itk.Image[PixelType, 2]

image0 = itk.imread(“Hard.png”) # as in itk color normalization demo
print(f’\n image0.shape = {image0.shape} from “Hard.png”’)
array = itk.GetArrayFromImage(image0)

image1 = itk.GetImageFromArray( array, ImageType )
print(f’\n image1.shape = {image1.shape} from “array” from “Hard.png”’)
print(f’ type(image1)={type(image1)}’)

patch = array[100:200,100:200,:]
image2 = itk.GetImageFromArray( patch, ImageType )
print(f’\n image2.shape = {image2.shape} from slice from “array”’)
print(f’ type(image2)={type(image2)}’)

print()

“”" # output from above

[why aren’t, how to make] all imake types identical?

image0.shape = (1046, 1790, 3) from “Hard.png”

image1.shape = (1046, 1790, 3) from “array” from “Hard.png”
type(image1)=<class ‘itk.itkImagePython.itkImageRGBUC2’>

image2.shape = (100, 100, 3) from slice from “array”
type(image2)=<class ‘itk.itkVectorImagePython.itkVectorImageUC2’>
“”" # output from above

Cheers,
Jon

Jon R. Sauer
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009

11

There is a Python variable called array in your code. I am curious as to what is its shape?

12

I am not sure what is going on. However, as a work-around you could use .copy() as in

image2 = itk.GetImageFromArray( patch.copy(), ImageType )
13

itk_test_script.py sent to ‘ITK discussion forum’ 220516

import numpy as np
import itk

ComponentType = itk.UC
PixelType = itk.RGBPixel[ComponentType]
ImageType = itk.Image[PixelType, 2]

image0 = itk.imread(“Hard.png”) # as in itk color normalization demo
print(f’\n image0.shape = {image0.shape} from “Hard.png”’)
print(f’ type(image0)={type(image0)}’)
array = itk.GetArrayFromImage(image0)

print(f’\n array.shape = {array.shape}’)

image1 = itk.GetImageFromArray( array, ImageType )
print(f’\n image1.shape = {image1.shape} from “array” from “Hard.png”’)
print(f’ type(image1)={type(image1)}’)

patch = array[100:200,100:200,:]
image2 = itk.GetImageFromArray( patch, ImageType )
print(f’\n image2.shape = {image2.shape} from slice from “array”’)
print(f’ type(image2)={type(image2)}’)

print(“\n # [why aren’t, how to make] all image types identical?\n")

“”" # output from above
image0.shape = (1046, 1790, 3) from “Hard.png”
type(image0)=<class ‘itk.itkImagePython.itkImageRGBUC2’>

array.shape = (1046, 1790, 3)

image1.shape = (1046, 1790, 3) from “array” from “Hard.png”
type(image1)=<class ‘itk.itkImagePython.itkImageRGBUC2’>

image2.shape = (100, 100, 3) from slice from “array”
type(image2)=<class 'itk.itkVectorImagePython.itkVectorImageUC2’>

[why aren’t, how to make] all image types identical?

“”" # output from above

Jon R. Sauer
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009

14

Hi Lee,

Thanks. There must have been a shallow copy somewhere that should have been a deepcopy

Jon R. Sauer
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009

itk_test_script.py sent to ‘ITK discussion forum’ 220516

import numpy as np
import itk
import copy

ComponentType = itk.UC
PixelType = itk.RGBPixel[ComponentType]
ImageType = itk.Image[PixelType, 2]

image0 = itk.imread(“Hard.png”) # as in itk color normalization demo
print(f’\n image0.shape = {image0.shape} from “Hard.png”’)
print(f’ type(image0)={type(image0)}’)
array = itk.GetArrayFromImage(image0)

print(f’\n array.shape = {array.shape}’)

image1 = itk.GetImageFromArray( array, ImageType )
print(f’\n image1.shape = {image1.shape} from “array” from “Hard.png”’)
print(f’ type(image1)={type(image1)}’)

patch = array[100:200,100:200,:]
#image2 = itk.GetImageFromArray( patch, ImageType )
image2 = itk.GetImageFromArray( patch.copy(), ImageType ) # Lee Newburg 220516:1406
print(f’\n image2.shape = {image2.shape} from slice from “array”’)
print(f’ type(image2)={type(image2)}’)

print("\n # [why aren’t, how to make] all image types identical?\n")

“”" # output from above
image0.shape = (1046, 1790, 3) from “Hard.png”
type(image0)=<class ‘itk.itkImagePython.itkImageRGBUC2’>

array.shape = (1046, 1790, 3)

image1.shape = (1046, 1790, 3) from “array” from “Hard.png”
type(image1)=<class ‘itk.itkImagePython.itkImageRGBUC2’>

image2.shape = (100, 100, 3) from slice from “array”
type(image2)=<class ‘itk.itkImagePython.itkImageRGBUC2’>

now all image types identical!!

“”" # output from above

15

There’s a bug in ITK or maybe numpy. The patch has a different C_CONTIGUOUS status than the array.

array = np.zeros((40, 50, 3))
patch = array[10:20, 10:20, 3]
print(array.flags)
print(patch.flags)

Either C_CONTIGUOUS shouldn’t be changing, which would be a bug in numpy, or ITK should look elsewhere for determining whether the color is the first or last dimension of the internal representation. Using copy() as above is a workaround for now.

16

I have submitted a bug report, BUG: RGBUC2 not recognized with slice of numpy array · Issue #3429 · InsightSoftwareConsortium/ITK · GitHub.

17

Hi Lee,

Many, many thanks for your workaround. I had been stuck on this for a long time. The context follows.

Whole Slide Images from more than one institution are examined. From a standard dataset labeled from 1024x1024 “tiles” extracted from WSI’s (more than 10k pixels on a side) generated under consistent conditions, deep learning training is done to label train a CNN model. The “tiles” are rather sloppily labeled, but the training from several k tiles is nevertheless pretty good.

Observed new data is obviously H&E colored quite differently than the training data. To attempt to use the trained model on new data I intend to use ITK’s color normalization. Physicians have been arranged to label similar 1024x1024 tiles (converted to numpy array format) of the new data, but, in the end, the goal is to label “patches” as small as 200x200 and color them for physician use in images based on the label. Also to determine the ratio of killed tumor patches to viable tumor patches, which crucially guides treatment. Hence the need for patches massaged to work with the ITK color normalization code.

Thanks again.

Cheers,
Jon

Jon R. Sauer
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009