seg fault when looping color normailzation

Beginner Questions
1

For a demo code based on the ITK color normalization example, remove the line
eager_normalized_image = itk.structure_preserving_color_normalization_filter(
and succeeding lines replacing them with:
< ++++++++++++++++++++
tile = input_image
n = 14 # make patch/sliced grid 14x14
l = min(tile.shape[0],tile.shape[1]) # side of sliced grid
s = int(l/n) # side of a slice
p = l - n*s
print(f’ *** tile.shape[0] = {tile.shape[0]}, tile.shape[1] = {tile.shape[1]}’)
print(f’ *** n = {n}, s = {s}, p = {p:.3g} before halving’)
p = int(p/2) # index offset

for i in range(n): # x patch index
I = p + is # patch top left in x
for j in range(n): # y patch index
J = p + js # patch top left in y
patch_npa = tile[J:J+s,I:I+s,:]
patch_itk_inp = itk.GetImageFromArray(patch_npa.copy(), is_vector=True)
input_image = patch_itk_inp

    patch_CN = itk.structure_preserving_color_normalization_filter(
        input_image,
        reference_image,
        color_index_suppressed_by_hematoxylin=0,
        color_index_suppressed_by_eosin=1
    )

    print(f'\t({i:2},{j:2}) ({I:4}:{I+s:4}, {J:4}:{J+s:4}) {type(patch_CN)}’)

++++++++++++++ >
For me, this produces a segmentation fault for n > 13 for this ‘input_image’. Different inputs still
give a seg fault, but at different ’n’ values.

I tried deleting ‘patch_npa’, ‘patch_itk_inp’, ‘patch_CN’ immediately to no avail.

2

note that, for this python code, somehow python-crucial formatting has been removed: indenting spacing, a few '’ as in
I = p + is # patch top left in x
and
J = p + j*s # patch top left in y

didn’t know this was going to happen, sorry

3

In markdown, you can put your code between triple-backticks to preserve formatting:

```
your code
is here
```
4

Thanks Andras

#  < =====================================================================
# try looping over slices of input_image
tile = input_image
n = 14  # make patch/sliced grid 14x14
l = min(tile.shape[0],tile.shape[1])  # side of sliced grid
s = int(l/n)  # side of a slice
p = l - n*s
print(f' *** tile.shape[0] = {tile.shape[0]}, tile.shape[1] = {tile.shape[1]}')
print(f' *** n = {n},  s = {s},  p = {p:.3g} before halving')
p = int(p/2)  # index offset

for i in range(n):  # x patch index
    I = p + i*s  # patch top left in x
    for j in range(n):  # y patch index
        J = p + j*s  # patch top left in y
        patch_npa = tile[J:J+s,I:I+s,:]
        patch_itk_inp = itk.GetImageFromArray(patch_npa.copy(), is_vector=True)
        input_image = patch_itk_inp

        patch_CN = itk.structure_preserving_color_normalization_filter(
            input_image,
            reference_image,
            color_index_suppressed_by_hematoxylin=0,
            color_index_suppressed_by_eosin=1
        )

        print(f'\t({i:2},{j:2}) ({I:4}:{I+s:4}, {J:4}:{J+s:4}) {type(patch_CN)}’)
# ====================================================================== >
5

Thanks Andras!

Cheers,
Jon

Jon R. Sauer
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009

6

Do you get any output from the print statements before the segmentation fault? In addition to what is already printed, I’d like to see all entries of tile.shape and patch_npa.shape, as well as the values of I and J when they are used. Also, type(input_image) and type(reference_image).

I see that the variable name input_image is re-used. If this program is run more than once in a row, maybe that could be an issue.

7

Hi Lee,
Many thanks for finding the time to look at this:

program output:


*** tile.shape=(1186, 1912, 3)
*** tile.shape[0] = 1186, tile.shape[1] = 1912
*** n = 14, s = 84, p = 10 before halving
type(input_image)=<class 'itk.itkImagePython.itkImageRGBUC2'>
type(reference_image)=<class 'itk.itkImagePython.itkImageRGBUC2'>

( 0, 0) ( 5: 89, 5: 89) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 1) ( 5: 89, 89: 173) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 2) ( 5: 89, 173: 257) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 3) ( 5: 89, 257: 341) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 4) ( 5: 89, 341: 425) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 5) ( 5: 89, 425: 509) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 6) ( 5: 89, 509: 593) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 7) ( 5: 89, 593: 677) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 8) ( 5: 89, 677: 761) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0, 9) ( 5: 89, 761: 845) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0,10) ( 5: 89, 845: 929) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0,11) ( 5: 89, 929:1013) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0,12) ( 5: 89, 1013:1097) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 0,13) ( 5: 89, 1097:1181) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 1, 0) ( 89: 173, 5: 89) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 1, 1) ( 89: 173, 89: 173) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 1, 2) ( 89: 173, 173: 257) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 1, 3) ( 89: 173, 257: 341) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 1, 4) ( 89: 173, 341: 425) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
( 1, 5) ( 89: 173, 425: 509) <class 'itk.itkImagePython.itkImageRGBUC2'>
patch_npa.shape=(84, 84, 3)
Segmentation fault: 11
$ /usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d ‘

Python script producing above:


# Import itk, which includes itk-spcn.
import itk
#from itkwidgets import view # JRS

# Fetch input images, if we don have them already.
input_image_filename = 'Easy1.png'

reference_image_filename = 'Hard.png'

# The pixels are RGB triplets of unsigned char. The images are 2 dimensional.
PixelType = itk.RGBPixel[itk.UC]
ImageType = itk.Image[PixelType, 2]

# Invoke the functional, eager interface for ITK
input_image = itk.imread(input_image_filename, PixelType)
reference_image = itk.imread(reference_image_filename, PixelType)

# try looping over slices of input_image
tile = input_image
n = 14 # make patch/sliced grid 14x14
print(f' *** tile.shape={tile.shape}')
l = min(tile.shape[0],tile.shape[1]) # side of sliced grid
s = int(l/n) # side of a slice
p = l - n*s
print(f' *** tile.shape[0] = {tile.shape[0]}, tile.shape[1] = {tile.shape[1]}')
print(f' *** n = {n}, s = {s}, p = {p:.3g} before halving')
p = int(p/2) # index offset

print(f' type(input_image)={type(input_image)}')
print(f' type(reference_image)={type(reference_image)}\n')
for i in range(n): # x patch index
I = p + i*s # patch top left in x
for j in range(n): # y patch index
J = p + j*s # patch top left in y
patch_npa = tile[J:J+s,I:I+s,:]
patch_itk_inp = itk.GetImageFromArray(patch_npa.copy(), is_vector=True)
input_image = patch_itk_inp

patch_CN = itk.structure_preserving_color_normalization_filter(
input_image,
reference_image,
color_index_suppressed_by_hematoxylin=0,
color_index_suppressed_by_eosin=1
)

print(f' ({i:2},{j:2}) ({I:4}:{I+s:4}, {J:4}:{J+s:4}) {type(patch_CN)}')
print(f'\t patch_npa.shape={patch_npa.shape}')

Jon R. Sauer
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009

8

My quick poking around shows that patches that have no hematoxylin (blue) are failing. Instead of a segmentation fault that bombs out your program, we should be gracefully receiving an error along the lines of The image to be normalized could not be processed; does it have white, blue, and pink pixels?

I realize that saving your program from bombing out will be only partially satisfying. We could try to extend the SPCN algorithm beyond the original academically published version, so that it works even when only one stain is detected, … but that is a bit of research project I am afraid.

As to why we are getting a segmentation fault rather than an error message … I’ll look into that.

9

But both the input and reference data come from the example images which are indeed H&E-stained histology images, so one would think H&E staining could be detected at levels adequate for the program even for relatively small patches?

Cheers,
Jon

Jon R. Sauer
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009

10

We’ve lost the diagnostic message the is supposed to appear, so I am not 100% confident, but it sure looks like those patches are being rejected for inadequate levels of hematoxylin. Yes, there surely is traces of blue stain in there, but if it doesn’t rise noticeably above the level of pixel-to-pixel noise variation, that may not be enough.

11

Hi Lee,

Of course, the reason I want to use H&E color normalization is that some new data comes in differently and almost poorly stained relative to my CNN training data. Knowing what the problem might be, I can find ways around it think. Namely, apply CN to a larger area which on average might not be rejected, make the slices down to the training sizes from the larger area.

Thanks.

Cheers,
Jon

Jon R. Sauer.
jon.sauer@gmail.com
Acton, MA, USA 01720
+1 303.579.3009