How to assert that images are equal, in Python unit test?

Engineering
1

How would you check that the “actual” output image of a filter is equal to the “excepted” image, in a Python unit test? Is there a helper function, like assert_equal_image(actual_image, expected_image)? I tried the following:

class MyTestCase(unittest.TestCase):

    def _assert_equal_image(self, actual_image, expected_image) -> None:
        """Asserts that the actual image is equal to the expected one."""
        self.assertTrue(actual_image.IsSameImageGeometryAs(expected_image))
        self.assertEqual(
            actual_image.GetBufferedRegion(), expected_image.GetBufferedRegion()
        )
        region = expected_image.GetBufferedRegion()
        region_index = region.GetIndex()
        region_size = region.GetSize()

        for pixel_index in itertools.product(
            range(region_index[0], region_index[0] + region_size[0]),
            range(region_index[1], region_index[1] + region_size[1]),
        ):
            self.assertEqual(
                actual_image.GetPixel(pixel_index), expected_image.GetPixel(pixel_index)
            )

My attempt may be nice (hopefully), but it isn’t complete (no 3D support yet, for example), and I’m afraid of re-inventing the wheel… any suggestion?

2

You can convert image to numpy array (itk.array_view_from_image) and use np.testing.assert_array_equal

2 Likes
3

Thank you very much @phcerdan So something like this, right?

def _assert_equal_image(self, actual_image, expected_image) -> None:
    """Asserts that the actual image is equal to the expected one."""
    self.assertTrue(actual_image.IsSameImageGeometryAs(expected_image))
    np.testing.assert_array_equal(
        actual=itk.array_view_from_image(actual_image),
        desired=itk.array_view_from_image(expected_image),
        strict=True,
    )
3 Likes
4

I generally end up having to use assert_approx_equal for all but trivial operations. Also do you need to check that the images are the same type?

1 Like
5

Thanks for the suggestion. Here is the use case that I have in mind: ENH: Python dispatch on the first RequiredInputName by thewtex · Pull Request #4921 · InsightSoftwareConsortium/ITK · GitHub As follows:

    assert_equal_image(
        itk.median_image_filter(primary=image),
        itk.median_image_filter(image),
    )

So it would test that in Python, a keyword argument (primary=image) would give the same result as a positional argument (image).

The unit test may use a synthetic input image, so that both function calls would produce exactly the same image.

1 Like
6

Hi,

A more complete way to compare images is with itk.comparison_image_filter:

comparison = itk.comparison_image_filter(
    itk.median_image_filter(primary=image),
    itk.median_image_filter(image),
    verify_input_information=True
)
assert np.sum(comparison) == 0.0

This will also check the image metadata.

For general usage, another option for this purpose is the compare_images function from the itkwasm-compare-images package.

This looks like:

from itkwasm_compare_images import compare_images
from itkwasm import Image

test_image = itk.median_image_filter(primary=image)
# Convert itk.Image to itkwasm.Image
test_image = Image(**itk.dict_from_image(test_image))

baseline_image = itk.median_image_filter(image)
baseline_image = Image(**itk.dict_from_image(baseline_image))

metrics, difference_image, difference_image_rendering = compare_images(
    test_image,
    baseline_images=[baseline_image,],
)
assert metrics['almostEqual']
3 Likes
7

Cool! Thanks Matt! Can you please explain, what is comparison? Is it also an image?

8

comparison = ...

it is a variable :smiley:

9

Aha, yes, I think it is an image where each pixel contains the difference between the two input images.

10

Thanks @dzenanz Would comparison just be the difference, or the absolute difference of the two images?

If itk.comparison_image_filter creates a new image, I guess it’s more expensive than calling np.testing.assert_array_equal on the views returned by itk.array_view_from_image (as was suggested by @phcerdan), right?

1 Like
11

comparison is an image with the signed difference between pixels.

There is a cost to create the difference image, but this is not prohibitive in general.

And it tests more than the pixel arrays.

There are also more options, if needed.

1 Like
12

Thanks for explaining, @matt.mccormick.

OK, so then np.sum(comparison) may have summarized positive and negative pixel differences together, when it returns zero?

13

Thanks for the link! I see, the documentation does not say that the image created by ComparisonImageFilter has absolute values. While this is essential if you want to do np.sum(comparison).

14

If all you need is exact comparison then you may want to look at the HashImageFilter. This filter generates an MD5 hash from the image’s buffer which can be used for efficient comparison.

2 Likes
15

Nice suggestion, thank Bradley. I guess such hash values would especially be useful when having large test images.

For unit tests, I would prefer to use small images, when possible, having around 16x16 pixels, for example. So for such small images, a simple straightforward “assert_array_equal” on the pixel data should be just fine, in my opinion.

I think it would be nice if itk.Image would support value-based equality comparison by image1 == image2. Just like Python lists:

list1 = [1, 2, 3]
list2 = [1, 2, 3]
assert list1 == list2  # OK, list1 == list2 is True :-)

What do you think?

16

There is ambiguity on if these comparison operators should be “broadcast” per pixel, or comparison on the whole image. Numpy has the following:


In [2]: a = np.array(range(4)); b = np.array(range(4))

In [3]: a == b
Out[3]: array([ True,  True,  True,  True])

SimpleITK has these comparison operators overloaded as well. You can experiment with them there as well. In C++ they are an “optional” header, and work well with the move operator too.

2 Likes
17

Yes, this is technically possible.

18

And, if it is a concern, np.sum(np.abs(comparison)) can also be used.

19

Looking at ComparisonImageFilter::ThreadedGenerateData:

Maybe it does take the absolute value of the pixel-wise differences. But if it does, apparently it’s an undocumented feature :person_shrugging:

20

The ComparisonImageFilter class also has the following methods to get the Min, Max, Mean, and Total difference.
https://itk.org/Doxygen/html/classitk_1_1Testing_1_1ComparisonImageFilter.html#a49803d4d6781b90f19aa21f5c0d171fe

So you may not need to use numpy to compute the sum.

1 Like