Testing with ctest fails when testing ITK5.2 with VTK9.2 with the itkVkGlue module in MSVC release mode

Engineering
1

All tests passed but those:
The following tests FAILED:
2538 - itkImageToVTKImageFilterTest (Failed)
2540 - itkVTKImageToImageFilterTest (Failed)

Trace shows very similar root causes for the failing tests:

  Debug: Off
  Modified Time: 237
  Reference Count: 1
  Registered Events: (none)
  Information: 000001C5223E74D0
  Data Released: False
  Global Release Data: Off
  UpdateTime: 192
  Field Data:
    Debug: Off
    Modified Time: 147
    Reference Count: 1
    Registered Events: (none)
    Number Of Arrays: 0
    Number Of Components: 0
    Number Of Tuples: 0
  Number Of Points: 441
  Number Of Cells: 400
  Cell Data:
    Debug: Off
    Modified Time: 155
    Reference Count: 1
    Registered Events: 
      Registered Observers:
        vtkObserver (000001C5223EB220)
          Event: 33
          EventName: ModifiedEvent
          Command: 000001C52223C2E0
          Priority: 0
          Tag: 1
    Number Of Arrays: 0
    Number Of Components: 0
    Number Of Tuples: 0
    Copy Tuple Flags: ( 1 1 1 1 1 0 1 1 1 1 1 )
    Interpolate Flags: ( 1 1 1 1 1 0 0 1 1 1 1 )
    Pass Through Flags: ( 1 1 1 1 1 1 1 1 1 1 1 )
    Scalars: (none)
    Vectors: (none)
    Normals: (none)
    TCoords: (none)
    Tensors: (none)
    GlobalIds: (none)
    PedigreeIds: (none)
    EdgeFlag: (none)
    Tangents: (none)
    RationalWeights: (none)
    HigherOrderDegrees: (none)
  Point Data:
    Debug: Off
    Modified Time: 191
    Reference Count: 1
    Registered Events: 
      Registered Observers:
        vtkObserver (000001C5223EB5E0)
          Event: 33
          EventName: ModifiedEvent
          Command: 000001C52223C2E0
          Priority: 0
          Tag: 1
    Number Of Arrays: 1
    Array 0 name = ImageScalars
    Number Of Components: 1
    Number Of Tuples: 441
    Copy Tuple Flags: ( 1 1 1 1 1 0 1 1 1 1 1 )
    Interpolate Flags: ( 1 1 1 1 1 0 0 1 1 1 1 )
    Pass Through Flags: ( 1 1 1 1 1 1 1 1 1 1 1 )
    Scalars: 
      Debug: Off
      Modified Time: 188
      Reference Count: 1
      Registered Events: (none)
      Name: ImageScalars
      Data type: double
      Size: 441
      MaxId: 440
      NumberOfComponents: 1
      Information: 0000000000000000
      Name: ImageScalars
      Number Of Components: 1
      Number Of Tuples: 441
      Size: 441
      MaxId: 440
      LookupTable: (none)
    Vectors: (none)
    Normals: (none)
    TCoords: (none)
    Tensors: (none)
    GlobalIds: (none)
    PedigreeIds: (none)
    EdgeFlag: (none)
    Tangents: (none)
    RationalWeights: (none)
    HigherOrderDegrees: (none)
  Bounds: 
    Xmin,Xmax: (-0.1, 39.9)
    Ymin,Ymax: (-12, -10)
    Zmin,Zmax: (0, 0)
  Compute Time: 238
  Spacing: (0.1, 2, 0)
  Origin: (-0.1, -10, 0)
  Direction: (0, 1, 0, -1, 0, 0, 0, 0, 1)
  Dimensions: (21, 21, 1)
  Increments: (0, 0, 0)
  Extent: (0, 20, 0, 20, 0, 0)

Image (000001C52242C820)
  RTTI typeinfo:   class itk::Image<double,2>
  Reference Count: 1
  Modified Time: 68
  Debug: Off
  Object Name: 
  Observers: 
    none
  Source: (000001C52014EC40) 
  Source output name: Primary
  Release Data: Off
  Data Released: False
  Global Release Data: Off
  PipelineMTime: 59
  UpdateMTime: 69
  RealTimeStamp: 0 seconds 
  LargestPossibleRegion: 
    Dimension: 2
    Index: [0, 0]
    Size: [21, 21]
  BufferedRegion: 
    Dimension: 2
    Index: [0, 0]
    Size: [21, 21]
  RequestedRegion: 
    Dimension: 2
    Index: [0, 0]
    Size: [21, 21]
  Spacing: [0.1, 2]
  Origin: [-0.1, -10]
  Direction: 
0 1
-1 0

  IndexToPointMatrix: 
0 2
-0.1 0

  PointToIndexMatrix: 
0 -10
0.5 0

  Inverse Direction: 
0 -1
1 0

  PixelContainer: 
    ImportImageContainer (000001C520168060)
      RTTI typeinfo:   class itk::ImportImageContainer<unsigned __int64,double>
      Reference Count: 1
      Modified Time: 67
      Debug: Off
      Object Name: 
      Observers: 
        none
      Pointer: 000001C5224166E0
      Container manages memory: false
      Size: 441
      Capacity: 441
Error: directions do not match for component (0,0).

If I remove Module_ItkVtkGlue from the build the testing is fine.

Any idea what’s wrong with the testing?

Exact ITK and VTK versions are respectively: 5.2.1 and 9.2.2
Also the failing code in tests is:

if VTK_MAJOR_VERSION >= 9 || (VTK_MAJOR_VERSION == 8 && VTK_MINOR_VERSION >= 90)
    for (int j = 0; j < dim; ++j)
    {
      if (input->GetDirection()[i][j] != output->GetDirectionMatrix()->GetData()[i * 3 + j])
      {
        std::cerr << "Error: directions do not match for component (" << i << "," << j << ")." << std::endl;
        return EXIT_FAILURE;
      }
    }
#endif

CMakeCache.txt (200.9 KB)

2

Extra information: I am using VS2022 17.2.11 (LTSC version).
When i copy paste the test code in a hacky startup code of mine to be able to debug, the test passes fine so it seems to happen only when running cdash on my itk build.

3

@matt.mccormick might shed some light on it.

4

Does this occur with older versions of VTK?

5

I will test with latest git trunk too, but I can tell you that direction components are indeed different I added logs for the input and ouput content for index (0,0) it is a round error and the values are very close in one case (about 1E-323 close!)
so i changed the code to more robust floating point comparison like this:

      if (fabs(output->GetDirection()[i][j] - input->GetDirectionMatrix()->GetData()[i * 3 + j]) > 1E-15 )
      {
        std::cerr << "Error: directions do not match for component (" << i << "," << j << ")." << std::endl;
		std::cerr << input->GetDirection()[i][j]  << " != " << output->GetDirectionMatrix()->GetData()[i * 3 + j] << std::endl;
        return EXIT_FAILURE;
      }

Then interestingly i still got error on the next index but this time my code show the conent diff so I got:
Error: directions do not match for component (0,1).
1 != 0

Which is not a rounding difference and therefore concerns me a bit…

2 Likes
6

I modified the code upper in the two tests function to not stop after one component failure,

and now for the itkImageToVTKImageFilterTest test, it looks like components are swapped:

Error: directions do not match for component (0,1).
1 != 0
Error: directions do not match for component (0,2).
0 != 1
Error: directions do not match for component (1,0).
-1 != 0
Error: directions do not match for component (1,1).
0 != -1
Error: directions do not match for component (2,2).
0.7 != 0

And for the reverse test (itkVTKImageToImageFilterTest), I get:

Error: directions do not match for component (0,1).
0 != 1
Error: directions do not match for component (1,0).
0 != -1
Error: directions do not match for component (1,1).
-1 != 0

EDIT: I was not capturing all i indexes yet, newest code does that now.

In all my tests additions I always log input content first, then output content at a given (i,j) index.
itkVTKImageToImageFilterTest.cxx (3.9 KB)
itkImageToVTKImageFilterTest.cxx (3.4 KB)

7

Rebuilt itk in debug mode and ran the tests in debug mode (itk side only), now both tests worked!
It really seems that in release mode, something is optimized and fails the final assertions.
Could it be a smart pointer released too early?

8

@fab672000 thanks for the investigation!

Do you observe the same behavior with VTK 9.0.0?

9

Sorry for not testing that sooner, ok so grabbed vtk9.0.0 from the git tags and rebuilt both vtk and itk again and tested it again,

EDIT: vtk dir was still poining on 9.2 in my CI build, so fixing that now it seems to work on 9.0.0 (one error for test 2542 seems unrelated):

The following tests FAILED:
        2542 - QuickViewTest (Not Run)
ImageFilterTest (Failed)
  Errors while running CTest

itk:
CMakeCache.txt (199.6 KB)

vtk:
CMakeCache.txt (166.9 KB)

10

vtk9.0.0 testing update: Previous test in my CI was still not using git 9.0.0 label version, fixed it and now the bridge tests work, see upper edit.

11

OK so tested my ITK build with various VTK versions and did a binary search that showed the following:

  1. VTK versions 9.0.0, 9.1.0, 9.2.5 & latest night build all pass the bridge tests
  2. vtk 9.2.2 did not pass the tests.

Conclusion: I’ll update our CI builds with latest 9.2.5 soon!

3 Likes