# Testing with ctest fails when testing ITK5.2 with VTK9.2 with the itkVkGlue module in MSVC release mode

**URL:** <https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622>\
**Category:** Engineering\
**Tags:** cdash\
**Created:** [January 11, 2023, 9:18pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622 "2023-01-11T21:18:59Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![fab672000](https://discourse.itk.org/letter_avatar_proxy/v4/letter/f/e8c25b/32.png) [@fab672000](https://discourse.itk.org/u/fab672000)\
**Post date:** [January 11, 2023, 9:18pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/1 "2023-01-11T21:18:59Z")

</div>

All tests passed but those:  
The following tests FAILED:  
2538 - itkImageToVTKImageFilterTest (Failed)  
2540 - itkVTKImageToImageFilterTest (Failed)

Trace shows very similar root causes for the failing tests:

```auto
  Debug: Off
  Modified Time: 237
  Reference Count: 1
  Registered Events: (none)
  Information: 000001C5223E74D0
  Data Released: False
  Global Release Data: Off
  UpdateTime: 192
  Field Data:
    Debug: Off
    Modified Time: 147
    Reference Count: 1
    Registered Events: (none)
    Number Of Arrays: 0
    Number Of Components: 0
    Number Of Tuples: 0
  Number Of Points: 441
  Number Of Cells: 400
  Cell Data:
    Debug: Off
    Modified Time: 155
    Reference Count: 1
    Registered Events: 
      Registered Observers:
        vtkObserver (000001C5223EB220)
          Event: 33
          EventName: ModifiedEvent
          Command: 000001C52223C2E0
          Priority: 0
          Tag: 1
    Number Of Arrays: 0
    Number Of Components: 0
    Number Of Tuples: 0
    Copy Tuple Flags: ( 1 1 1 1 1 0 1 1 1 1 1 )
    Interpolate Flags: ( 1 1 1 1 1 0 0 1 1 1 1 )
    Pass Through Flags: ( 1 1 1 1 1 1 1 1 1 1 1 )
    Scalars: (none)
    Vectors: (none)
    Normals: (none)
    TCoords: (none)
    Tensors: (none)
    GlobalIds: (none)
    PedigreeIds: (none)
    EdgeFlag: (none)
    Tangents: (none)
    RationalWeights: (none)
    HigherOrderDegrees: (none)
  Point Data:
    Debug: Off
    Modified Time: 191
    Reference Count: 1
    Registered Events: 
      Registered Observers:
        vtkObserver (000001C5223EB5E0)
          Event: 33
          EventName: ModifiedEvent
          Command: 000001C52223C2E0
          Priority: 0
          Tag: 1
    Number Of Arrays: 1
    Array 0 name = ImageScalars
    Number Of Components: 1
    Number Of Tuples: 441
    Copy Tuple Flags: ( 1 1 1 1 1 0 1 1 1 1 1 )
    Interpolate Flags: ( 1 1 1 1 1 0 0 1 1 1 1 )
    Pass Through Flags: ( 1 1 1 1 1 1 1 1 1 1 1 )
    Scalars: 
      Debug: Off
      Modified Time: 188
      Reference Count: 1
      Registered Events: (none)
      Name: ImageScalars
      Data type: double
      Size: 441
      MaxId: 440
      NumberOfComponents: 1
      Information: 0000000000000000
      Name: ImageScalars
      Number Of Components: 1
      Number Of Tuples: 441
      Size: 441
      MaxId: 440
      LookupTable: (none)
    Vectors: (none)
    Normals: (none)
    TCoords: (none)
    Tensors: (none)
    GlobalIds: (none)
    PedigreeIds: (none)
    EdgeFlag: (none)
    Tangents: (none)
    RationalWeights: (none)
    HigherOrderDegrees: (none)
  Bounds: 
    Xmin,Xmax: (-0.1, 39.9)
    Ymin,Ymax: (-12, -10)
    Zmin,Zmax: (0, 0)
  Compute Time: 238
  Spacing: (0.1, 2, 0)
  Origin: (-0.1, -10, 0)
  Direction: (0, 1, 0, -1, 0, 0, 0, 0, 1)
  Dimensions: (21, 21, 1)
  Increments: (0, 0, 0)
  Extent: (0, 20, 0, 20, 0, 0)

Image (000001C52242C820)
  RTTI typeinfo: class itk::Image&lt;double,2&gt;
  Reference Count: 1
  Modified Time: 68
  Debug: Off
  Object Name: 
  Observers: 
    none
  Source: (000001C52014EC40) 
  Source output name: Primary
  Release Data: Off
  Data Released: False
  Global Release Data: Off
  PipelineMTime: 59
  UpdateMTime: 69
  RealTimeStamp: 0 seconds 
  LargestPossibleRegion: 
    Dimension: 2
    Index: [0, 0]
    Size: [21, 21]
  BufferedRegion: 
    Dimension: 2
    Index: [0, 0]
    Size: [21, 21]
  RequestedRegion: 
    Dimension: 2
    Index: [0, 0]
    Size: [21, 21]
  Spacing: [0.1, 2]
  Origin: [-0.1, -10]
  Direction: 
0 1
-1 0

  IndexToPointMatrix: 
0 2
-0.1 0

  PointToIndexMatrix: 
0 -10
0.5 0

  Inverse Direction: 
0 -1
1 0

  PixelContainer: 
    ImportImageContainer (000001C520168060)
      RTTI typeinfo: class itk::ImportImageContainer&lt;unsigned __int64,double&gt;
      Reference Count: 1
      Modified Time: 67
      Debug: Off
      Object Name: 
      Observers: 
        none
      Pointer: 000001C5224166E0
      Container manages memory: false
      Size: 441
      Capacity: 441
Error: directions do not match for component (0,0).

```

If I remove Module\_ItkVtkGlue from the build the testing is fine.

Any idea what’s wrong with the testing?

Exact ITK and VTK versions are respectively: 5.2.1 and 9.2.2  
Also the failing code in tests is:

```auto
if VTK_MAJOR_VERSION >= 9 || (VTK_MAJOR_VERSION == 8 && VTK_MINOR_VERSION >= 90)
    for (int j = 0; j < dim; ++j)
    {
      if (input->GetDirection()[i][j] != output->GetDirectionMatrix()->GetData()[i * 3 + j])
      {
        std::cerr << "Error: directions do not match for component (" << i << "," << j << ")." << std::endl;
        return EXIT_FAILURE;
      }
    }
#endif

```

[CMakeCache.txt](https://discourse.itk.org/uploads/short-url/zBOLY70My1bn5a4tFJqglJjRBmo.txt) (200.9 KB)

---

<div class="post-metadata">

**Author:** ![fab672000](https://discourse.itk.org/letter_avatar_proxy/v4/letter/f/e8c25b/32.png) [@fab672000](https://discourse.itk.org/u/fab672000)\
**Post date:** [January 11, 2023, 10:17pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/2 "2023-01-11T22:17:09Z")

</div>

Extra information: I am using VS2022 17.2.11 (LTSC version).  
When i copy paste the test code in a hacky startup code of mine to be able to debug, the test passes fine so it seems to happen only when running cdash on my itk build.

---

<div class="post-metadata">

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\
**Post date:** [January 12, 2023, 4:13pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/3 "2023-01-12T16:13:39Z")

</div>

@matt.mccormick might shed some light on it.

---

<div class="post-metadata">

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\
**Post date:** [January 12, 2023, 4:34pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/4 "2023-01-12T16:34:52Z")

</div>

Does this occur with older versions of VTK?

---

<div class="post-metadata">

**Author:** ![fab672000](https://discourse.itk.org/letter_avatar_proxy/v4/letter/f/e8c25b/32.png) [@fab672000](https://discourse.itk.org/u/fab672000)\
**Post date:** [January 13, 2023, 6:38pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/5 "2023-01-13T18:38:57Z")

</div>

I will test with latest git trunk too, but I can tell you that direction components are indeed different I added logs for the input and ouput content for index (0,0) it is a round error and the values are very close in one case (about 1E-323 close!)  
so i changed the code to more robust floating point comparison like this:

```auto
      if (fabs(output->GetDirection()[i][j] - input->GetDirectionMatrix()->GetData()[i * 3 + j]) > 1E-15 )
      {
        std::cerr << "Error: directions do not match for component (" << i << "," << j << ")." << std::endl;
		std::cerr << input->GetDirection()[i][j] << " != " << output->GetDirectionMatrix()->GetData()[i * 3 + j] << std::endl;
        return EXIT_FAILURE;
      }

```

Then interestingly i still got error on the next index but this time my code show the conent diff so I got:  
Error: directions do not match for component (0,1).  
1 != 0

Which is not a rounding difference and therefore concerns me a bit…

---

<div class="post-metadata">

**Author:** ![fab672000](https://discourse.itk.org/letter_avatar_proxy/v4/letter/f/e8c25b/32.png) [@fab672000](https://discourse.itk.org/u/fab672000)\
**Post date:** [January 13, 2023, 7:45pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/6 "2023-01-13T19:45:26Z")

</div>

I modified the code upper in the two tests function to not stop after one component failure,

and now for the itkImageToVTKImageFilterTest test, it looks like components are swapped:

```auto
Error: directions do not match for component (0,1).
1 != 0
Error: directions do not match for component (0,2).
0 != 1
Error: directions do not match for component (1,0).
-1 != 0
Error: directions do not match for component (1,1).
0 != -1
Error: directions do not match for component (2,2).
0.7 != 0

```

And for the reverse test (itkVTKImageToImageFilterTest), I get:

```auto
Error: directions do not match for component (0,1).
0 != 1
Error: directions do not match for component (1,0).
0 != -1
Error: directions do not match for component (1,1).
-1 != 0

```

EDIT: I was not capturing all i indexes yet, newest code does that now.

In all my tests additions I always log input content first, then output content at a given (i,j) index.  
[itkVTKImageToImageFilterTest.cxx](https://discourse.itk.org/uploads/short-url/wS5AJcfpLwPanCDQQC3ZuATb12s.cxx) (3.9 KB)  
[itkImageToVTKImageFilterTest.cxx](https://discourse.itk.org/uploads/short-url/46HGwS5dag1b2acyVsSNqLTVTIN.cxx) (3.4 KB)

---

<div class="post-metadata">

**Author:** ![fab672000](https://discourse.itk.org/letter_avatar_proxy/v4/letter/f/e8c25b/32.png) [@fab672000](https://discourse.itk.org/u/fab672000)\
**Post date:** [January 13, 2023, 9:05pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/7 "2023-01-13T21:05:38Z")

</div>

Rebuilt itk in debug mode and ran the tests in debug mode (itk side only), now both tests worked!  
It really seems that in release mode, something is optimized and fails the final assertions.  
Could it be a smart pointer released too early?

---

<div class="post-metadata">

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\
**Post date:** [January 16, 2023, 3:39am UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/8 "2023-01-16T03:39:59Z")

</div>

@fab672000 thanks for the investigation!

Do you observe the same behavior with VTK 9.0.0?

---

<div class="post-metadata">

**Author:** ![fab672000](https://discourse.itk.org/letter_avatar_proxy/v4/letter/f/e8c25b/32.png) [@fab672000](https://discourse.itk.org/u/fab672000)\
**Post date:** [January 16, 2023, 3:40pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/9 "2023-01-16T15:40:43Z")

</div>

Sorry for not testing that sooner, ok so grabbed vtk9.0.0 from the git tags and rebuilt both vtk and itk again and tested it again,

EDIT: vtk dir was still poining on 9.2 in my CI build, so fixing that now it seems to work on 9.0.0 (one error for test 2542 seems unrelated):

```auto
The following tests FAILED:
        2542 - QuickViewTest (Not Run)
ImageFilterTest (Failed)
  Errors while running CTest

```

itk:  
[CMakeCache.txt](https://discourse.itk.org/uploads/short-url/oXIiifeUSGQfwccdwZOUQUGmbs5.txt) (199.6 KB)

vtk:  
[CMakeCache.txt](https://discourse.itk.org/uploads/short-url/zSzZnFVAAmWrxKChrjwkAFg7Y2K.txt) (166.9 KB)

---

<div class="post-metadata">

**Author:** ![fab672000](https://discourse.itk.org/letter_avatar_proxy/v4/letter/f/e8c25b/32.png) [@fab672000](https://discourse.itk.org/u/fab672000)\
**Post date:** [January 16, 2023, 4:17pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/10 "2023-01-16T16:17:56Z")

</div>

vtk9.0.0 testing update: Previous test in my CI was still not using git 9.0.0 label version, fixed it and now the bridge tests work, see upper edit.

---

<div class="post-metadata">

**Author:** ![fab672000](https://discourse.itk.org/letter_avatar_proxy/v4/letter/f/e8c25b/32.png) [@fab672000](https://discourse.itk.org/u/fab672000)\
**Post date:** [January 16, 2023, 7:30pm UTC](https://discourse.itk.org/t/testing-with-ctest-fails-when-testing-itk5-2-with-vtk9-2-with-the-itkvkglue-module-in-msvc-release-mode/5622/11 "2023-01-16T19:30:56Z")

</div>

OK so tested my ITK build with various VTK versions and did a binary search that showed the following:

1. VTK versions 9.0.0, 9.1.0, 9.2.5 & latest night build all pass the bridge tests
2. vtk 9.2.2 did not pass the tests.

Conclusion: I’ll update our CI builds with latest 9.2.5 soon!
