Need guidance to address the regression error ?

parth721 · December 13, 2022, 12:45pm

As a beginner I take up an issue, where I need to perform regression testing.

In order to understand the code, I had read the ItkSoftwareGuide pdf, google about it, watch lectures on youtube about regression testing, asked people on linkedin through direct messages.
I understand what is regression testing? when we perform it?

But I got stuck on the question :
1.how to perform it ?
2. do I need any tool like selerium?
Because everywhere showing what is testing cycle etc Just theoretical stuffs. But not how to run the test.

If you can suggest some resources or any piece of advice, it will be helpful.

Thank you

dzenanz · December 13, 2022, 4:07pm

selenium is used for browser-testing the code of web applications.

For testing ITK’s code, we use CTest (docs). Here is a somewhat outdated description.

Are you looking to add a new test to ITK, or to start adding tests to your application?

parth721 · December 14, 2022, 11:06am

yes I am looking to add a new regression test to ITK.

dzenanz · December 14, 2022, 2:19pm

You should read Tests section of the software guide, and possibly skim the rest of that chapter.

For concrete examples, you could take a look at tests of one of ITK’s built-in modules, such as ITKSmoothing, or one of remote modules, such as ITKCuberille. CuberilleTest04.cxx is on the simpler side.

parth721 · December 26, 2022, 1:19am

Really I am still clueless.
Somebody changed the instance variable value to 100.
so, I am trying to test is it showing any error. But my knowledge lags behind with few question :-

where should I write the code add_test(...) in visual studio/ notepad?
how I get to know my tests going right ?
do I need the knowledge of root directory ? etc.

I want to learn something from the issue. At minimum performing regression testing.
Thank you.

dzenanz · December 27, 2022, 5:49pm

add_test should go into CMakeLists.txt, see example. Here is an example of quantitative checking.

To execute the test, run ctest in ITK build directory. Example invocation: ctest -VV -C Debug -R itkMultiLabelSTAPLEImageFilterTest. This will print the actual command line, which you can then use in your IDE (Visual Studio). Sample output:

test 3
    Start 3: itkOMEZarrNGFFImageIOReadConvertTestCT1

3: Test command: C:\Dev\ITKIOOMEZarrNGFF-vs19\bin\Debug\IOOMEZarrNGFFTestDriver.exe "--compare" "C:/Dev/ITKIOOMEZarrNGFF-vs19/ExternalData/test/Input/cthead1.mha" "C:/Dev/ITKIOOMEZarrNGFF-vs19/Testing/Temporary/cthead1.zarr" "itkOMEZarrNGFFImageIOTest" "C:/Dev/ITKIOOMEZarrNGFF-vs19/ExternalData/test/Input/cthead1.mha" "C:/Dev/ITKIOOMEZarrNGFF-vs19/Testing/Temporary/cthead1.zarr"
3: Working Directory: C:/Dev/ITKIOOMEZarrNGFF-vs19/test

And how it translates into Visual Studio debugging setup:

Then you can use the usual debugging process: run to, breakpoints, watches etc.

parth721 · January 1, 2023, 4:37pm

Happy new year to all .
Ok I checked both the files itkMultiLabelSTAPLEImageFilterTest.cxx and the itkLabelVotingImageFilterTest.cxx for quantitative checking. They are almost same.
Now, as the issue states that :
" a regression error was introduced by setting the m_MaximumNumberOfIterations instance Value to 0."

But the Value of MaximumNumberOfIterations is changed to 100 from 0.

Does this means that the issue was solved? or I need to perform certain kind of tasks?

parth721 · January 1, 2023, 5:06pm

How to verify the output values of the confusion matrices ? and compare them to check that no regressions are introduced ? since I don’t have any past data of matrices to compare

thank you for your kindness

dzenanz · January 2, 2023, 4:35pm

The usual way is to establish a baseline from current numbers. That is practically done by expecting all zeroes, and the test failure will tell you what are the actual numbers produced. Then put those numbers as expected.

parth721 · January 8, 2023, 2:20pm

I ran the test using ctest -R itkMultiLabelSTAPLEImageFilterTest -VV -C Debug. And it provides some output but the test was passed with MaximumNumberOfIteration = 0.
and showing ITK ERROR with m_PriorProbabilities array.

what should I do now because no test failure its showing ?

Thanks you

dzenanz · January 9, 2023, 5:24pm

This is expected, testing that the class throws an error in some conditions:

2073: Trying filter->Update()
2073: Caught expected exception
2073:
2073: itk::ExceptionObject (00000064534FE6F8)
2073: Location: "void __cdecl itk::MultiLabelSTAPLEImageFilter<class itk::Image<unsigned int,3>,class itk::Image<unsigned int,3>,float>::InitializePriorProbabilities(void)"
2073: File: C:\Dev\ITK-git\Modules\Segmentation\LabelVoting\include\itkMultiLabelSTAPLEImageFilter.hxx
2073: Line: 185
2073: Description: ITK ERROR: MultiLabelSTAPLEImageFilter(000001AFB44A0DA0): m_PriorProbabilities array has wrong size [0]; should be at least 9

Look at how combinationABC is quantitatively checked here:

github.com

InsightSoftwareConsortium/ITK/blob/v5.3.0/Modules/Segmentation/LabelVoting/test/itkMultiLabelSTAPLEImageFilterTest.cxx#L212-L222


      
          // Compare to correct results
          it = IteratorType(outputImage, outputImage->GetBufferedRegion());
          for (unsigned int i = 0; i < 8; ++i, ++it)
          {
            if (combinationABC[i] != it.Get())
            {
              std::cout << "Incorrect result using images A,B,C: i = " << i << ", correct = " << combinationABC[i]
                        << ", got = " << it.Get() << std::endl;
              return EXIT_FAILURE;
            }
          }

The ground truth is provided here:

github.com

InsightSoftwareConsortium/ITK/blob/v5.3.0/Modules/Segmentation/LabelVoting/test/itkMultiLabelSTAPLEImageFilterTest.cxx#L39


      
          
          
// Declare the types of the images
          using ImageType = itk::Image<unsigned int, Dimension>;
          
          
// Input data arrays for test images
          const unsigned int dataImageA[imageSize] = { 0, 1, 3, 3, 4, 6, 6, 0 };
          const unsigned int dataImageB[imageSize] = { 1, 1, 2, 4, 4, 5, 7, 1 };
          const unsigned int dataImageC[imageSize] = { 0, 2, 2, 3, 5, 5, 6, 8 };
          
          
// Correct combinations of input images
          const unsigned int combinationABC[imageSize] = { 0, 1, 2, 3, 4, 5, 6, 9 };
          const unsigned int combinationAB[imageSize] = { 8, 1, 8, 8, 4, 8, 8, 8 };
          const unsigned int combinationABundecided255[imageSize] = { 255, 1, 255, 255, 4, 255, 255, 255 };
          
          
// Declare the type of the index to access images
          using IndexType = itk::Index<Dimension>;
          
          
// Declare the type of the size
          using SizeType = itk::Size<Dimension>;
          
          
// Declare the type of the Region

What that issue was suggesting to add similar quantitative comparison for the 3 matrices and PriorProbabilities.

parth721 · January 14, 2023, 5:07am

Hoping you doing well.
I want develop some clarity, in order to find the value of maximumNumberOfIteration

STEP 1 : I need to perform the ctest and test failure tell me what is the value of maximumNumberOfIteration.

STEP 2 : Assign the value to maximumNumberOfIteration

Now the challenge is, my test was passed with maximumNumberOfIteration = 0 instead of failed test. As shown before. So I tried hit & trial methodology to find the solution but I knew its not right and nor an efficient approach.

So, how can I find what should be the original value of maximumNumberOfIteration?
Am I applying a right approach to fix the issue or I am missing some concepts?

Thank you for your precious time

dzenanz · January 16, 2023, 2:23pm

This works for outputs, not inputs!

The main point of Make the `itkMultiLabelSTAPLEImageFilterTest` quantitative · Issue #3657 · InsightSoftwareConsortium/ITK · GitHub is to regression-check the outputs, so if an input (maximumNumberOfIteration) changes from 100 to 0 the change will be detected via test errors.

parth721 · January 20, 2023, 12:49am

Good morning/afternoon/evening/night

1.I tried the test syntax like :
itk_add_test(NAME itkMulti... COMMAND ITKLabelV... itkMulti... )
by looking at the “Test command”. But it doesn’t work.
2. Then checkout this approach

if I create a content link of baseline image from kitware, does it can check MaximumNumberOfIterator?
should I go with python to build the image with given matrices?
is the file type is correct or I need to convert them into .mha/.png?
what 5 100 0 at the end of ${ITK_TEST_OUTPUT_DIR} how they are decided?
should I try out other macros likeITK_TRY_EXCEPT_NO_EXCEPTION() or something else & put if MaximumNumberOfIterator = 0 give error.

thanks

dzenanz · January 20, 2023, 2:48pm

The test should have argc/argv parameters which get translated into internal parameters which are currently constants, such as maximumNumberOfIterations.

To get an output image which can be regression tested against a baseline, it needs to be written first. But that is not really necessary in this case, as the inputs are constant, and the outputs are “manually” checked.

I would like @jhlegarreta to pitch in here.