Need guidance to address the regression error ?

As a beginner I take up an issue, where I need to perform regression testing.

In order to understand the code, I had read the ItkSoftwareGuide pdf, google about it, watch lectures on youtube about regression testing, asked people on linkedin through direct messages.
I understand what is regression testing? when we perform it?

But I got stuck on the question : to perform it ?
2. do I need any tool like selerium?
Because everywhere showing what is testing cycle etc Just theoretical stuffs. But not how to run the test.

If you can suggest some resources or any piece of advice, it will be helpful.

Thank you :slight_smile:

selenium is used for browser-testing the code of web applications.

For testing ITK’s code, we use CTest (docs). Here is a somewhat outdated description.

Are you looking to add a new test to ITK, or to start adding tests to your application?

1 Like

yes I am looking to add a new regression test to ITK.

You should read Tests section of the software guide, and possibly skim the rest of that chapter.

For concrete examples, you could take a look at tests of one of ITK’s built-in modules, such as ITKSmoothing, or one of remote modules, such as ITKCuberille. CuberilleTest04.cxx is on the simpler side.

1 Like

Really I am still clueless.
Somebody changed the instance variable value to 100.
so, I am trying to test is it showing any error. But my knowledge lags behind with few question :-

  1. where should I write the code add_test(...) in visual studio/ notepad?
  2. how I get to know my tests going right ?
  3. do I need the knowledge of root directory ? etc.

I want to learn something from the issue. At minimum performing regression testing.
Thank you.

add_test should go into CMakeLists.txt, see example. Here is an example of quantitative checking.

To execute the test, run ctest in ITK build directory. Example invocation: ctest -VV -C Debug -R itkMultiLabelSTAPLEImageFilterTest. This will print the actual command line, which you can then use in your IDE (Visual Studio). Sample output:

test 3
    Start 3: itkOMEZarrNGFFImageIOReadConvertTestCT1

3: Test command: C:\Dev\ITKIOOMEZarrNGFF-vs19\bin\Debug\IOOMEZarrNGFFTestDriver.exe "--compare" "C:/Dev/ITKIOOMEZarrNGFF-vs19/ExternalData/test/Input/cthead1.mha" "C:/Dev/ITKIOOMEZarrNGFF-vs19/Testing/Temporary/cthead1.zarr" "itkOMEZarrNGFFImageIOTest" "C:/Dev/ITKIOOMEZarrNGFF-vs19/ExternalData/test/Input/cthead1.mha" "C:/Dev/ITKIOOMEZarrNGFF-vs19/Testing/Temporary/cthead1.zarr"
3: Working Directory: C:/Dev/ITKIOOMEZarrNGFF-vs19/test

And how it translates into Visual Studio debugging setup:

Then you can use the usual debugging process: run to, breakpoints, watches etc.

1 Like

Happy new year to all :smile:.
Ok I checked both the files itkMultiLabelSTAPLEImageFilterTest.cxx and the itkLabelVotingImageFilterTest.cxx for quantitative checking. They are almost same.
Now, as the issue states that :
" a regression error was introduced by setting the m_MaximumNumberOfIterations instance Value to 0."

But the Value of MaximumNumberOfIterations is changed to 100 from 0.

Does this means that the issue was solved? or I need to perform certain kind of tasks?

How to verify the output values of the confusion matrices ? and compare them to check that no regressions are introduced ? since I don’t have any past data of matrices to compare

thank you for your kindness :slight_smile:

The usual way is to establish a baseline from current numbers. That is practically done by expecting all zeroes, and the test failure will tell you what are the actual numbers produced. Then put those numbers as expected.

1 Like

I ran the test using ctest -R itkMultiLabelSTAPLEImageFilterTest -VV -C Debug. And it provides some output but the test was passed with MaximumNumberOfIteration = 0.
and showing ITK ERROR with m_PriorProbabilities array.

what should I do now because no test failure its showing ?

Thanks you :slight_smile:

This is expected, testing that the class throws an error in some conditions:

2073: Trying filter->Update()
2073: Caught expected exception
2073: itk::ExceptionObject (00000064534FE6F8)
2073: Location: "void __cdecl itk::MultiLabelSTAPLEImageFilter<class itk::Image<unsigned int,3>,class itk::Image<unsigned int,3>,float>::InitializePriorProbabilities(void)"
2073: File: C:\Dev\ITK-git\Modules\Segmentation\LabelVoting\include\itkMultiLabelSTAPLEImageFilter.hxx
2073: Line: 185
2073: Description: ITK ERROR: MultiLabelSTAPLEImageFilter(000001AFB44A0DA0): m_PriorProbabilities array has wrong size [0]; should be at least 9

Look at how combinationABC is quantitatively checked here:

The ground truth is provided here:

What that issue was suggesting to add similar quantitative comparison for the 3 matrices and PriorProbabilities.

1 Like

Hoping you doing well. :innocent: :slightly_smiling_face:
I want develop some clarity, in order to find the value of maximumNumberOfIteration

STEP 1 : I need to perform the ctest and test failure tell me what is the value of maximumNumberOfIteration.

STEP 2 : Assign the value to maximumNumberOfIteration

Now the challenge is, my test was passed with maximumNumberOfIteration = 0 instead of failed test. As shown before. So I tried hit & trial methodology to find the solution :sweat_smile: but I knew its not right and nor an efficient approach.

So, how can I find what should be the original value of maximumNumberOfIteration?
Am I applying a right approach to fix the issue or I am missing some concepts?

Thank you for your precious time :slight_smile:

This works for outputs, not inputs!

The main point of Make the `itkMultiLabelSTAPLEImageFilterTest` quantitative · Issue #3657 · InsightSoftwareConsortium/ITK · GitHub is to regression-check the outputs, so if an input (maximumNumberOfIteration) changes from 100 to 0 the change will be detected via test errors.

1 Like

Good morning/afternoon/evening/night :smiley:

1.I tried the test syntax like :
itk_add_test(NAME itkMulti... COMMAND ITKLabelV... itkMulti... )
by looking at the “Test command”. But it doesn’t work.
2. Then checkout this approach

  • if I create a content link of baseline image from kitware, does it can check MaximumNumberOfIterator?
  • should I go with python to build the image with given matrices?
  • is the file type is correct or I need to convert them into .mha/.png?
  • what 5 100 0 at the end of ${ITK_TEST_OUTPUT_DIR} how they are decided?
  • should I try out other macros likeITK_TRY_EXCEPT_NO_EXCEPTION() or something else & put if MaximumNumberOfIterator = 0 give error.

thanks :v:

The test should have argc/argv parameters which get translated into internal parameters which are currently constants, such as maximumNumberOfIterations.

To get an output image which can be regression tested against a baseline, it needs to be written first. But that is not really necessary in this case, as the inputs are constant, and the outputs are “manually” checked.

I would like @jhlegarreta to pitch in here.

1 Like