Intermittent SIGSEGV in SignedMaurerDistanceMap

Engineering
, ,
1

Hello Everyone.

I am experiencing an intermittent Segmentation Fault while using the itk::simple::SignedMaurerDistanceMap() function.

Environment

SimpleITK 2.4 (ITK 5.4), Ubuntu 24 with intel oneAPI 2025.1 (icpx) compiler.

This happens once or twice in 1000 runs. The crash happens when my application is run with 30 or more threads. For Itk I have set global default number of threads to 1 using itk::simple::ProcessObject::SetGlobalDefaultNumberOfThreads(1). I am very new to itk so I would like to ask if I am doing something wrong here. Thank you in advance for any guidance and ideas.

Please find my function below.

#include <vector>
#include <algorithm>
#include <cstdint>
#include <iostream> 
#include <type_traits>
#include <SimpleITK/Common/SimpleITK.h>

enum class DistanceType {
    Euclidean,
    SquaredEuclidean
};


void computeDistanceMap(const std::vector<float> inputData,
                        size_t rows,
                        size_t cols,
                        std::vector<float>& outputData,
                        DistanceType distanceType)
{

    if (inputData.size() != rows * cols) {
        std::cerr << "Error: Input data size does not match dimensions." << std::endl;
        return; 
    }

    std::vector<std::int16_t> binaryVec(rows * cols, 0);
    for (size_t i = 0; i < inputData.size(); ++i) {
        binaryVec[i] = (inputData[i] > 0) ? 1 : 0;
    }

    // Convert to SimpleITK image object.
    const std::vector<unsigned int> dataDimension = {static_cast<unsigned int>(rows),
                                                     static_cast<unsigned int>(cols)};

    /*
        1. Made deep copy of binaryImage and passed it to SignedMaurerDistanceMap. 
        2. Verified dataDimension 
        It still crashes ocassionally  ( 1 in 1000 runs ) when application is run using
        30 threads. The machine has a total of 32 logical cores.

    */

    const itk::simple::Image binaryImage = itk::simple::ImportAsInt16(binaryVec.data(), dataDimension);

    itk::simple::Image distanceImage;
    if (distanceType == DistanceType::Euclidean) {
        distanceImage = itk::simple::SignedMaurerDistanceMap(binaryImage, false, false);
    } else {
        distanceImage = itk::simple::SignedMaurerDistanceMap(binaryImage, false, true);
    }
    
    size_t totalPixels = rows * cols;
    outputData.resize(totalPixels);

    float *buff = distanceImage.GetBufferAsFloat();
    std::copy(buff, buff + totalPixels, outputData.begin() );
 

}

This is the log from gdb

received signal SIGSEGV, Segmentation fault.

0x00007ffff0f98f9f in itk::SignedMaurerDistanceMapImageFilter<itk::Image<short, 2u>, itk::Image<float, 2u> >::Voronoi(unsigned int, itk::Index<2u>, itk::Image<float, 2u>*) () from /libso/libSimpleITK_ITKDistanceMap-2.4.so.1

#0  0x00007ffff0f98f9f in itk::SignedMaurerDistanceMapImageFilter<itk::Image<short, 2u>, itk::Image<float, 2u> >::Voronoi(unsigned int, itk::Index<2u>, itk::Image<float, 2u>*) () from /libso/test/libSimpleITK_ITKDistanceMap-2.4.so.1

#1  0x00007ffff0f966b5 in itk::SignedMaurerDistanceMapImageFilter<itk::Image<short, 2u>, itk::Image<float, 2u> >::ThreadedGenerateData(itk::ImageRegion<2u> const&, unsigned int) () from /libso/libSimpleITK_ITKDistanceMap-2.4.so.1

#2  0x00007ffff0d7daae in itk::ImageSource<itk::Image<float, 2u> >::ThreaderCallback(void*) () from /libso/libSimpleITK_ITKDistanceMap-2.4.so.1

#3  0x00007fffed184d6d in itk::PlatformMultiThreader::SingleMethodExecute() () from /libso/libITKCommon-5.4.so.1

#4  0x00007ffff0f96372 in itk::SignedMaurerDistanceMapImageFilter<itk::Image<short, 2u>, itk::Image<float, 2u> >::GenerateData() () from /libso/libSimpleITK_ITKDistanceMap-2.4.so.1

#5  0x00007fffed17aee6 in itk::ProcessObject::UpdateOutputData(itk::DataObject*) () from /libso/libITKCommon-5.4.so.1

#6  0x00007ffff0f95a6a in itk::simple::Image itk::simple::SignedMaurerDistanceMapImageFilter::ExecuteInternal<itk::Image<short, 2u> >(itk::simple::Image const&) () from /libso/libSimpleITK_ITKDistanceMap-2.4.so.1

#7  0x00007ffff0f6d013 in std::_Function_handler<itk::simple::Image (itk::simple::Image const&), itk::simple::detail::MemberFunctionFactoryBase<itk::simple::Image (itk::simple::SignedMaurerDistanceMapImageFilter::*)(itk::simple::Image const&), std::pair<unsigned int, int>, 1u>::BindObject(itk::simple::Image (itk::simple::SignedMaurerDistanceMapImageFilter::*)(itk::simple::Image const&), itk::simple::SignedMaurerDistanceMapImageFilter*)::{lambda(auto:1&&)#1}>::_M_invoke(std::_Any_data const&, itk::simple::Image const&) () from /libso/libSimpleITK_ITKDistanceMap-2.4.so.1

#8  0x00007ffff0f6c47e in itk::simple::SignedMaurerDistanceMapImageFilter::Execute(itk::simple::Image const&) () from /libso/libSimpleITK_ITKDistanceMap-2.4.so.1

#9  0x00007ffff0f6c539 in itk::simple::SignedMaurerDistanceMap(itk::simple::Image const&, bool, bool, bool, double) () from /libso/libSimpleITK_ITKDistanceMap-2.4.so.1
2

Sounds like something you should try with Thread Sanitizer…

1 Like
3

Thank you for your response. I agree. I have been running it with tsan. It is not very reproducible. So, waiting for a crash to get more details. No crashes yet.

4

You say your application is running with 30 threads? What are these threads doing? All they all working on the computeDistanceMap? doing other ITK tasks? Or other application task? Additional explanation of the application may be beneficial.

5

Thank you for your response. This is an application that runs in a single process. The application processes a group of images. When the application starts it sets itk::simple::ProcessObject::SetGlobalDefaultNumberOfThreads(1). The application then creates a pool of threads. Each thread is used to process a single image. Each thread performs several processing steps independently on the assigned image. There is no shared input or output between threads.

All threads use the computeDistanceMap function as part of a processing step. computeDistanceMap is the first Itk task the threads perform. They also use several other ITK functions after using computeDistanceMap in other processing steps in the pipeline. This issue happens only when the number of threads used to run the application is greater or equal to 30. When the application is run with fewer threads, I have not seen a crash. All systems used for testing have at least 30 logical cores. Please let me know if you need any other information. Thank you.

6

When the segfault is encountered, it is at the beginning of processing, after a while or does it vary?

7

The threads perform several processing steps before reaching the processing step that uses computeDistanceMap. Some threads even finish executing computeDistanceMap. The crashes observed so far have been across multiple different images rather than being tied to a specific image. However, the stack trace always shows that the crash happened in the Voronoi method.

8

There is some “initialization” code that may get executed the first time certain ITK code is executed that is not concurrent thread safe. So if the error isn’t happening during the initial called to your distance functions this does not sounds like the issue.

I looked at the Voronoi code class and not thing stood out at as an issue between concurrency between multiple instances.

The other area in ITK/SimpleITK that comes to mind as a potential issue is ITK’s multi-threader. There are potentially 3 different backends which could be available via ProcessObject::SetGlobalDefaultThreader(). The TBB theader may need to be enable with specific ITK build flags so that may not be available. Perhaps trying a different one may help?

But it sounds like @seanm advice is best to use a thread analysis tool to help track down the issue may be the best approach.

9

Thank you for your suggestions and checking the Voronoi code class. ITK was not built with TBB. Hence, I am currently using the platform multi-threader. I will switch to the pool multi-threader and verify if that helps. I am also running the application with thread sanitizer. I will update this thread with the results if a crash happens.

Regarding your comment about potential thread-unsafe initialization the first time certain ITK code is executed: I am trying to understand how this should be handled. Would you recommend a “warm up” call where I execute a single sequential call to SignedMaurerDistanceMap or similar function on a dummy image at the beginning of the application before spawning worker threads to trigger all non-thread safe global initializations.

Or in this specific case will wrapping this section in a mutex as shown below help to trigger all non-thread safe global initializations.

{
    std::lock_guard<std::mutex> lock(appMutex)

    const itk::simple::Image binaryImage = itk::simple::ImportAsInt16(binaryVec.data(), dataDimension);

    if (distanceType == DistanceType::Euclidean) {
        distanceImage = itk::simple::SignedMaurerDistanceMap(binaryImage, false, false);
    } else {
        distanceImage = itk::simple::SignedMaurerDistanceMap(binaryImage, false, true);
    }
}