Add possibility to control TBB region splitter

illia.kramarenko · January 31, 2020, 2:44pm

Hi,
In the ITK 5.0 you’ve added new multithreading approach using the TBB library. Now it’s possible to parallelize work on a pixel level. My question is in following - how can I force TBB splitter to stop on slice level? I’m using my own data structures and I want to run default ITK filters on these data structures. Earlier ITK provides a virtual method SplitRequestedRegion(…) and this was enough for me.

But now almost the same functionality (TBBImageRegionSplitter) is in file Modules/Core/Common/src/itkTBBMultiThreader.cxx under the anonymous namespace and now I cannot control splitting without changing ITK.

Maybe it is possible to pass region splitter as lambda or provide a method to override it in ITK?
I do realize that I can use old parallelization but it will be great to use a new TBB approach.

Thanks for your help!
Illia.

dzenanz · January 31, 2020, 2:54pm

Welcome to the community Illia!

Currently there is no way for ITK user to modify TBB’s splitting behavior. You are welcome to propose a PR which introduces an API for controlling the region splitting process. If you decide to try that, beware that the default multi-threader can be selected by environment variable, and that it might not be TBBMultiThreader despite that being the default if it is built.

matt.mccormick · January 31, 2020, 3:21pm

Perhaps ParallelizeImageRegionRestrictDimension meets your needs:

github.com

InsightSoftwareConsortium/ITK/blob/ca670a797bac1109cc5cc63b3d26d5a1c8c1fd9e/Modules/Core/Common/include/itkMultiThreaderBase.h#L316-L325


/** Similar to ParallelizeImageRegion, but do not split the region along one
 * of the directions. If VDimension is 1, restrictedDirection is ignored
 * and no parallelization occurs. */
template <unsigned int VDimension>
ITK_TEMPLATE_EXPORT void
ParallelizeImageRegionRestrictDirection(unsigned int                              restrictedDirection,
                                        const ImageRegion<VDimension> &           requestedRegion,
                                        TemplatedThreadingFunctorType<VDimension> funcP,
                                        ProcessObject *                           filter)
{

blowekamp · January 31, 2020, 3:28pm

If you can consider splitting your data into N things, then the ParallelizeArray method may be a good fit. N things could be N ImageRegions computed on the fly.

github.com

InsightSoftwareConsortium/ITK/blob/ca670a797bac1109cc5cc63b3d26d5a1c8c1fd9e/Modules/Core/Common/include/itkMultiThreaderBase.h#L280-L289


/** Parallelize an operation over an array. If filter argument is not nullptr,
 * this function will update its progress as each index is completed.
 *
 * This implementation simply delegates parallelization to the old interface
 * SetSingleMethod+SingleMethodExecute. This method is meant to be overloaded! */
virtual void
ParallelizeArray(SizeValueType             firstIndex,
                 SizeValueType             lastIndexPlus1,
                 ArrayThreadingFunctorType aFunc,
                 ProcessObject *           filter);

illia.kramarenko · January 31, 2020, 4:52pm

Hi,

Thanks for the reply! I’ve looked at this function but it didn’t help me because I want to run default ITK filters on my data structures. I can write a new filter using ParallelizeImageRegionRestrictDimension but my goal is to run out of box ITK filters without changing source code.

blowekamp · January 31, 2020, 5:05pm

I don’t think any of the multi-threaders use the Filter’s Splitter any more. It looks like it might be possible to restore it for the PlatformMultiThread and the PoolMultiThreader. Currently the multthreaders just use the global default as opposed to the filters:

github.com

InsightSoftwareConsortium/ITK/blob/master/Modules/Core/Common/src/itkMultiThreaderBase.cxx#L600




ITK_THREAD_RETURN_FUNCTION_CALL_CONVENTION
MultiThreaderBase ::ParallelizeImageRegionHelper(void * arg)
{
  using ThreadInfo = MultiThreaderBase::WorkUnitInfo;
  auto *       threadInfo = static_cast<ThreadInfo *>(arg);
  ThreadIdType threadId = threadInfo->WorkUnitID;
  ThreadIdType threadCount = threadInfo->NumberOfWorkUnits;
  auto *       rnc = static_cast<struct RegionAndCallback *>(threadInfo->UserData);


  const ImageRegionSplitterBase * splitter = ImageSourceCommon::GetGlobalDefaultSplitter();
  ImageIORegion                   region(rnc->dimension);
  for (unsigned d = 0; d < rnc->dimension; d++)
  {
    region.SetIndex(d, rnc->index[d]);
    region.SetSize(d, rnc->size[d]);
  }
  ThreadIdType total = splitter->GetSplit(threadId, threadCount, region);


  MultiThreaderBase::HandleFilterProgress(rnc->filter);

Could you use one of the other threaders?

dzenanz · February 1, 2020, 4:08pm

DynamicThreadedGenerateData does not use the filter’s splitter. When that is needed, ThreadedGenerateData should be used, as per classic multi-threading model.