# MattesMutualInformationImageToImageMetricv4 sampling heuristics

**URL:** <https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140>\
**Category:** Engineering\
**Created:** [July 20, 2018, 7:20pm UTC](https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140 "2018-07-20T19:20:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gdevenyi](https://discourse.itk.org/user_avatar/discourse.itk.org/gdevenyi/32/355_2.png) [@gdevenyi](https://discourse.itk.org/u/gdevenyi)\
**Post date:** [July 20, 2018, 7:20pm UTC](https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140/1 "2018-07-20T19:20:07Z")

</div>

As has been discussed on the ANTs issue tracker [https://github.com/ANTsX/ANTs/issues/543](https://github.com/ANTsX/ANTs/issues/543), the existing heuristics for the MI overlap are not very good.

Right now, its based on a ratio of points at [https://github.com/hinerm/ITK/blob/master/Modules/Registration/Metricsv4/include/itkMattesMutualInformationImageToImageMetricv4.hxx#L218](https://github.com/hinerm/ITK/blob/master/Modules/Registration/Metricsv4/include/itkMattesMutualInformationImageToImageMetricv4.hxx#L218)

This really a good heuristic as it makes the success/failure depend on things like having an uncropped image with lots of background (where the background is unimportant). It also causes failures where you want to use a mask to do some finer adjustments to an affine registration to match specific anatomy, but the mask is “too small” according to the fraction.

The idea as a replacement is something based on the size/number of samples for the joint histogram.

I’d like to open this up for discussion as to how to implement.

---

<div class="post-metadata">

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\
**Post date:** [July 20, 2018, 8:29pm UTC](https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140/2 "2018-07-20T20:29:20Z")

</div>

`Too many samples map outside of [something]` is message quite a few people have encountered. Mostly it appears when the initial transformation for registration is wrong, but sometimes simply because overlap is not big enough. An improvement to that would be welcome. I have not participated in writing the metric so I am a bit short on implementation proposals.

---

<div class="post-metadata">

**Author:** ![gdevenyi](https://discourse.itk.org/user_avatar/discourse.itk.org/gdevenyi/32/355_2.png) [@gdevenyi](https://discourse.itk.org/u/gdevenyi)\
**Post date:** [July 20, 2018, 8:34pm UTC](https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140/3 "2018-07-20T20:34:55Z")

</div>

Indeed @dzenanz this is specifically address the point where the overlap is _very good_ by either manual or some other initialization, but due to this heuristic the registration bails out.

Another way to understand the problem here is, if I had resampled the images and cropped them rather than passing the initialization and mask into the registration, it would work, but because I’ve doing this in the middle of the pipeline, ratio heuristic bails out.

---

<div class="post-metadata">

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\
**Post date:** [July 20, 2018, 10:10pm UTC](https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140/4 "2018-07-20T22:10:26Z")

</div>

@stnava has proposed a fix for this:

[http://review.source.kitware.com/#/c/23580/](http://review.source.kitware.com/#/c/23580/)

---

<div class="post-metadata">

**Author:** ![gdevenyi](https://discourse.itk.org/user_avatar/discourse.itk.org/gdevenyi/32/355_2.png) [@gdevenyi](https://discourse.itk.org/u/gdevenyi)\
**Post date:** [July 22, 2018, 2:36pm UTC](https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140/5 "2018-07-22T14:36:29Z")

</div>

Wow, well that’s totally the simplest implementation. Clearly one sample wouldn’t actually work, but is it work figuring out the range 1-N where Mattes fails, or is this fix good enough.

I’m down for just accepting this.

---

<div class="post-metadata">

**Author:** ![blowekamp](https://discourse.itk.org/user_avatar/discourse.itk.org/blowekamp/32/79_2.png) [@blowekamp](https://discourse.itk.org/u/blowekamp)\
**Post date:** [July 23, 2018, 1:23pm UTC](https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140/6 "2018-07-23T13:23:42Z")

</div>

xref: [https://github.com/InsightSoftwareConsortium/ITK/pull/49](https://github.com/InsightSoftwareConsortium/ITK/pull/49)

---

<div class="post-metadata">

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\
**Post date:** [July 23, 2018, 2:18pm UTC](https://discourse.itk.org/t/mattesmutualinformationimagetoimagemetricv4-sampling-heuristics/1140/7 "2018-07-23T14:18:00Z")

</div>

> [@gdevenyi](#):
>
> Wow, well that’s totally the simplest implementation. Clearly one sample wouldn’t actually work, but is it work figuring out the range 1-N where Mattes fails, or is this fix good enough.
> 
> I’m down for just accepting this.

Yes, agreed – it is a good step forward.

Thanks to Brian for proposing a patch.