Updating ITK's HDF5

Hi all,

ITK master includes HDF5 1.8.17, and 1.8.20 was released today so I thought I’d try to update it. But the current scripts seem to be out of date. HDF switched from svn to git in late 2016 or early 2017, ie since 1.8.17. So I figure this fact perhaps changes the entire strategy that those scripts use. How to proceed?

Thanks,

2 Likes

Hello @seanm,

Thanks for looking into this. Best would be to create an UpdateFromUpstream.sh like there are for other third party libraries. This script calls update-third-party.bash. You can try to do this, and report here the problems you are encountering.

Hope this helps.

1 Like

François,

So I gave it a try…

The last few lines of output are:

Receiving objects: 100% (551/551), 2.62 MiB | 0 bytes/s, done.
Resolving deltas: 100% (152/152), done.
From /Users/sean/external/ITK/work/extract
 * [new branch]      upstream-HDF5 -> upstream-HDF5
++ git merge --log -s ours --no-commit --allow-unrelated-histories upstream-HDF5
Merge with strategy ours failed.
+ rm -rf /Users/sean/external/ITK/work

It creates an ‘upstream-HDF5’ branch, but Modules/ThirdParty/HDF5/src/itkhdf5 doesn’t have new contents, I think perhaps because that folder already exists. Should I git rm it prior?

Cheers,

@matt.mccormick Any idea or suggestion to help @seanm ?

Yes, it is prudent to start apply on a fresh ITK clone.

If it still fails, git status may be able to help determine the issues with the merged, and conflicts may need to be resolved manually.

The second warning here comes from HDF5. I thought that updating HDF5 might help. Here is a patch which should finish the process which @hjmjohnson started in March.

2 Likes

Now that the update patch has been merged, some machines have new ~300 test failures. This seems to be the prevailing error message:

Assertion failed: (( '_'!=((const char *)__func__)[2] && '_'!=((const char *)__func__)[3] && !( ((const char *)__func__)[4] && (isupper((int)(__func__[3])) || isdigit((int)(__func__[3]))) && '_'==((const char *)__func__)[4] )) && "Function naming conventions are incorrect - check H5_IS_API|PUB|PRIV|PKG macros in H5private.h (this is usually due to an incorrect number of underscores)"), function itk_H5dont_atexit, file /Users/builder/external/ITK/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5.c, line 446.
itkTestDriver: Process exception: Child aborted

@blowekamp @seanm This seems to be affecting your machines. Is there anything special in these builds related to HDF5 configuration?

1 Like

Debug is on. I don’t think it’s enabled for the Gerrit CI.

This gives me deja vu. I think it’s related to the symbol mangling. IIRC, the HDF5 code actually introspects the names of their symbols… let me see if I can find old emails about it…

Cheers,

Hi,

It looks like the HDF5 update broke the MINCIO in a weird way.

ITK’s MINCIO now creates MINC files that only ITK can read/write. They’re somewhat right, as I can use ITK’s MINCIO to convert the file to a NIFTI and that NIFTI file is fine, however minc-toolkit proper can no longer read the MINC files.

Thanks for the feedback Gabriel.

I was able to reproduce the problem. As @blowekamp said, it appears in a debug build. The crash occurs during static initialization of ITKHDF5 library, so everything which is linked to it crashes (e.g. MINCIO). It is not obvious what the problem is. As @seanm mentioned the mangling, here is a patch which updates that. But that does not fix the crash, it only adds some warnings about duplicate symbol re-definitions.

I think this is what I was recalling:

http://review.source.kitware.com/#/c/23177/

https://gitlab.kitware.com/third-party/hdf5/commit/72b9dd90c1c189a5e9059d375e4c20444bfa4a9c#f49d7c12a9a817d8d3e0e7397caff73fc79e6b9c

Sean

3 Likes

We are missing this. I will check whether that is enough or we need some other changes VTK did.

Thanks for pointing this out @seanm!

1 Like

This patch solves the problem for me.

Unfortunately, this doesn’t seem to have fixed the MINCIO issue for me.

Any suggestions as to how to debug?

Try clean and rebuild, or do a fresh build (configure and build ITK in a new directory).

Done and done.

Using the .hdf5 format does seem to work, so this is a MINCIO specific issue.

Paging @Vladimir_Fonov

Oh, I see that HDF5 has bumped to “hdf5_1_10_2”, that’s a big jump that libminc hasn’t been tested or patched against.