# Add a parallel compression method to NRRD and/or MetaImage? **URL:** \ **Category:** Engineering\ **Created:** [February 21, 2018, 4:33pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696 "2018-02-21T16:33:03Z")\ **Posts on this page:** 20\ **Page:** 1

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\ **Post date:** [February 21, 2018, 4:33pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/1 "2018-02-21T16:33:03Z")

It used to be faster to write compressed images, due to compression being faster than hard disk transfer rates. Not any more! Solid state disks are now pretty fast, which makes (single-threaded) compression the bottleneck. We are still mostly using the outdated single-threaded GunZip for compression of most of image formats. What do you think about adding something more modern, e.g. [zstd](https://github.com/facebook/zstd) to NRRD and/or MetaImage? ---

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\ **Post date:** [February 21, 2018, 5:37pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/2 "2018-02-21T17:37:43Z")

+1 zstd adds a huge performance and compression ratio bump. ---

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\ **Post date:** [February 21, 2018, 7:11pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/3 "2018-02-21T19:11:00Z")

Should we contact the [teem](http://teem.sourceforge.net/) group and see if they are willing to contribute to the effort? ---

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\ **Post date:** [February 21, 2018, 7:24pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/4 "2018-02-21T19:24:40Z")

@glk maintains Teem / NRRD. Gordon, what do you think? ---

**Author:** ![glk](https://discourse.itk.org/user_avatar/discourse.itk.org/glk/32/69_2.png) [@glk](https://discourse.itk.org/u/glk)\ **Post date:** [February 21, 2018, 10:52pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/5 "2018-02-21T22:52:06Z")

That sounds intriguing. Thanks for mentioning me by name or else I would not have noticed this. I should learn the learn the answer to this myself, but what are the 3rd party dependencies of zstd (like how png depends on zlib)? ---

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\ **Post date:** [February 21, 2018, 11:00pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/6 "2018-02-21T23:00:06Z")

By quickly glancing at their CMake sources, it seems not to have external dependencies. ---

**Author:** ![glk](https://discourse.itk.org/user_avatar/discourse.itk.org/glk/32/69_2.png) [@glk](https://discourse.itk.org/u/glk)\ **Post date:** [February 21, 2018, 11:01pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/7 "2018-02-21T23:01:33Z")

So how would it be using parallel computing of the compression? ---

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\ **Post date:** [February 21, 2018, 11:05pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/8 "2018-02-21T23:05:45Z")

It depends on PThreads on \*nix and Kernel32 on Windows which provide threading capabilities. I assume it also depends on STL and CRT. Libraries which do not depend on these system things are rare and usually of questionable utility. I meant it does not depend on some other non-standard library. ---

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\ **Post date:** [February 21, 2018, 11:09pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/9 "2018-02-21T23:09:41Z")

It looks like it is vanilla C as opposed to C++. ---

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\ **Post date:** [February 21, 2018, 11:36pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/10 "2018-02-21T23:36:36Z")

That explains the need for PThreads, instead of using C++11 threads 😃 ---

**Author:** ![glk](https://discourse.itk.org/user_avatar/discourse.itk.org/glk/32/69_2.png) [@glk](https://discourse.itk.org/u/glk)\ **Post date:** [February 23, 2018, 1:02pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/11 "2018-02-23T13:02:13Z")

Cool - so it’s feasible to add this new encoding to NRRD, and I’m intrigued by the possibility. Is there anyone willing to help me? I would like a side-effect of this to be a re-synching of ITK’s NrrdIO and the Teem sources from which it originates. ---

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\ **Post date:** [February 23, 2018, 2:28pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/12 "2018-02-23T14:28:29Z")

What seems like a logical way forward to me is adding zstd as another third party library to ITK, and have MetaIO and NRRDIO depend on it (in addition to depending on zlib), and make NRRD and MetaIO support additional encoding zstd (in addition to raw and gzip). I don’t know why ITK’s copy diverged from official teem. But teem has a lot of other stuff in addition to NRRD - maybe that was the reason. @matt.mccormick @blowekamp @hjmjohnson can you give more background? @glk what would be required to re-synchronize ITK and Teem? ---

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\ **Post date:** [February 23, 2018, 3:23pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/13 "2018-02-23T15:23:21Z")

We now have good infrastructure in place to pull from and push contributions to upstream third party projects with `git subtree`. We just have not enabled it for NRRD yet. We should do that as part of this effort 🙂 ---

**Author:** ![glk](https://discourse.itk.org/user_avatar/discourse.itk.org/glk/32/69_2.png) [@glk](https://discourse.itk.org/u/glk)\ **Post date:** [February 23, 2018, 3:46pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/14 "2018-02-23T15:46:53Z")

Yea, Teem has a lot of stuff that doesn’t make any sense for ITK. So, when Nrrd was first added to ITK, I wrote some scripts to extract just those portions of Teem (not just which files, but which parts of which files) that are built to make NrrdIO (look at Utilities/NrrdIO/unteem.pl). I sort of regret that now, because it always means that updates to NrrdIO from the ITK side have to be merged back into Teem in a more tedious/manual manner. I forget when this last happened (sometime after 2009). Matt - can you describe more how this “pull from and push contributions to upstream third party projects” might work? Or, is there another third-party project for which it is working, which could be an informative example? ---

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\ **Post date:** [February 23, 2018, 7:28pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/15 "2018-02-23T19:28:09Z")

Essentially, the [Git subtree merge](https://help.github.com/articles/about-git-subtree-merges/) process is, - Create a new orphan branch in the ITK repository. This has a completely independent history. This is where we keep the NrrdIO files. - Whenever we want to perform an update, we run a script that pulls in the new NrrdIO sources, possibly redacts files we don’t need, etc. – similar to your `unteem.pl`. We create a new commit on this branch from the update. - Git has the ability to merge this branch into a _subdirectory_ of the main `master` branch. This allows us to (theoretically) keep any modifications we have to make, like symbol name mangling related changes, etc, with Git intelligently performing the merge. - We have a client side hook, that checks for changes to Nrrd code, suggests submitting them upstream, and includes instructions on how to do so: > The process is automated with a shell script. We have common driver scripts for third party repositories managed by Git, but we could use something similar for Teem if it is still in SVN. The driver scripts: - [https://github.com/InsightSoftwareConsortium/ITK/blob/8a00e7a95206679aa92351c9b1c6b249b466759a/Utilities/Maintenance/UpdateThirdPartyFromUpstream.sh](https://github.com/InsightSoftwareConsortium/ITK/blob/8a00e7a95206679aa92351c9b1c6b249b466759a/Utilities/Maintenance/UpdateThirdPartyFromUpstream.sh) - We are migrating to this one: [https://github.com/InsightSoftwareConsortium/ITK/blob/master/Utilities/Maintenance/update-third-party.bash](https://github.com/InsightSoftwareConsortium/ITK/blob/master/Utilities/Maintenance/update-third-party.bash) Some third party libraries currently managed this way: - pygccxml (does not use the driver): [https://github.com/InsightSoftwareConsortium/ITK/blob/8a00e7a95206679aa92351c9b1c6b249b466759a/Modules/ThirdParty/pygccxml/src/UpdatepygccxmlFromUpstream.sh](https://github.com/InsightSoftwareConsortium/ITK/blob/8a00e7a95206679aa92351c9b1c6b249b466759a/Modules/ThirdParty/pygccxml/src/UpdatepygccxmlFromUpstream.sh) - VNL: [https://github.com/InsightSoftwareConsortium/ITK/blob/master/Modules/ThirdParty/VNL/UpdateFromUpstream.sh](https://github.com/InsightSoftwareConsortium/ITK/blob/master/Modules/ThirdParty/VNL/UpdateFromUpstream.sh) - GDCM: [https://github.com/InsightSoftwareConsortium/ITK/blob/8a00e7a95206679aa92351c9b1c6b249b466759a/Modules/ThirdParty/GDCM/UpdateFromUpstream.sh](https://github.com/InsightSoftwareConsortium/ITK/blob/8a00e7a95206679aa92351c9b1c6b249b466759a/Modules/ThirdParty/GDCM/UpdateFromUpstream.sh) - MetaIO: [https://github.com/InsightSoftwareConsortium/ITK/blob/8a00e7a95206679aa92351c9b1c6b249b466759a/Modules/ThirdParty/MetaIO/UpdateFromUpstream.sh](https://github.com/InsightSoftwareConsortium/ITK/blob/8a00e7a95206679aa92351c9b1c6b249b466759a/Modules/ThirdParty/MetaIO/UpdateFromUpstream.sh) ---

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\ **Post date:** [February 23, 2018, 7:46pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/16 "2018-02-23T19:46:23Z")

A blog post on ITK’s Git subtrees: [https://blog.kitware.com/stay-in-sync-with-upstream-via-git-subtrees/](https://blog.kitware.com/stay-in-sync-with-upstream-via-git-subtrees/) ---

**Author:** ![glk](https://discourse.itk.org/user_avatar/discourse.itk.org/glk/32/69_2.png) [@glk](https://discourse.itk.org/u/glk)\ **Post date:** [March 1, 2018, 11:01am UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/17 "2018-03-01T11:01:40Z")

Thanks for the detailed information. Work on Teem continues to be an unfunded effort, which I continue to do in a scatterbrained as-needed way, so I appreciate your patience and willingness to explain rudimentary things. It sounds like the first step is getting all the Teem source onto github, right? Sourceforge says it’s not dead yet, but I’m skeptical. ---

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\ **Post date:** [March 1, 2018, 10:06pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/18 "2018-03-01T22:06:32Z")

Your work on Teem is appreciated, Gordon. It is a valuable asset to the community! Yes, migration of Teem sources to GitHub is not absolutely necessary, but it would be a good first step that could help make this effort easier and the maintenance of Teem easier in the long term. ---

**Author:** ![glk](https://discourse.itk.org/user_avatar/discourse.itk.org/glk/32/69_2.png) [@glk](https://discourse.itk.org/u/glk)\ **Post date:** [March 1, 2018, 10:33pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/19 "2018-03-01T22:33:19Z")

Not to get off on too much of a tangent, but do you have any tips on how I can get a little bit of development support for Teem? The two main things I need are (1) complete refresh of all the CMakeLists (they were written a long time ago when Teem was first being eyed by NAMIC, but haven’t gotten updated to more modern CMake idioms, afaik), and (2) getting Teem CTest’d on the same variety of platforms as ITK is tested on. Currently, the only modern CMake stuff for anything in Teem is the way that ITK builds NrrdIO (not all of Teem, and not all of the most recent Teem), and only NrrdIO (not all of Teem) benefits from the testing reported on the ITK dashboard. Part of being scatterbrained is feeling like I’ve said all of this before, maybe even to you, years ago, but I’m not sure if there was a concrete plan. ---

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\ **Post date:** [March 1, 2018, 10:47pm UTC](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696/20 "2018-03-01T22:47:36Z")

GitHub would help in this case, too, due to the availability of free CI services, a larger developer community, and tools for collaboration. [Next page](https://discourse.itk.org/t/add-a-parallel-compression-method-to-nrrd-and-or-metaimage/696.md?page=2)