Doxygen with Github Actions and Pages.

blowekamp · February 7, 2022, 8:48pm

I recently made some changes with the ITK Doxygen configuration to minimize the about of changes from night to night in the HTML generated by Doxygen. Minimizing the number of files modified each day enabled regular web caching based on modified time to work, and adds with small daily updated to a git hub pages branch.

I have created a GitHub repository here: https://github.com/blowekamp/ITKDoxygen with associated gh-pages here: https://github.com/blowekamp/ITKDoxygen/deployments/activity_log?environment=github-pages

It also uses a described docker container, a steps so that anyone can easily regenerate and debug the Doxygen generation process. The GitHub Actions has been successfully updating the gh-pages for over a month now. Unfortunately, the searching does not work currently but there are likely options for Doxygen that should enable the functionality.

I’m not sure if there is any wider interest in using this infrastructure for main ITK Doxygen generation. Please let me know.

Also there is some specialized javascript code for advanced caching that may no longer be needed with the removal of daily modification of HTML file.

dzenanz · February 7, 2022, 9:06pm

These changes sound useful. @matt.mccormick @kitware-sysadmin might be able to help.

blowekamp · February 11, 2022, 1:54pm

I recently merged a PR do fix some issues the itkgroup.pl doxygen processing file:

github.com/InsightSoftwareConsortium/ITK

BUG: Fix bad doxygen strings

InsightSoftwareConsortium:master ← blowekamp:FixDoxygen_itkgroup

opened 04:35PM - 10 Feb 22 UTC

blowekamp

+8 -3

Fix many methods being documented in Doxygen with wrong strings. The the proces…ses performed in the itkgroup.pl is fixed to produce the proper indents for the modified auto-groupings. Co-authored-by: Dave Chen <dchen@mail.nih.gov>   ## PR Checklist - [ ] No [API changes](https://github.com/InsightSoftwareConsortium/ITK/blob/master/CONTRIBUTING.md#breaking-changes) were made (or the changes have been approved) - [ ] No [major design changes](https://github.com/InsightSoftwareConsortium/ITK/blob/master/CONTRIBUTING.md#design-changes) were made (or the changes have been approved) - [ ] Added test (or behavior not changed) - [ ] Updated API documentation (or API not changed) - [ ] Added [license](https://github.com/InsightSoftwareConsortium/ITK/blob/master/Utilities/KWStyle/ITKHeader.h) to new files (if any) - [ ] Added Python wrapping to new files (if any) as described in [ITK Software Guide](https://itk.org/ItkSoftwareGuide.pdf) Section 9.5 - [ ] Added [ITK examples](https://github.com/InsightSoftwareConsortium/ITKSphinxExamples) for all new major features (if any) Refer to the [ITK Software Guide](https://itk.org/ItkSoftwareGuide.pdf) for further development details if necessary.

I have noticed that the ITK doxygen has a time stamp of Jan 18. After trying another web browser and clearing the cache to make sure it was not the odd ITK cache thing going on the old data remained. So I am concluding the the ITK doxygen has not been updated in almost a month.

I was able to check the GHPages based documentation, and it appears the ImageBase::~ImageBase no longer has the erroneous doc string, along with many other methods.

blowekamp · March 18, 2022, 1:21pm

@matt.mccormick @dzenanz @hjmjohnson The itk.org nightly Doxygen has not been updated. Should this repo be move to the InsightSoftwareConsortium ?

dzenanz · March 18, 2022, 3:58pm

Doxygen build succeeded without errors or warnings. Maybe @Jake_Stookey could look into blaster machine and why the documentation is not published?

@hjmjohnson is PR 3311 attempting to fix it? Or just make the existing process more efficient?

As for Brad’s ITKDoxygen, it should be moved to InsightSoftwareConsortium organization and the search issue fixed before we can switch to that as the official documentation build process. @blowekamp do you need any special permissions from Kitware sysadmin team to make the switch?

Jake_Stookey · March 18, 2022, 4:20pm

We can help if it’s something on the server end, but SysAdmins don’t typically get into the developer process of pushing this out. Maybe @matt.mccormick can look into it on Blaster?

The server to push it out to changed: it used to need to be pushed to public.kitware.com, and now it needs to be pushed to web.kitware.com (or simply itk.org which is hosted on that system). So that might be something to look into first.

dzenanz · March 18, 2022, 4:50pm

I updated the script which was pushing docs, changing public into web. I guess we will know tomorrow whether that worked.

hjmjohnson · March 19, 2022, 12:48pm

@dzenanz PR 3311 is not related to this thread or errors. That work is focused on making the wrapping easier to debug with the ultimate goal of fixing the type hints generation.

dzenanz · March 20, 2022, 11:36am

I guess that didn’t work. Docs footer still says “Generated on Tue, 18 Jan 2022 06:30:40 +0000”.

blowekamp · March 21, 2022, 7:41pm

The repository has been moved to be under the InsightConsortium:

And the pages here: ITK: Insight Toolkit

There are some problems with the GH builds. Hopefully this PR fixes them:

github.com/InsightSoftwareConsortium/ITK

BUG: Add DOXYGEN_(FORMAT) options to DoxygenConfig.cmake

InsightSoftwareConsortium:master ← blowekamp:RestoreDoxygenOutputs

opened 07:04PM - 21 Mar 22 UTC

blowekamp

+8 -19

When ITK_DOXYGEN_XML was enabled XML was not being produced. These options were… omitted from the configuration file.   ## PR Checklist - [ ] No [API changes](https://github.com/InsightSoftwareConsortium/ITK/blob/master/CONTRIBUTING.md#breaking-changes) were made (or the changes have been approved) - [ ] No [major design changes](https://github.com/InsightSoftwareConsortium/ITK/blob/master/CONTRIBUTING.md#design-changes) were made (or the changes have been approved) - [ ] Added test (or behavior not changed) - [ ] Updated API documentation (or API not changed) - [ ] Added [license](https://github.com/InsightSoftwareConsortium/ITK/blob/master/Utilities/KWStyle/ITKHeader.h) to new files (if any) - [ ] Added Python wrapping to new files (if any) as described in [ITK Software Guide](https://itk.org/ItkSoftwareGuide.pdf) Section 9.5 - [ ] Added [ITK examples](https://github.com/InsightSoftwareConsortium/ITKSphinxExamples) for all new major features (if any) Refer to the [ITK Software Guide](https://itk.org/ItkSoftwareGuide.pdf) for further development details if necessary.

blowekamp · April 13, 2022, 5:10pm

Most of the kinks with the ITK Doxygen changes have been worked out and address to the ITKDoxygen Github Action is now daily publishing the generated Doxygen here:

https://insightsoftwareconsortium.github.io/ITKDoxygen/index.html

The search is also fixed, by using the client side search of the index. It may be slower than the server size, there is no option to run PHP on GitHub Pages.

dzenanz · April 13, 2022, 5:36pm

What would it take to move “Detailed Description” to the top of the page? It is almost always more important than “Public Types” etc.

Minor: AddImageFilter docs look incomplete and strange.

blowekamp · April 13, 2022, 5:55pm

What would it take to move “Detailed Description” to the top of the page? It is almost always more important than “Public Types” etc.

That has been like that, and is not a “bug”.

Minor: AddImageFilter docs look incomplete and strange.

Agreed. Something is off there. And the Sphinx links are broken, and the figures don’t seems to be working… many issue.

But it’s running right now. Lets get this currently proposed search fix/improvement in. And hopefully once the Doxygen on GHA is running regularly, the community can be help with some of the remaining issues.

dzenanz · April 13, 2022, 6:02pm

I always assumed it is not a bug, but I don’t like this layout. Can we change it easily? If so, we could put it up for a vote?

blowekamp · April 13, 2022, 6:04pm

It looks like a custom layout may be needed:

matt.mccormick · May 12, 2022, 2:33pm

@blowekamp this is looking great!

Is it possible to publish and link to the html, xml, and tag tarballs in the output HTML? It looks like there is some work to upload the xml on GitHub Actions.

blowekamp · May 12, 2022, 2:43pm

Currently, the tarball for both the HTML and XML are uploaded as a single Github Artifact named “doxygen”. I’m not sure linking to the Github Action Artifact is the thing todo. Uploading the generated files to an archive like S3 or Kitware Data seems like a better location to store the artifice for public consumptions and downloading.

The other issue with the Doxygen generation/configuration is a “dot” executable issue:

Generating dependency graph for group Registrsh: 1: /usr/sbin/dot: not found
10339
/work/ITK-bld/Utilities/Doxygen/Modules/ITKCommon.dox:24: error: Problems running dot: exit code=127, command='/usr/sbin/dot', arguments='"/work/ITK-bld/Utilities/Doxygen/html/inline_dotgraph_1.dot" -Tsvg -o "/work/ITK-bld/Utilities/Doxygen/html/dot_inline_dotgraph_1.svg"'

The figures are not currently being generated.

Overall it seems to be working well it just needs a couple contributions and fixed to be

matt.mccormick · May 12, 2022, 5:31pm

I am taking a look at this.

matt.mccormick · May 12, 2022, 5:35pm

Right – I believe they are cleaned up after a certain time period. It may be better to push to a GitHub Release Artifact on the repository – for tagged releases, then a latest, etc. for the latest.

Generating dependency graph for group Registrsh: 1: /usr/sbin/dot:

Installing the graphviz package should fix this.

blowekamp · May 12, 2022, 5:57pm

Ah, good suggestions. That should work well even for ITK tags too with a little more work.

Unfortunately it appears to be more complicated than that as it is already installed.

github.com

InsightSoftwareConsortium/ITKDoxygen/blob/master/Dockerfile#L12


      
          
          RUN apt-get update && \
            DEBIAN_FRONTEND=noninteractive apt-get install -y \
            cmake \
            bison \
            build-essential \
            curl \
            flex \
            git \
            ninja-build \
            graphviz \
            python3 \
            && apt-get clean
          
          RUN ln /usr/bin/dot /usr/sbin/dot
          
          RUN git clone https://github.com/doxygen/doxygen.git && \
              ( cd doxygen && \
                git checkout Release_1_9_3 && \
                mkdir bld && cd bld && \
                cmake -G Ninja -DCMAKE_BUILD_TYPE=Release .. && \

The above error message is picking up /usr/sbin/dot despite the package providing /usr/bin/dot. It needs further investigation.