# Handling of combined accents by ITK system tools

**URL:** <https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619>\
**Category:** Uncategorized\
**Created:** [February 22, 2019, 3:13pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619 "2019-02-22T15:13:05Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![tim-evain](https://discourse.itk.org/user_avatar/discourse.itk.org/tim-evain/32/220_2.png) [@tim-evain](https://discourse.itk.org/u/tim-evain)\
**Post date:** [February 22, 2019, 3:13pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/1 "2019-02-22T15:13:05Z")

</div>

Hello everyone,

I’m using the itksys tools to do some file parsing and I’ve encountered a nasty problem with some filepaths containing accented characters.  
For example I have this folder name in my database : “aorte\_déroulée”. It appears that in fact, the “é” symbol is a “e” char followed by a combining acute accent char (I think that data came from a MacOS station).  
When using itksys::Directory::Load() on the parent folder, the given string is then “aorte\_de´roule´e”, which is fine. Problem is that this path is not recognized as valid afterward: using itksys::Directory::Load("[…]/aorte\_de´roule´e") or itksys::SystemTools::FileIsDirectory("[…]/aorte\_de´roule´e") fails. Replacing directly by the full char “é” in the string doesn’t help either.

Any insights on why it’s happening and how to fix it ?  
(I can rename the folder with the accented letter char, then it will be correctly processed but I can’t control the file naming in my application)

Tim

EDIT: It appears to be a Windows-specific issue, as tests on MacOS and Linux run fine. I’ve tracked down the problem to:

- itksys::Directory::Load() fails on (Directory.cxx, l 121)

```auto
  srchHandle = _wfindfirst_func(
    (wchar_t*)Encoding::ToWindowsExtendedPath(buf).c_str(), &data);

```

where `srchHandle` is `-1`

- itksys::SystemTools::FileIsDirectory fails on (SystemTools.cxx, l 2902)

```auto
  DWORD attr =
    GetFileAttributesW(Encoding::ToWindowsExtendedPath(name).c_str());

```

where `attr == INVALID_FILE_ATTRIBUTES`

---

<div class="post-metadata">

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\
**Post date:** [February 24, 2019, 1:43am UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/2 "2019-02-24T01:43:45Z")

</div>

Hi Tim,

Could you please test with this PR that updates KWSys?

> <https://github.com/InsightSoftwareConsortium/ITK/pull/534>

If there still are issues, we can fix KWSys.

Thanks,  
Matt

---

<div class="post-metadata">

**Author:** ![tim-evain](https://discourse.itk.org/user_avatar/discourse.itk.org/tim-evain/32/220_2.png) [@tim-evain](https://discourse.itk.org/u/tim-evain)\
**Post date:** [February 25, 2019, 12:12pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/3 "2019-02-25T12:12:38Z")

</div>

Hi Matt,

I’ve tried the current master state this morning ; it doesn’t fix this specific problem 😄.

Tim

---

<div class="post-metadata">

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\
**Post date:** [February 25, 2019, 2:51pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/4 "2019-02-25T14:51:25Z")

</div>

Hi Tim,

Thanks for testing 👍

Is it possible to create a test that reproduces the issue?

Thanks,  
Matt

---

<div class="post-metadata">

**Author:** ![tim-evain](https://discourse.itk.org/user_avatar/discourse.itk.org/tim-evain/32/220_2.png) [@tim-evain](https://discourse.itk.org/u/tim-evain)\
**Post date:** [February 25, 2019, 3:59pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/5 "2019-02-25T15:59:50Z")

</div>

Sure !

Here it is:  
[CombinedAccents\_TestCase.zip](https://discourse.itk.org/uploads/default/original/1X/b9d3891d022d7fe99877937a19fc14a5e1edd280.zip) (1.9 KB)  
It’s a very simple case, it should output the name of the accented folder, but it does not.

Tim

---

<div class="post-metadata">

**Author:** ![matt.mccormick](https://discourse.itk.org/user_avatar/discourse.itk.org/matt.mccormick/32/7_2.png) [@matt.mccormick](https://discourse.itk.org/u/matt.mccormick)\
**Post date:** [February 25, 2019, 4:16pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/6 "2019-02-25T16:16:48Z")

</div>

> [@tim-evain](#):
>
> It appears to be a Windows-specific issue, as tests on MacOS and Linux run fine. I’ve tracked down the problem to:

Thanks, Tim!

I created this issue to track this:

> <https://github.com/InsightSoftwareConsortium/ITK/issues/540>
>
> \### Description
> 
> As discussed on Discourse:
> 
> https://discourse.itk.org/t/h…andling-of-combined-accents-by-itk-system-tools/1619/5
> 
> reported by Tim Evain,
> 
> \> I’m using the itksys tools to do some file parsing and I’ve encountered a nasty problem with some filepaths containing accented characters.
> For example I have this folder name in my database : “aorte\_déroulée”. It appears that in fact, the “é” symbol is a “e” char followed by a combining acute accent char (I think that data came from a MacOS station).
> When using itksys::Directory::Load() on the parent folder, the given string is then “aorte\_de´roule´e”, which is fine. Problem is that this path is not recognized as valid afterward: using itksys::Directory::Load("\[…\]/aorte\_de´roule´e") or itksys::SystemTools::FileIsDirectory("\[…\]/aorte\_de´roule´e") fails. Replacing directly by the full char “é” in the string doesn’t help either.
> 
> \### Steps to Reproduce
> 
> \<!--
> 1. \[First Step\]
> 2. \[Second Step\]
> 3. \[and so on...\]
> 
> Provide a minimal, complete, compilable, and verifiable example (commonly
> abbreviated as MWE, Minimal Working Example, or sometimes referred to as SSEE,
> Short, Self Contained, Correct (Compilable) Example, SSCCE) or code snippet,
> either through a GitHub gist (https://gist.github.com/) or providing your own
> files (including your source code, \`CMakeLists.txt\` file if applicable, and your
> data) reproducing the issue or showing an incorrect result. --\>
> 
> \### Expected behavior
> 
> Accented characters are handled.
> 
> \### Actual behavior
> 
> \- itksys::Directory::Load() fails on (Directory.cxx, l 121) 
> \`\`\`
> srchHandle = \_wfindfirst\_func(
> (wchar\_t\*)Encoding::ToWindowsExtendedPath(buf).c\_str(), &data);
> \`\`\`
> where \`srchHandle\` is \`-1\` 
> \- itksys::SystemTools::FileIsDirectory fails on (SystemTools.cxx, l 2902)
> \`\`\`
> DWORD attr =
> GetFileAttributesW(Encoding::ToWindowsExtendedPath(name).c\_str());
> \`\`\`
> where \`attr == INVALID\_FILE\_ATTRIBUTES\`
> 
> \### Reproducibility
> 
> Recent ITK Git \`master\` (with updated KWSys) was tested.
> 
> \### Environment
> 
> \> It appears to be a Windows-specific issue, as tests on MacOS and Linux run fine. I’ve tracked down the problem to:
> 
> \### Additional Information
> 
> Test case from Tim attached.
> \[CombinedAccents\_TestCase.zip\](https://github.com/InsightSoftwareConsortium/ITK/files/2901365/CombinedAccents\_TestCase.zip)
> 
> CC: @bradking

---

<div class="post-metadata">

**Author:** ![tim-evain](https://discourse.itk.org/user_avatar/discourse.itk.org/tim-evain/32/220_2.png) [@tim-evain](https://discourse.itk.org/u/tim-evain)\
**Post date:** [February 25, 2019, 4:32pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/7 "2019-02-25T16:32:27Z")

</div>

> [@matt.mccormick](#):
>
> Thanks, Tim!
> 
> I created this issue to track this:

You’re welcome, thanks a million for the help.

Tim

---

<div class="post-metadata">

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\
**Post date:** [February 27, 2019, 4:17pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/8 "2019-02-27T16:17:14Z")

</div>

@tim-evain please review the [PR 546](https://github.com/InsightSoftwareConsortium/ITK/pull/546). You can find more information in the [issue comments](https://github.com/InsightSoftwareConsortium/ITK/issues/540#issuecomment-467916611).

---

<div class="post-metadata">

**Author:** ![tim-evain](https://discourse.itk.org/user_avatar/discourse.itk.org/tim-evain/32/220_2.png) [@tim-evain](https://discourse.itk.org/u/tim-evain)\
**Post date:** [February 27, 2019, 5:04pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/9 "2019-02-27T17:04:44Z")

</div>

@dzenanz I’ve just tried on Windows, but switching the default codepage has solved the issue 👍.  
I will try with other OS whenever I get a chance.

Thank you and @brad.king for looking into this.

Tim

---

<div class="post-metadata">

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\
**Post date:** [February 27, 2019, 8:53pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/10 "2019-02-27T20:53:09Z")

</div>

@tim-evain do you have a GitHub profile? If so it would be good if you formally approved the PR.

---

<div class="post-metadata">

**Author:** ![tim-evain](https://discourse.itk.org/user_avatar/discourse.itk.org/tim-evain/32/220_2.png) [@tim-evain](https://discourse.itk.org/u/tim-evain)\
**Post date:** [February 28, 2019, 11:07am UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/11 "2019-02-28T11:07:45Z")

</div>

I do. I see the PR is closed now. Sorry for the delayed answer.  
I will try to do so next time; but can I do it as a non-member of the ISC ?

---

<div class="post-metadata">

**Author:** ![dzenanz](https://discourse.itk.org/user_avatar/discourse.itk.org/dzenanz/32/1093_2.png) [@dzenanz](https://discourse.itk.org/u/dzenanz)\
**Post date:** [February 28, 2019, 3:09pm UTC](https://discourse.itk.org/t/handling-of-combined-accents-by-itk-system-tools/1619/12 "2019-02-28T15:09:13Z")

</div>

I think you can. I think anyone can review a PR. But it takes an approving review from someone with write access to enable the green merge button.