# ITK ERROR: CUDA ERROR: out of memory

**URL:** https://discourse.itk.org/t/itk-error-cuda-error-out-of-memory/6583
**Category:** Algorithms
**Tags:** python, rtk
**Created:** [April 18, 2024, 1:20am UTC](https://discourse.itk.org/t/itk-error-cuda-error-out-of-memory/6583 "2024-04-18T01:20:45Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![zhangm](https://discourse.itk.org/letter_avatar_proxy/v4/letter/z/34f0e0/32.png) [@zhangm](https://discourse.itk.org/u/zhangm)
#### Post date: [April 18, 2024, 1:20am UTC](https://discourse.itk.org/t/itk-error-cuda-error-out-of-memory/6583/1 "2024-04-18T01:20:45Z")

</div>

My data size is 5.03G with type uint16. When use FDK to reconstrunction the data type will cast into float, so my data size is 10.04G. And my output size set to 1.31G. According to this setting to reconstructing data by FDK, there will appear error:

```auto
RuntimeError: C:\runner\_work\im\src\rtkCudaFFTProjectionsConvolutionImageFilter.cu:83:
ITK ERROR: CUDA ERROR: out of memory

```

My GPU memory is about 23G. Before the error message appeared, the GPU had already used about 10G, indicating that the data had been loaded onto the GPU. According to the error message, at line 83, a new space needs to be created on the GPU, but my GPU memory is running low, so an error occurred.  
In this case, can I use two GPUs? I set `os.environ['CUDA_VISIBLE_DEVICES'] = '0, 1'`, but it doesn’t work.  
Any reply will be appreciate.

---

<div class="post-metadata">

### Author: ![simon.rit](https://discourse.itk.org/letter_avatar_proxy/v4/letter/s/f08c70/32.png) [@simon.rit](https://discourse.itk.org/u/simon.rit)
#### Post date: [April 18, 2024, 7:32am UTC](https://discourse.itk.org/t/itk-error-cuda-error-out-of-memory/6583/2 "2024-04-18T07:32:18Z")

</div>

No, ITKCudaCommon is single GPU, you cannot use two for the computation. The GPU is automatically selected by the Nvidia driver:

> <https://github.com/RTKConsortium/ITKCudaCommon/blob/master/src/itkCudaDataManager.cxx#L27>

---

<div class="post-metadata">

### Author: ![zhangm](https://discourse.itk.org/letter_avatar_proxy/v4/letter/z/34f0e0/32.png) [@zhangm](https://discourse.itk.org/u/zhangm)
#### Post date: [April 18, 2024, 7:41am UTC](https://discourse.itk.org/t/itk-error-cuda-error-out-of-memory/6583/3 "2024-04-18T07:41:00Z")

</div>

Thank you for your reply. So I can only solve this problem by reducing the data size. Also, may I ask if in Python, in order to ensure smooth FDK reconstruction, the size of the GPU must be at least twice the size of the data? There seems to be no such limitation in the use of C++. Is it because Python and C++ load data differently?

---

<div class="post-metadata">

### Author: ![simon.rit](https://discourse.itk.org/letter_avatar_proxy/v4/letter/s/f08c70/32.png) [@simon.rit](https://discourse.itk.org/u/simon.rit)
#### Post date: [April 18, 2024, 9:01am UTC](https://discourse.itk.org/t/itk-error-cuda-error-out-of-memory/6583/4 "2024-04-18T09:01:30Z")

</div>

> [@zhangm](#):
>
> So I can only solve this problem by reducing the data size.

There is a solution: you can stream the reconstruction by pieces. This is done in the C++ code:

> <https://github.com/SimonRit/RTK/blob/master/applications/rtkfdk/rtkfdk.cxx#L201-L208>

Thanks to the fantastic pipeline mechanism of ITK, if you don’t update the projections but only the end of the pipeline (the streaming filter), it will reconstruct the image piece by piece and only load the required parts of the projections. That can be done in Python too.

> [@zhangm](#):
>
> Also, may I ask if in Python, in order to ensure smooth FDK reconstruction, the size of the GPU must be at least twice the size of the data? There seems to be no such limitation in the use of C++. Is it because Python and C++ load data differently?

There shouldn’t be any difference if the codes are similar. There must be a difference, maybe an extra `Update()`.

---

<div class="post-metadata">

### Author: ![zhangm](https://discourse.itk.org/letter_avatar_proxy/v4/letter/z/34f0e0/32.png) [@zhangm](https://discourse.itk.org/u/zhangm)
#### Post date: [April 19, 2024, 12:51am UTC](https://discourse.itk.org/t/itk-error-cuda-error-out-of-memory/6583/5 "2024-04-19T00:51:35Z")

</div>

> [@simon.rit](#):
>
> Thanks to the fantastic pipeline mechanism of ITK

Thank you for your advice. I have this question beacause with the same data C++ can reconstruction correctly even though the GPU memory is 15.9G. I will check the difference between my code.
