You cannot expect faster loading than about 50-100 slices/second for the most common legacy DICOM files (single slice per file). If you want to load an image series in a fraction of a second then you can use more efficient file formats, for example DICOM enhanced multiframe or research formats (such as NRRD).
I’ve done a quick performance test of DICOM import in 3D Slicer. We regularly perform profiling of DICOM import and fix any obvious performance bottlenecks, so while probably the performance could be improved, probably not too much and not too easily. In this test, I’ve used TCIA CPTAC-HNSCC collection if 20256 DICOM files (most of them is a single MRI slice, there are some scout images and reports), 10.7GB in total.
In Slicer we first read the entire collection of DICOM files we want to process. This is important, because DICOM is not aware of filenames and folders, so you cannot rely on having a complete series in a folder (and there can be UID-based links between series, so you cannot always interpret a series by itself anyway).
Indexing of all the data sets took 332sec (cold loading, after computer restart), 290sec (warm loading, right after the same 10GB data set was loaded from disk previously) in total, which means processing about 60 files (about 30MB) per second.
After this, cold loading of a 600-slice MRI takes about 12 seconds (of this, about 1-2 second is spent in geometry consistency check - retrieving all slice position and orientation from the DICOM database and create warping transform if they are not equally spaced, do not have the same orientation, or axes are not orthogonal). Warm loading (loading the file again, just after it was loaded) took about 5 seconds. Speed was about the same with DCMTK and GDCM.
Overall, it looks like that with standard commodity hardware and non-optimized system configuration you can parse and load DICOM at about a rate of about 50-100 files/sec (something like 30-60MB/sec). So, the performance that you have looks about right. Maybe you can upgrade your hardware and with a lot of effort improve your software and get maybe up to 2-5x faster loading, but if you need significantly faster speeds then it is better to switch to a more efficient file format. Modern DICOM (enhanced multiframe files) store the entire series in a single file, so you can expect much better performance there. You can also use simpler file formats, for example, the same MR series (320 MB, 600 slices) that takes 12 seconds to load from DICOM, can be loaded in 0.3 seconds from uncompressed NRRD.