Is there any relevant content on CBCT system calibration in RTK?

Recently, while studying CBCT calibration, I can only calculate the projection matrix from the position of the steel ball in one projection image and the mold. I can calculate a projection matrix for each projection image, but how can I combine multiple projection images to calculate?

Sorry but I do not fully understand the question. There should be one projection matrix (and image) per source position. RTK’s geometry object, rtk::ThreeDCircularProjectionGeometry can store all them, simply use AddProjection for each one. How would you like to combine them?

What I want to know is how to obtain high-precision geometric information of my CBCT system. I have a set of 3D points and 2D projection points, and what should I do to obtain accurate circular geometric information

There is no such thing in RTK. You can probably estimate the projection matrix which minimizes the distance between projected 3D points and measured 2D points. That should not be too difficult.
Once you have estimated the projection matrix, you can use it with AddProjection.