Unwanted light does not come from one mechanism. Ghosting from specular reflection at surfaces, reflection and transmission at coatings, scattering at rough surfaces and mechanical parts, and multiple and total internal reflection all overlap so that light not travelling the designed path reaches the image plane. The model therefore has to include the barrel and mechanical parts as well as the optical elements, and you also confirm that chief and marginal rays pass the apertures as intended.
Scattering is given as a bidirectional scattering distribution function. Per surface, choose from Lambertian scattering, Gaussian scattering, the ABg model, tabulated scattering data and user-defined models. The ABg model is written as a function of direction cosines, a form suited to scattering from isotropic surface roughness.
From the theory side, scattering at a rough surface can be formulated as a linear system in direction-cosine space. Under the condition that the surface is smooth enough, the scattering function is proportional to the power spectral density of the surface roughness. The translational invariance of that formulation does not hold universally, however, and it has been shown that a different transfer function is needed for each angle of incidence. Scattering characteristics measured at one angle of incidence cannot simply be extrapolated to another.