In this work, we propose and verify experimentally a model that describes the concomitant influence of the beam size and optical roughness on the temporal contrast of optical pulses passing through a pulse stretcher in chirped-pulse amplification laser systems. We develop an analytical model that is capable of predicting the rising edge caused by the reflection from an optical element in a pulse stretcher, based on the power spectral density of the surface and the spatial beam profile on the surface. In an experimental campaign, we characterize the temporal contrast of a laser pulse that passed through either a folded or an unfolded stretcher design and compare these results with the analytical model. By varying the beam size for both setups, we verify that optical elements in the near- and the far-field act opposed to each with respect to the temporal contrast and that the rising edge caused by a surface benefits from a larger spatial beam size on that surface.