Non-line-of-sight (NLOS) imaging has emerged as a prominent technique for reconstructing obscured objects from images that undergo multiple diffuse reflections. This imaging method has garnered significant attention in diverse domains, including remote sensing, rescue operations, and intelligent driving, due to its wide-ranging potential applications. Nevertheless, accurately modeling the incident light direction, which carries energy and is captured by the detector amidst random diffuse reflection directions, poses a considerable challenge. This challenge hinders the acquisition of precise forward and inverse physical models for NLOS imaging, which are crucial for achieving high-quality reconstructions. In this study, we propose a point spread function (PSF) model for the NLOS imaging system utilizing ray tracing with random angles. Furthermore, we introduce a reconstruction method, termed the physics-constrained inverse network (PCIN), which establishes an accurate PSF model and inverse physical model by leveraging the interplay between PSF constraints and the optimization of a convolutional neural network. The PCIN approach initializes the parameters randomly, guided by the constraints of the forward PSF model, thereby obviating the need for extensive training data sets, as required by traditional deep-learning methods. Through alternating iteration and gradient descent algorithms, we iteratively optimize the diffuse reflection angles in the PSF model and the neural network parameters. The results demonstrate that PCIN achieves efficient data utilization by not necessitating a large number of actual ground data groups. Moreover, the experimental findings confirm that the proposed method effectively restores the hidden object features with high accuracy.