The permanently shadowed regions (PSRs) of the Moon are located at the Moon's polar regions that are permanently in shadow due to their inability to receive direct sunlight. Images of these areas are usually dark and significantly affected by noise, obscuring the lunar terrain information. Although image denoising has made considerable progress, there is still limited study on images denoising of lunar PSRs. The main challenge lies in the fact that images of PSRs are characterized by low contrast, complex noise type, and uneven illumination. The existing deep learning-based methods exhibit poor physical interpretability and cannot effectively remove complex noise. Therefore, this study introduces a novel denoising method by using combination of physical noise models and deep Learning. Specially, the physical noise model is used to simulate the noise of lunar PSRs according to the imaging principles of the lunar reconnaissance orbiter camera narrow angle camera. The improved deep learning model, which incorporates full-scale skip connections and Transformer is used to denoise the images. The proposed method is tested in 297 PRSs images with latitudes below -70 degrees and compared with state-of-the-art methods. Experimental results demonstrate that the proposed method outperforms existing methods in restoring terrain details and provides better quantitative and visual outcomes. This approach has the potential to improve the clarity of lunar PSR images and support future lunar exploration.