We report a method to reduce the detection delay temperature drift for a single-photon detector based on the avalanche photodiode (SPAD). Both the SPAD and the comparator were temperature stabilized, resulting in an ultra-low temperature drift at 0.01 ps/°C. A stable time deviation as 0.15 ps over 1000 s was realized, while the ambient temperature fluctuated rapidly from 24°C to 44°C. To the best of our knowledge, this is the first report on the ultra-stable delay SPAD detector in the case of rapid increase or decrease of ambient temperature. It is helpful to improve the stability of onboard detectors for optical laser time transfer between ground and space.