Optical cryptanalysis is essential to the further investigation of more secure optical cryptosystems. Learning-based attack of optical encryption eliminates the need for the retrieval of random phase keys of optical encryption systems but it is limited for practical applications since it requires a large set of plaintext-ciphertext pairs for the cryptosystem to be attacked. Here, we propose a two-step deep learning strategy for ciphertext-only attack (COA) on the classical double random phase encryption (DRPE). Specifically, we construct a virtual DRPE system to gather the training data. Besides, we divide the inverse problem in COA into two more specific inverse problems and employ two deep neural networks (DNNs) to respectively learn the removal of speckle noise in the autocorrelation domain and the de-correlation operation to retrieve the plaintext image. With these two trained DNNs at hand, we show that the plaintext can be predicted in real-time from an unknown ciphertext alone. The proposed learning-based COA method dispenses with not only the retrieval of random phase keys but also the invasive data acquisition of plaintext-ciphertext pairs in the DPRE system. Numerical simulations and optical experiments demonstrate the feasibility and effectiveness of the proposed learning-based COA method.