Solar speckle images acquired by ground-based optical telescope usually are blurred or degraded seriously with more noise and fuzzy local details. Most of the deep learning deblurring methods are suitable for the natural images which have sufficient contextualized and gradient information, but for solar speckle images may cause some problems such as over-smoothing, high-frequency loss, and artifacts generated. In this paper, we propose a deblurring method based on coupling double-stage feature pyramid networks (FPN) with a single pipe-line (DSFSP) to reconstruct high-resolution solar speckle images. In stage1, one FPN is used to recover structure features; In stage2, another FPN is used to enhance the structural contextualized, and the single pipe-line coupled with this FPN is used to extract gradient information. After fusing these to generate a reconstructed image, discriminators are used to make it closer to the reference. Experiments show that DSFSP has a strong ability to strengthen the gradient spatial and contextualized information, improve image clarity, restore high-frequency details and drop artifacts. © 2021, Springer Nature Switzerland AG.