High-frequency surface wave radar (HFSWR) plays an important role in wide area monitoring of the marine target and the sea state. However, the detection ability of HFSWR is severely limited by the strong clutter and the interference, which are difficult to be detected due to many factors such as random occurrence and complex distribution characteristics. Hence the automatic detection of the clutter and interference is an important step towards extracting them. In this paper, an automatic clutter and interference detection method based on deep learning is proposed to improve the performance of HFSWR. Conventionally, the Range-Doppler (RD) spectrum image processing method requires the target feature extraction including feature design and preselection, which is not only complicated and time-consuming, but the quality of the designed features is bound up with the performance of the algorithm. By analyzing the features of the target, the clutter and the interference in RD spectrum images, a lightweight deep convolutional learning network is established based on a faster region-based convolutional neural networks (Faster R-CNN). By using effective feature extraction combined with a classifier, the clutter and the interference can be automatically detected. Due to the end-to-end architecture and the numerous convolutional features, the deep learning-based method can avoid the difficulty and absence of uniform standard inherent in handcrafted feature design and preselection. Field experimental results show that the Faster R-CNN based method can automatically detect the clutter and interference with decent performance and classify them with high accuracy.