Generally, the existence of surface defects in hot-rolled steel strip can lead to adverse influences on the appearance and quality of industrial products. Therefore, it is significant to timely recognize the surface defects for hot-rolled steel strip. In order to improve the efficiency and accuracy of surface defects, a deep neural network, namely, deep attention residual convolutional neural network (DARCNN), is proposed to automatically distinguish 6 kinds of hot-rolled steep strip surface defects. In this network, a channel attention mechanism is combined with residual blocks so that the network can focus on the significant feature channels without information loss. The experimental results show that the accuracy, precision and area under curve (AUC) of DARCNN reach 99.5%, 99.51% and 99.98%, respectively, and the application of DARCNN can improve the accuracy, precision and AUC for surface defect recognition tasks by 1.17%, 1.03% and 0.58%, respectively, which verifies the applicability of deep learning technologies to materials.