Frequency hopping (FH) technology is one of the most effective technologies in the field of radio countermeasures, meanwhile, the recognition of FH signal has become a research hotspot. FH signal is a typical non-stationary signal whose frequency varies nonlinearly with time and the time-frequency analysis technique provides a very effective method for processing this kind of signal. With the renaissance of deep learning, methods based on time-frequency analysis and deep learning are widely studied. Although these methods have achieved good results, the recognition accuracy still needs to be improved. Through the observation of the datasets, we found that there are still difficult samples that are difficult to identify. Through further analysis, we propose a horizontal spatial attention (HSA) block, which can generate spatial weight vector according to the signal distribution, and then readjust the feature map. The HSA block is a plug-and-play module that can be integrated into common convolutional neural network (CNN) to further improve their performance and these networks with HSA block are collectively called HANets. The HSA block also has the advantages of high recognition accuracy (especially under low SNRs), easy to implant, and almost no influence on the number of parameters. We verified our method on two datasets and a series of comparative experiments show that the proposed method achieves good results on FH datasets.