Electroencephalography (EEG)-based BCIs have experienced a significant growth in recent years, especially the passive Brain Computer Interfaces (BCIs) with a wide application in the detection of cognitive and emotional states. But it is still unclear whether more subtle states, e.g., covert selective attention can be decoded with EEG signals. Here we used a behavioral paradigm to introduce the shift of selective attention between the visual and auditory domain. With EEG signals, we extracted features based on Grange Causality (GC) and successfully decoded the attentional shift through a support vector machine (SVM) based classifier. The decoding accuracy was significantly above the chance level for all 8 subjects tested. The features based on GC were further analyzed with tree-based feature importance analysis and recursive feature elimination (RFE) method to search for the optimal features for classification. Our work demonstrate that specific patterns of brain activities reflected by GC can be used to decode subtle state changes of the brain related to cross-modal selective attention, which opens new possibility of using passive BCIs in sophisticated perceptual and cognitive tasks.