Atrial fibrillation (AF) is a common heart rhythm which occurs when the upper chambers of the heart beat irregularly. With the rapid development of the deep learning algorithm, the Convolutional Neural Networks (CNN) is widely investigated for the ECG classification task. However, for AF detection, the performance of CNN is greatly limited due to the lack of consideration for temporal characteristic of the ECG signal. In order to improve the discriminative ability of CNN, we introduce the attention mechanism to help the network concentrate on the informative parts and obtain the temporal features of the signals. Inspired by this idea, we propose a temporal attention block (TA-block) and a temporal attention convolutional neural network (TACNN) for the AF detection tasks. The TA-block can adaptively learn the temporal features of the signal and generate the attention weights to enhance informative features. With a stack architecture of TA-blocks, the TA-CNN obtains better performance as a result of paying more attention to the informative parts of the signal. We validate our approach on the single lead ECG classification dataset of The PhysioNet Computing in Cardiology Challenge 2017. The experimental results indicate that the proposed framework outperform state-of-the-arts classification networks.Clinical Relevance-The proposed algorithm can be potentially applied to the portable cardiovascular monitoring devices reducing the danger of AF.