Breast cancer has the second highest frequency of death rate among women worldwide. Early-stage prevention becomes complex due to reasons unknown. However, some typical signatures like masses and micro-calcifications upon investigating mammograms can help diagnose women better. Manual diagnosis is a hard task the radiologists carry out frequently. For their assistance, many computer-aided diagnosis (CADx) approaches have been developed. To improve upon the state of the art, we proposed a deep ensemble transfer learning and neural network classifier for automatic feature extraction and classification. In computer-assisted mammography, deep learning-based architectures are generally not trained on mammogram images directly. Instead, the images are pre-processed beforehand, and then they are adopted to be given as input to the ensemble model proposed. The robust features extracted from the ensemble model are optimized into a feature vector which are further classified using the neural network (nntraintool). The network was trained and tested to separate out benign and malignant tumors, thus achieving an accuracy of 0.88 with an area under curve (AUC) of 0.88. The attained results show that the proposed methodology is a promising and robust CADx system for breast cancer classification. Graphical Abstract Flow diagram of the proposed approach. Figure depicts the deep ensemble extracting the robust features with the final classification using neural networks.