Congenital heart defects (CHD) is one of the most common types of birth defects. Thanks to advances in surgical techniques and intensive care, the majority of children with severe forms of CHD survive into adulthood. However, this increase in survival comes with a cost. CHD survivors have neurological functioning at the bottom of the normal range. A large spectrum of central nervous system dysmaturation leads to the deficits seen in critical CHD. The heart develops early during gestation, and CHD has a profound effect on fetal brain development for the remainder of gestation. Term infants with critical CHD are born with an immature brain, which is highly susceptible to hypoxic-ischemic injuries. Perioperative blood flow disturbances due to the CHD and the use of cardiopulmonary bypass or circulatory arrest during surgery cause additional neurological injuries. Innate patient factors, such as genetic syndromes and preterm birth, and postoperative complications play a larger role in neurological injury than perioperative factors. Strategies to reduce the disability burden in critical CHD survivors are urgently needed.