Preterm birth is an extreme environmental stress associated with an increased risk of later cognitive dysfunction and mental health problems. However, the extent to which preterm birth is modulated by genetic variation remains largely unclear. Here, we test for an interaction effect between psychiatric polygenic risk and gestational age at birth on cognition at age four. Our sample comprises 4934 unrelated individuals (2066 individuals born < 37 weeks, 918 born <  = 34 weeks). Genome-wide polygenic scores (GPS's) were calculated for each individual for five different psychiatric pathologies: Schizophrenia, Bipolar Disorder, Major Depressive Disorder, Attention Deficit Hyperactivity Disorder and Autism Spectrum Disorder. Linear regression modelling was used to estimate the interaction effect between psychiatric GPS and gestational age at birth (GA) on cognitive outcome for the five psychiatric disorders. We found a significant interaction effect between Schizophrenia GPS and GA (β = 0.038 ± 0.013, p = 6.85 × 10) and Bipolar Disorder GPS and GA (β = 0.038 ± 0.014, p = 6.61 × 10) on cognitive outcome. Individuals with greater genetic risk for Schizophrenia or Bipolar Disorder are more vulnerable to the adverse effects of birth at early gestational age on brain development, as assessed by cognition at age four. Better understanding of gene-environment interactions will inform more effective risk-reducing interventions for this vulnerable population.