The infant gut microbiota affects childhood health. This pioneer microbiota may be vulnerable to antibiotic exposures, but could be supported by prebiotic oligosaccharides found in breast milk and some infant formulas. We sought to characterize the effects of several exposures on the neonatal gut microbiota, including human milk oligosaccharides (HMOs), galacto-oligosaccharides (GOS), and infant/maternal antimicrobial exposures. We profiled the stool microbiota of 1023 one-month-old infants from the KOALA Birth Cohort using 16S rRNA gene amplicon sequencing. We quantified 15 HMOs in breast milk from the mothers of 220 infants, using high-performance liquid chromatography-mass spectrometry. Both breastfeeding and antibiotic exposure decreased gut microbial diversity, but each was associated with contrasting shifts in microbiota composition. Other factors associated with microbiota composition included C-section, homebirth, siblings, and exposure to animals. Neither infant exposure to oral antifungals nor maternal exposure to antibiotics during pregnancy were associated with infant microbiota composition. Four distinct groups of breast milk HMO compositions were evident, corresponding to maternal Secretor status and Lewis group combinations defined by the presence/absence of certain fucosylated HMOs. However, we found the strongest evidence for microbiota associations between two non-fucosylated HMOs: 6′-sialyllactose (6′-SL) and lacto-N-hexaose (LNH), which were associated with lower and higher relative abundances of , respectively. Among 111 exclusively formula-fed infants, the GOS-supplemented formula was associated with a lower relative abundance of . In conclusion, the gut microbiota is sensitive to some prebiotic and antibiotic exposures during early infancy and understanding their effects could inform future strategies for safeguarding a health-promoting infant gut microbiota.