Building and functioning of the human brain requires the precise orchestration and execution of myriad molecular and cellular processes, across a multitude of cell types and over an extended period of time. Dysregulation of these processes affects structure and function of the brain and can lead to neurodevelopmental, neurological, or psychiatric disorders. Multiple environmental stimuli affect neural stem cells (NSCs) at several levels, thus impairing the normal human neurodevelopmental program. In this review article, we will delineate the main mechanisms of infection adopted by several neurotropic pathogens, and the selective NSC vulnerability. In particular, TORCH agents, i.e., , others (including Zika virus and Coxsackie virus), Rubella virus, Cytomegalovirus, and Herpes simplex virus, will be considered for their devastating effects on NSC self-renewal with the consequent neural progenitor depletion, the cellular substrate of microcephaly. Moreover, new evidence suggests that some of these agents may also affect the NSC progeny, producing long-term effects in the neuronal lineage. This is evident in the paradigmatic example of the neurodegeneration occurring in Alzheimer’s disease.