The immune microenvironment plays a pivotal role in cancer development and progression. Therefore, we studied the status of immune cells in esophageal adenocarcinoma (EAC) and adjacent Barrett’s esophagus (BE) and their association with the clinical course of patients. We included 87 patients with EAC who underwent surgical resection or endoscopic submucosal dissection. CD3, CD8, Foxp3, p53, and Ki-67 were immunolocalized in EAC and adjacent BE (N = 87) and BE without EAC (N = 13). BE adjacent to EAC exhibited higher CD3+ lamina propria lymphocyte (LPL) numbers than BE without EAC. Abundant Foxp3+ LPLs in BE were associated with dysplasia and increased Ki-67 labeling index (LI) in BE glandular cells and tended to link to aberrant p53 expression. Abundant CD8+ LPLs in adjacent BE were associated with worse prognosis of EAC patients (P = 0.019). Results of our present study firstly revealed the potential influence of the tissue immune microenvironment of BE adjacent to EAC on cancer development and eventual clinical outcome of EAC patients. T cell infiltration could play pivotal roles in facilitating the dysplasia-adenocarcinoma sequence in BE. The number of Foxp3+ T cells is increased at the early stage of carcinogenesis and could help identify patients harboring dysplastic and highly proliferating cells. CD8+ T cells could reflect unfavorable inflammatory response in adjacent tissue microenvironment and help predict worse prognosis of EAC patients.