The aim of this study was to investigate factors associated with sarcopenia among elderly patients with poorly controlled diabetes mellitus (DM). We retrospectively analyzed 41 patients with type 2 DM, aged ≥65 years who required diabetes education hospitalization. Patients were classified into two groups according to the presence or absence of a weakened hand grip, and clinical characteristics were compared. Patients with a weakened hand grip ( = 21) scored worse on a mini-mental state examination (24.3 vs. 26.5, = 0.04), showed a higher prevalence of diabetic peripheral neuropathy (76% vs. 40%, = 0.03), and had a higher serum phosphorus concentration (3.8 vs. 3.3 mg/dL, < 0.01) compared to those without a weakened hand grip ( = 20). The serum phosphorus concentration was inversely correlated to hand grip strength ( = -0.501, < 0.001) among the total of 41 patients. This inverse association was also confirmed after adjusting the effects of estimated glomerular filtration rate, age, and glycated hemoglobin. Thus, cognitive impairment, diabetic peripheral neuropathy, and high serum phosphorus concentrations are associated with hand grip weakness in elderly patients with type 2 DM.