Caloric restriction (CR) is now a popular lifestyle choice due to its ability in experimental animals to improve lifespan, reduce body weight, and lessen oxidative stress. However, more and more emerging evidence suggests this treatment requires careful consideration because of its detrimental effects on the skeletal system. Experimental and clinical studies show that CR can suppress bone growth and raise the risk of fracture, but the specific mechanisms are poorly understood. Reduced mechanical loading has long been thought to be the primary cause of weight loss-induced bone loss from calorie restriction. Despite fat loss in peripheral depots with calorie restriction, bone marrow adipose tissue (BMAT) increases, and this may play a significant role in this pathological process. Here, we update recent advances in our understanding of the effects of CR on the skeleton, the possible pathogenic role of BMAT in CR-induced bone loss, and some strategies to mitigate any potential side effects on the skeletal system.