Bone fragility and fracture risk are assessed by measuring the areal bone mineral density (aBMD) using dual-energy X-ray absorptiometry (DXA). While aBMD correlates with bone strength, it is a poor predictor of fragility fracture risk. Alternatively, fracture toughness assesses the bone’s resistance to crack propagation and fracture, making it a suitable bone quality metric. Here, we explored how femoral midshaft measurements from DXA, micro-computed tomography (µCT), and Raman spectroscopy could predict fracture toughness. We hypothesized that ovariectomy (OVX) decreases aBMD and fracture toughness compared to controls and we can optimize a multivariate assessment of bone quality by combining results from X-ray and Raman spectroscopy. Female mice underwent an OVX (n = 5) or sham (n = 5) surgery at 3 months of age. Femurs were excised 3 months after ovariectomy and assessed with Raman spectroscopy, µCT, and DXA. Subsequently, a notch was created on the anterior side of the mid-diaphysis of the femurs. Three-point bending induced a controlled fracture that initiated at the notch. The OVX mice had a significantly lower aBMD, cortical thickness, and fracture toughness when compared to controls (p < 0.05). A leave one out cross-validated (LOOCV) partial least squares regression (PLSR) model based only on the combination of aBMD and cortical thickness showed no significant predictive correlations with fracture toughness, whereas a PLSR model based on principal components derived from the full Raman spectra yielded significant prediction (r = 0.71, p < 0.05). Further, the PLSR model was improved by incorporating aBMD, cortical thickness, and principal components from Raman spectra (r = 0.92, p < 0.001). This exploratory study demonstrates combining X-ray with Raman spectroscopy leads to a more accurate assessment of bone fracture toughness and could be a useful diagnostic tool for the assessment of fragility fracture risk.