Measuring “home time,” number of days not in facility-based care, with medical claims is a promising approach to assess person-centered outcomes on a population level. Generally, spending more time at home matches long-term care preferences and improves quality of life. However, existing “home time” measures have not incorporated key stakeholder perspectives. We sought to understand how patients and family caregivers value time spent in diverse facility-based health care settings (Emergency Department, Nursing Home, Post-Acute Care/Skilled Nursing, Inpatient Hospital) to help determine whether various settings have different effects on quality of life and thus merit different weighting in a “home time” measure. We conducted three focus groups among patients and family caregivers within the U.S. Veterans Health Care System. We identified themes pertaining to patients’ quality of life in each of the four facility-based care settings. Discussions about both emergency department and post-acute/skilled nursing care reflected loss of personal control, counterbalanced by temporary stay. Inpatient hospital care evoked discussion about greater loss of personal control due to the intensity of care. Nursing homes ultimately signified decline. These findings illuminate differences in quality of life across health-care settings and help justify the need for different weights in a measure of “home time.”