Trials in baby formula rife with bias and lacking in transparency

Regulators assessing the safety of new formula milk products can’t place much faith in trial results, according to a systematic review of medical literature on infant formula studies, which found that the bulk of these trials lacked transparency, contained selective reporting, and were riddled with biases and industry connections.

Formula milks are consumed on a daily basis by the majority of infants in Europe and North America, representing up to 20% of a baby’s body weight per day and contributing the majority of the child’s nutrition. If used in place of breastfeeding, they have the potential to confer both short- and long-term health risks, “from burns, colic, and constipation to pneumonia, developmental effects, and fatal diarrhea,” Robert J. Boyle, PhD, of the National Heart and Lung Institute at Imperial College London and the Center of Evidence Based Dermatology at the University of Nottingham in Nottingham, UK, and colleagues wrote in The BMJ.

As such, regulators rely on clinical trial data in order to determine the safety of new products and back up the claims of formula manufacturers. However, though many representatives from the industry emphasize their investments in top-line scientific research to develop their products, several regulatory and academic groups have raised concerns over the conduct and reporting of these trials.

Boyle and colleagues conducted their systematic review to assess the conduct and reporting of clinical trials of formula milk products, with a particular focus in the risk of bias in published results and whether trial procedures undermined participants’ ability to breastfeed and therefore, potentially harm participants.

“Our study suggests that formula trials are not reliable and might not adequately protect participants in the trials,” they wrote. “The formula industry is closely involved in formula trials, findings are almost always reported as favourable, and little transparency exists about the aims of the trial or reporting of results. Our findings support the need for a substantial change in the conduct and reporting of formula trials to adequately protect participants from harm and protect consumers from misleading information.”

For their analysis, Boyle and colleagues searched Medline, Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) for clinical trials that compared two or more formula products for children less than 3 years of age that were published from Jan. 1, 2015–Dec. 31, 2020; they also looked for trials registered on the International Clinical Trials Registry Platform and the stuatus of trial registration of all trials published since Jan. 1, 2006, in order to assess the risk of selective reporting at the trial level. Trials were excluded if they focused on supplements given separately to formula, breast milk/fortifiers of breast milk, or comparisons of frequency/volume of consumption with no change in ingredients or formulation.

The review’s primary outcome was overall risk of bias for the trial’s primary outcome, as determined by the Cochrane risk of bias version 2.0 (ROB2); secondary outcomes included risk of bias in individual domains of ROB2, how favorable the primary outcomes and conclusions of the trial were, measures of selective reporting at the trial level, and risk of trial procedures undermining breastfeeding.

“22,201 titles were screened and 307 trials were identified that were published between 2006 and 2020, of which 73 (24%) trials in 13,197 children were prospectively registered,” the study authors wrote. “Another 111 unpublished but registered trials in 17,411 children were identified. Detailed analysis was undertaken for 125 trials (23,757 children) published since 2015.”

Of the 125 trials published from 2015-2020, the review authors found that:

  • Only 17 (14%) were conducted independently of formula companies; 26 (21%) were prospectively registered with a clear aim and primary outcome; and the authors/sponsors only shared prospective protocols for 11 (9%) trials.
  • Risk of bias was deemed to be low in five (4%) of trials and high in 100 (80%) of these trials.
    • Four trials (3%) were at high risk of bias due to inadequate concealment of allocation at randomization.
    • Fifty two trials (42%) were at high risk due to deviations from intended interventions: of these, 32 trials evaluating adherence used a per protocol analysis that excluded ≥10% of participants without accounting for missing data in the analysis; the remaining 20, which evaluated assignment, inappropriately excluded ≥10% of participants from the analysis due to adverse events or non-adherence.
    • Eighteen trials (14%) were at high risk due to missing 10% or more of the trial outcome data “with no evidence that the estimate was stable with methods such as multiple imputation, and a high likelihood that missingness in the outcome data depended on their true value.”
    • Five trials (4%) were at high risk in outcome measurements due to assessors not being blinded and outcomes were subjective.
    • Sixteen trials (13%) were at high risk due to selective reporting, usually because of “analysis of an inappropriate dataset for the stated aim.
  • In 83 (66%) trials, there were some concerns about selective reporting because prospective trial registration was absent, multiple primary outcomes or evaluation methods were registered, methods for measurement or analysis of primary outcomes were poorly specified, or reporting was incomplete.”

In addition, primary outcomes were reported by authors as favorable in 86 trials (69%), while 115 abstract conclusions (92%) were also favorable—and, in a subset of 68 recently published superiority trials, the publication favored the intervention formula in 57 (84%), the control formula in eight (12%), and neither product in three (4%).

Troublingly, only one of 38 trials (3%) in partially breastfed infants reported adequate support for breastfeeding, while only 14 of 87 trials (16%) in non-breastfed infants confirmed the decision not to breastfeed was firmly established prior to trial enrollment.

Boyle and colleagues argued that their findings establish “an almost universal lack of transparency, and evidence of selective reporting between and within trials.”

Selective reporting had a particular impact in the favorability of outcomes, they noted: “almost half of the recent trials designed to show adequate weight gain based on regulatory guidance failed to show equivalence, but authors’ own interpretation of the data supported adequate weight gain in 95% of trials. In most recent formula trials, investigators were employed by, or had financial links to, the formula industry, who were often involved in the statistical analysis and writing. Conflicts of interest could explain some of the transparency issues identified in formula trials because work in other disciplines has shown a relation between commercial funding for clinical trials and favorable results.”

Based on these results, Boyle and colleagues concluded that a substantial amount of the current information about formula products may be misleading.

“Given the lack of transparency about the aims of the trials and almost universally favorable conclusions, some trials might have a marketing aim and no robust scientific aim,” they wrote. “…Some have called for a change in the regulatory environment for formula products, including a ban on health and nutrition claims. Our findings suggest that such changes should include improved oversight, conduct, and reporting of formula trials to ensure they provide a rigorous evidence base to inform nutrition in infants and young children.”

Study limitations included an inability to fully analyze all trials completed during the study period because “many… were unpublished or unregistered, trial authors and funders were reluctant to share protocols, and regulators could not share clinical study reports;” they did not include formal patient and public involvement in the conduct or reporting of this review; they did not formally compare the risk of bias in this area with clinical trials on other topics; and their analysis of primary outcomes likely underestimates reporting biases compared with exploratory outcomes or post hoc subgroup analyses, “which are often used by the formula industry as the basis of health and nutrition claims.”

  1. A systematic review of medical literature on infant formula studies found that most of these trials lacked transparency, contained selective reporting, and were full of bias, potentially undermining the reliability of these trials.

  2. These findings support the need for a substantial change in the conduct and reporting of formula trials to adequately protect participants from harm and protect consumers from misleading information.

John McKenna, Associate Editor, BreakingMED™

This work was funded by Imperial Health Charity.

Boyle received personal fees from Cochrane, DBV Technologies, and Prota Therapeutics, and from expert witness work in cases of food anaphylaxis and class actions related to infant formula health claims, outside the submitted work, and received personal fees from Public Health England as a member of the UK Nutrition and Health Claims Committee and the Maternal and Child Nutrition Subgroup of the Scientific Advisory Committee on Nutrition.

Cat ID: 138

Topic ID: 85,138,191,138,43,192,925

Author