Smartphone photos move into body composition research
Google Research has presented PhotoScan, an investigational deep learning approach that estimates body composition from standard smartphone images. In a clinical research setting described by Google, the system predicted insulin resistance with performance close to DXA-based features, while remaining far from a consumer diagnostic product. The result points to a possible screening signal for cardiometabolic research, not a replacement for clinical testing.The research target: risk that BMI can miss
PhotoScan is aimed at a narrow but clinically meaningful problem: body shape and fat distribution can carry information that body mass index alone does not capture. Google Research describes insulin resistance as a critical and underdiagnosed driver of metabolic disease that can appear years before type 2 diabetes is diagnosed.The study uses HOMA-IR, a fasting glucose and insulin-based measure, with a score above 2.9 treated as insulin resistant according to epidemiological reviews cited by Google. The body composition features emphasized in the work include body fat percentage, the android-to-gynoid fat ratio and the visceral-to-subcutaneous fat area ratio. The implication is that where fat is stored may matter as much as how much fat is present when assessing early cardiometabolic risk.
How PhotoScan estimates body composition
PhotoScan estimates three-dimensional body composition metrics from ordinary two-dimensional smartphone imagery. Google says the investigational framework uses standard frontal and lateral views to infer body fat percentage, A/G ratio and V/S ratio.The model was built with a ResNet-50 backbone initialized with ImageNet weights, then trained to predict body composition against DXA ground truth. During pre-training, Google used frontal and lateral projection images generated from 3D MRI scans and fused image features with sex, height, weight and an internal BMI measure. In plain terms, the system is not just reading a scale number; it is learning geometry-linked signals from images and mapping them to clinical composition measurements.
Three cohorts shaped the experiment
Google reports three stages: pre-training, fine-tuning and validation. The pre-training set used 35,323 UK Biobank participant records with MRI-derived projection images and DXA-based ground truth.The fine-tuning stage used the PhotoBIA cohort of 677 adults, pairing real-world smartphone photos with DXA measurements. Google says an automated landmark detection pipeline selected frontal and lateral pose frames from 360-degree participant videos to improve training data. The independent validation stage used 132 participants from the MetabolicMosaic cohort, a 30-week longitudinal study in San Francisco with paired DXA, PhotoScan, smartwatch BIA, anthropometric, fasting blood lab and passive Fitbit tracking data.
That structure strengthens the research design because the final test was not merely another slice of the training data. It also keeps the finding bounded: the independent validation cohort was clinically rich but small.
Accuracy compared with BIA and DXA
Google says PhotoScan outperformed smartwatch bioelectrical impedance analysis for body fat percentage in the PhotoBIA cohort and estimated composition features that BIA did not provide. In five-fold cross-validation on PhotoBIA, PhotoScan reached a mean absolute error of 2.15 for body fat percentage, compared with 2.91 for the BIA-based model.For regional composition, Google reports average mean absolute error of 0.107 for A/G ratio and 0.094 for V/S ratio in PhotoBIA. In the independent MetabolicMosaic cohort, the reported errors were 2.13 for body fat percentage, 0.085 for A/G ratio and 0.085 for V/S ratio. Google attributes the lower A/G and V/S error in MetabolicMosaic partly to its higher proportion of female records, which can reduce regional ratio variance.
DXA remains the clinical reference in the article, but Google notes that DXA scans are expensive, require specialized infrastructure and expose patients to low doses of radiation. PhotoScan is presented as a scalable research middle ground, not as a superior replacement for DXA.
Insulin resistance classification results
The strongest news claim is the insulin resistance classifier. Google tested feature sets on the MetabolicMosaic cohort using a gradient boosting classifier and compared demographics alone with demographics plus tape measurements, smartwatch BIA, PhotoScan and DXA.The baseline demographic model, using age, sex and BMI, achieved an AUROC of 0.692. Adding PhotoScan-based body composition features raised AUROC to 0.760 and produced a Net Reclassification Index of 0.593. DXA features reached AUROC of 0.773 and NRI of 0.748. By contrast, Google says adding BIA to demographics produced no improvement in AUROC or NRI for insulin resistance classification.
AUROC measures how well a model separates people with and without insulin resistance. NRI measures how much a new feature set improves classification relative to a baseline. These results suggest that image-estimated regional body composition may add useful signal beyond BMI, though the conclusion depends on further validation outside the described cohorts.
Limits for medicine and privacy
Google explicitly describes PhotoScan as a research prototype. The source does not present it as an approved diagnostic tool, consumer medical feature or substitute for fasting labs and clinician assessment.The method also depends on body imagery, which is inherently sensitive health-adjacent data. Any future deployment would need clear consent, strong privacy handling, fairness testing across body types and clinical workflow validation before it could be used responsibly at scale. For now, the practical takeaway is narrower: smartphone imagery may be able to contribute measurable body composition signals to cardiometabolic research.
Conclusion
PhotoScan is a notable example of medical AI moving from simple proxies toward richer digital phenotyping. Google Research reports that smartphone images can estimate body composition features linked to insulin resistance and that those features nearly matched DXA-based inputs in one independent clinical cohort.The finding is promising but early. The validation cohort was limited, the system remains investigational and the article does not establish a consumer launch or clinical authorization. If future studies confirm the results across broader populations, smartphone-based body composition analysis could become one component of earlier, less invasive metabolic risk screening.
Sources
Editorial Team - CoinBotLab