Photos help record meals, but measuring them is still hard#
The unsolved problems in automatic diet tracking are not the ones that get demonstrated on stage. Food recognition is further developed than portion and calorie estimation. Working out how much of it is there, what it was cooked in, and whether the whole day added up is not. The durable open problems are portion size, invisible ingredients, and the fact that any single-meal estimate carries error that no current method removes.
The first wave of "new methods" was already reviewed a decade ago. A 2017 review of image-assisted and image-based dietary assessment covered studies from late 2013 to early 2016 and reached a useful conclusion: in every study it examined, adding images reduced underreporting compared with traditional methods1. The same review asked for larger studies to pin down the degree of error. Images can supplement recalled details and help assess portions; the meal you forgot to write down may be in the camera roll.
The best-known validation of that approach shows both halves of the result. In a study of 45 community-dwelling adults who were instructed to photograph every eating occasion for 7.5 days with a mobile food record (40 supplied all 7.5 days), trained analysts estimated energy intake from the images, helped by the fact that much of the food was supplied pre-weighed with known menus. That estimate was compared with energy expenditure measured by doubly labeled water2. Reported intake correlated with expenditure at 0.58, but it averaged 2,353 kcal/day against 2,932 kcal/day expended, a 579 kcal/day gap, or roughly 20% of expenditure (our division). The authors judged that comparable to traditional food records, and the discrepancy did not clearly grow with intake level. Note who did the measuring: people with nutrition training looking at pictures, not software. Automating even that level of accuracy is the open question.
If you arrived from the overview of how accurate calorie counting is, this is the other side of the coin: the tools are changing, but the physics of a plate has not.
Recognizing food is ahead of sizing it#
An automatic photo system does three jobs in sequence: find the food items on the plate (segmentation), name them (classification), and estimate their volume, calories or nutrients. A 2022 systematic review in Advances in Nutrition screened 159 studies of image-based food-recognition systems and included 78 studies3. Forty-five of them (58%) used deep learning in at least one phase.
The review's verdict on the final step was blunt. Volume, calorie and nutrient estimation is "the less addressed phase" of these systems, held back by a lack of suitably annotated datasets and "the difficulty in obtaining depth information from a 2-dimensional image." Using a single photo "does not carry sufficient information" for reliable 3D reconstruction. Two-photo methods run into occlusion, and the common assumption that a food item has constant height "may introduce large errors in volume estimation." Many systems ask users to include a coin, a card or a thumb in the frame for scale.
| Pipeline step | State of the research (per the 2022 review) | Why it is hard |
|---|---|---|
| Find items on the plate | Active, improving with deep learning | Mixed and overlapping foods |
| Name each item | Most studied; strong on large public datasets | Lighting, regional dishes, lookalike foods |
| Estimate volume and calories | Least addressed | Depth is missing from a flat image; few annotated datasets |
Benchmark accuracy on public datasets also does not transfer automatically to your kitchen. It measures how well a system labels curated photos, not how close its calorie number lands to what you ate. For how that plays out with consumer photo apps, including the ingredients no camera can see, our separate piece on AI photo calorie counters goes into detail.
Hidden ingredients are missing information, not a weak model#
It is tempting to assume the next model generation will close the gap. Some errors can likely be reduced. Recognition keeps improving, and better depth sensing on phones may help with volume. But part of the gap is not a modeling problem at all: oil absorbed into rice, butter melted into a sauce, sugar dissolved in a drink. These leave little or no visual trace, and a picture cannot contain information that was never in the scene.
Blended and poured foods are the extreme case. A smoothie's energy is decided by what went into the blender, which is exactly what the glass hides; our guide to logging smoothies and protein shakes deals with that practically. Text and voice descriptions ("fried in two tablespoons of oil") can supply what the photo lacks, so a mix of inputs is plausibly more informative than any one input. That is a reasonable expectation, not a demonstrated result: the studies cited here do not establish whether combining photos and descriptions improves individual accuracy.
Passive capture does not escape this either. Wearables that detect eating and then take pictures, reviewed in our piece on automatic intake detection, can help detect eating occasions and trigger the camera, but they inherit the camera's problems with "how much."
The bathroom scale is a second instrument#
A different line of research sidesteps the plate entirely. If you know how a body's energy stores change over time, you can back-calculate how much energy must have come in.
Dynamic energy-balance modeling made this rigorous. A 2011 Lancet model of adult metabolism showed that body weight responds slowly to a sustained change in intake, with half-times of about a year: roughly a year to reach half of the eventual weight change4. That slowness is why a single weigh-in cannot reliably reveal that day's intake, while months of weigh-ins can say something.
A 2015 validation turned the model around. In 140 people from the two-year CALERIE calorie-restriction trial, measured over four intervals, a mathematical model given only baseline demographics and repeated body-weight measurements (and built-in assumptions about energy expenditure) estimated changes in intake to within 40 kcal/day of the expensive reference method (doubly labeled water plus DXA scans) on average5. Across individuals, the overall root-mean-square difference between the two methods was 215 kcal/day.
That is a demonstrated capability with a clear boundary. The weight-based method estimates the change in intake over months, closely for group averages and less closely for individuals. It says nothing about which meal did it. A food log says which meal but carries its own error. The open research problem is how best to combine the two, so that the trend corrects the log and the log explains the trend. Practical versions already exist in the form of adjusting targets from your weight trend; a validated method for doing it per person, in real time, is not established by the studies cited here.
The scale can't tell you what you ate. The log can't tell you what it added up to. The unsolved part is making each check the other.
Separating what is shown from what is promised#
Much of what is written about the future of calorie tracking blurs demonstrated capability with extrapolation. The published record supports a narrower list.
| Demonstrated in peer-reviewed studies | Not established by the studies cited here |
|---|---|
| Photos reduce underreporting versus traditional methods | Fully automatic photo-to-calorie accuracy matching trained analysts in free-living use |
| Trained analysts estimate intake from photos (with supplied, pre-weighed foods) at ~20% below measured expenditure on average | Reliable portion size from a single ordinary phone photo |
| Food classification works well on large public datasets | Detecting cooking fats and dissolved sugars from images |
| Weight-based models track group intake change within ~40 kcal/day | Per-person, real-time fusion of food logs and weight trends |
Calibrated uncertainty is a research aim#
Every validated method above reports substantial error: a 579 kcal/day average shortfall in the photo study, a 215 kcal/day root-mean-square difference for the weight model. These studies show real uncertainty. Some of it can be reduced with better methods; hidden ingredients require additional information that a plate photo does not hold.
That is the case for showing estimates as ranges rather than single numbers, argued in full elsewhere. What remains to be validated is the harder part: ranges that are calibrated, so the true value falls inside them about as often as they claim, and daily logs that are corrected by the slower signal of the scale. Both are research aims, not results in hand.
FAQ#
Will AI eventually count calories perfectly from a photo?#
Not for every meal, on current evidence. Recognition keeps improving, but a single photo lacks depth and cannot show fats or sugars mixed into food. Some uncertainty will remain for any method that only sees the plate.
What is the least studied part of photo-based calorie tracking?#
Estimating volume and calories. A 2022 systematic review of 78 studies called it the least addressed phase, citing missing depth information in flat images and a shortage of annotated datasets.
Can body weight changes reveal how much I really ate?#
Over months, models can estimate changes in intake from repeated weigh-ins, using assumptions about energy expenditure. In one two-year trial, group estimates landed within about 40 kcal/day of the reference method; the overall root-mean-square difference was 215 kcal/day, and the method cannot attribute intake to specific meals.
Sources#
- Boushey CJ, Spoden M, Zhu FM, Delp EJ, Kerr DA. New mobile methods for dietary assessment: review of image-assisted and image-based dietary assessment methods. Proc Nutr Soc. 2017.
- Boushey CJ, et al. Reported energy intake accuracy compared to doubly labeled water and usability of the mobile food record among community dwelling adults. Nutrients. 2017.
- Dalakleidi KV, Papadelli M, Kapolos I, Papadimitriou K. Applying image-based food-recognition systems on dietary assessment: a systematic review. Adv Nutr. 2022.
- Hall KD, et al. Quantification of the effect of energy imbalance on bodyweight. Lancet. 2011.
- Sanghvi A, Redman LM, Martin CK, Ravussin E, Hall KD. Validation of an inexpensive and accurate mathematical method to measure long-term changes in free-living energy intake. Am J Clin Nutr. 2015.
Source: BurnWeek — "Unsolved problems in automatic diet tracking", https://burnweek.fit/blog/future-of-calorie-tracking/. Licensed CC BY 4.0: free to quote or reuse with a link to this page.
