Research preview Nutrition5k ↗ OmniFood8K ↗

NutriVision

Ingredient-aware hierarchical frequency fusion for single-image food nutrition estimation.

From a single RGB photo — plus an optional ingredient list — we estimate calories, mass, fat, carbs, and protein.

Try the live demo See results
13.52%Nutrition5k mean PMAE
10.89%OmniFood8K mean PMAE
5nutrients per image
≈3 sper prediction
Research preview · NutriVision

Single-image food nutrition estimation, with ingredient-aware frequency fusion.

Abstract

From one RGB plate photo we predict calories, mass, fat, carbs, protein — no depth sensor, no rig. A dual ConvNeXt-Base + IC-FAFM frequency fusion + IA-MPH head, conditioned on CLIP-encoded ingredients.

13.52 % Nutrition5k (new SOTA) · 10.89 % OmniFood8K (closes the gap with DA3-estimated depth).

Why this is hard

From one flat photo, both unknowns must come back:

  • What is on the plate  (ingredients).
  • How much of it  (3-D volume × density).

Geometry returns via estimated depth; semantics via ingredient text — fused in the frequency domain.

Contributions

  1. IC-FAFM — depth → low band · RGB → high band · CLIP-gated mix.
  2. IA-MPH — ingredient-gated head, 5 nutrients jointly.
  3. Two-dataset SOTA  13.52 % N5k · 10.89 % Omni.
  4. Single-image inference — RGB + ingredient text; depth estimated via DA3 (precomputed).
Method

Architecture

RGB and DA3 depth pass through a dual ConvNeXt-Base + ISM stack; CLIP-encoded ingredient text conditions both the frequency-domain fusion (IC-FAFM) and the prediction head (IA-MPH).

NutriVision architecture diagram
1

SSRA

Refines DA3 monocular depth via global scale-shift + a local CNN residual.

2

Dual ConvNeXt-B + ISM

Two ImageNet-22k pretrained backbones with window-attention refinement at 4 scales.

3

IC-FAFM

FFT split + ingredient cross-attention fuses RGB and depth at every scale.

4

IA-MPH

Ingredient-gated channel selection + cross-attention head predicts 5 nutrients.

Results

Performance on two datasets

Nutrition5k (test split, 5:1 dish-level)

MethodVenueCalMassFatCarbProtMean ↓
Google-Nutrition-rgbdCVPR'2118.818.918.123.820.920.1%
Swin-NutritionFoods'2215.312.522.120.815.417.2%
DPF-NutritionFoods'2314.710.622.620.720.217.8%
OmniFood8KCVPR'2614.110.221.018.918.416.5%
NuNet (real depth)TMM'2512.808.7219.6718.6618.4215.65%
IGSMNetFoods'2512.29.419.118.316.015.0%
IGSMNet (our reimpl.†)—13.611.417.615.514.414.49%
NutriVision (ours)—12.6510.3716.4514.7613.3513.52%

†IGSMNet's full training code was not publicly available; the “our reimpl.” row is our re-implementation of the IGSMNet architecture from its published description, trained under identical conditions to NutriVision. The 0.51 pp gap to IGSMNet's published 15.0% may reflect implementation details (CLIP encoding, alignment schedule, batching) we did not exactly recover.

OmniFood8K (synth → 8K finetune)

MethodSettingCalMassFatCarbProtMean ↓
ConvNeXt (best baseline, Yu et al. Tab. 3)RGB only10.15.610.89.513.39.8%
OmniFood8K (Yu et al., full SSRA+FAFM+MPH)RGB → est. depth8.84.59.58.012.68.6%
NutriVision (ours, synth → 8K finetune, 30 epochs)RGB + ingr + DA3 depth9.47.810.613.113.610.89%
Live demo

Try it on your own photo

Upload a plated-meal photo. Auto-suggest will hint at the ingredients via BLIP + CLIP; you can edit before predicting. Expected error band: ±13.52% PMAE.