Accuracy in SugarSnap

SugarSnap aims to provide carbohydrate estimates that are as accurate as possible.

The latest version of SugarSnap has been tested against a difficult set of over 3,000 meal images, and 91% of snaps returned a carb estimate within 15 grams of the actual carbohydrates.

For comparison, we tested ChatGPT with the same image set, and 79% of results were within 15 grams.

We’re really pleased with this result, particularly given the high degree of difficulty with these images. The images typically don’t have an obvious scale reference (such as cutlery) and there are a lot of ambiguous and partially-hidden ingredients.

Here are some example images from the set:

Example images from the test set

Where did the test set images come from?

The meal images were originally captured by a Google research team, in Google cafeterias, and made available in public data set called Nutrition5k.

We started with a subset of this data (3,260 images) that is publicly available here, that had been cleaned and filtered. We did a small amount of additional filtering of images where either:

One or more of the ingredients was listed as “deprecated”, or
There was a clear mislabelling error

This reduced the final set we used to 3,146 images.

How did we test ChatGPT?

We wanted to get a realistic result for ChatGPT that is a fair representation of the result a user would get if using the ChatGPT website or mobile app, so while we did use the OpenAI API for the test run, we got it to provide an open text response, and then recorded its carb estimation from its text response.

The user prompt was “How many carbs in this meal?”, and the specific OpenAI model used is ‘4o’ which is currently the default model in the ChatGPT app.

Wrapping up

We aim to provide carb estimates that are as accurate as possible, and this new release is a significant step. You can help improve SugarSnap further by using the ‘Report a problem’ button on an entry, if you see a carb estimate that doesn’t look right.

Feel free to get in touch with any feedback or questions!

SugarSnap aims to provide carbohydrate estimates that are as accurate as possible, however it can make mistakes or provide inaccurate estimates. SugarSnap is not designed or intended for managing medical conditions, and we make no representations or warranties about the accuracy of results. See our Terms of Service for more information.