Estimation of food volume and carbs

Patent metadata

Field	Value
Publication number	US-2018122063-A1
Application number	US-201715856631-A
Country	US
Kind code	A1
Filing date	Dec 28, 2017
Priority date	Jul 2, 2013
Publication date	May 3, 2018
Grant date	—

Abstract

There is disclosed a system for estimating the volume of food on a plate, for example meal, with a mobile device. The system uses a camera and a light pattern projector. Images of the food with and without a projected light pattern on it enable to compute the tridimensional shape and volume, while image segmentation and recognition steps estimate one or more food types in said images. By applying accessible knowledge databases, the carbs content is estimated and the associated insulin bolus doses are provided. Developments comprise coding of the light pattern, different light sources and associated wavelengths, motion compensations, additional optics, estimation of fat content and associated multi-wave boluses. The invention can be implemented in a glucometer or in an insulin pump controller provided with a test strip port or with a mobile phone.

First claim

What is claimed is:

1 . A system for estimating carbs content of food, said system comprising instructions which when performed by a processor causes the system to operate to:
identify a movement of an image acquisition component between a first image of the food and a second image of the food, the second image having been acquired with the food highlighted with a projected light pattern; correct the first image and/or the second image by an operation which compensates movements of the image acquisition component throughout an image acquisition operation based on artifacts to get first and second images of a corrected pair of images; subtract the first and second images of the corrected pair of images; identify the projected light pattern based upon the subtraction of the first and second images of the corrected pair of images; compute the tridimensional shape and the volume of the food given the deformations of the projected light pattern; make segmentation and recognition of the food from the first image; and compute the carbs content of the food based upon the volume, segmentation and recognition of food via use of nutritional databases.

2 . The system if claim 1 , further comprising a light pattern projection component to provide the projected light pattern on a surface of the food.

3 . The system of claim 2 , wherein the irradiation power of the light pattern projection component is inferior to 5 mW and operates continuously, or in impulse mode with pulse frequencies between around 0.5 Hz up to 10 Hz.

4 . The system of claim 2 , wherein the light pattern projection component comprises one or more light sources chosen from the list comprising a low intensity semiconductor LASER diode, a LED, an organic LED (OLED), a pre-existing mobile phone flash such as a white LED or a miniaturized halogen lamp or a combination thereof with an electrical power consumption less than 0.7 W.

5 . The system of claim 2 , wherein the light pattern projection component comprises a light source and an optical objective adapted to form and project and/or to focus the projected light pattern onto the food, wherein the power density is inferior to the predefined value of 55 mWt/cm 2 .

6 . The system of claim 2 , wherein the relative pose and orientation of the light pattern projection component and the image acquisition component is static during image acquisition or light pattern projection.

7 . The system of claim 1 , wherein power density for the projected light pattern is inferior to 55 mWt/cm 2 .

8 . The system of claim 1 , wherein the projected light pattern is composed of geometrical motifs such as sequences of stripes, or dots, or repetitive graphical elements or of a combination thereof.

9 . The system of claim 1 , wherein the projected light pattern is coded by color and/or phase and/or amplitude modulation.

10 . The system of claim 2 , wherein the projected light pattern is coded by color and/or phase and/or amplitude modulation, and wherein the coding of the projected light pattern is predefined and is synchronized with the image acquisition component and data processing.

11 . The system of claim 1 , wherein the compensation to the movements of the image acquisition component throughout the image acquisition operation is performed by processing data received from a motion sensor.

12 . The system of claim 1 , wherein the compensation of the movements of the image acquisition component throughout the image acquisition operation is performed by multi-view geometry methods, by projective warping, by piecewise linear, projective, or higher order warping, by deconvolution, by oriented image sharpening, or by optical flow detection before or after or in an iterative refinement process with the subtraction of the images.

13 . The system of claim 1 , wherein the food segmentation and recognition comprise one or more of the operations of segmenting the image, identifying color and/or texture features of segmented parts of the image and performing machine learning based classification for one or more segmented parts of the image or a combination thereof.

14 . The system of claim 13 , further comprising instructions which when performed by a processor causes the system to operate to estimate one or more meal characteristics of the meal captured in the first or second image by multiplying the estimated volumes of the determined food types by unitary volumetric values retrieved from a database, said database being accessed from the Internet, and/or stored locally on the device, and/or determined from food labels by using OCR and/or associated with geolocation data and/or provided by the user.

15 . The system of claim 13 , wherein the one or more characteristics, of the meal or of parts thereof, are one or more of carbs content, fat content, protein content, Glycemic Index (GI), Glycemic Load (GL) and/or Insulin Index (II) or a combination thereof.

16 . The system of claim 14 , further comprising instructions which when performed by a processor causes the system to operate to provide an insulin dose recommendation and/or a bolus profile advice based on said one or more meal characteristics.

17 . The system of claim 1 , wherein the system is configured to run automatically and project the light pattern via a voice command or a gesture command and/or by a touchscreen command and/or by a geo position and/or by following a predefined time schedule.

18 . The system of claim 2 , wherein the image acquisition component and the light pattern projection component are embedded in a mobile device, a glucometer, an insulin pump, a mobile phone or a smartphone.

19 . The system of claim 2 , wherein the light pattern projection component is attachable to a mobile device which embeds the image acquisition component via a clip or via insertion into an electrical contact slot, a power charge slot or a USB slot.

20 . The system of claim 1 , wherein the first or the second image are based upon a video frame.

21 . The system of claim 1 , wherein one or more operations of the system are continuously automatically repeated until the acquisition of the first and/or the second image is considered as sufficient based on predefined thresholds associated with criteria that comprises one or more of image quality, associated measurements of handshakes, time delays between still images or video frames, resulting light pattern or a combination thereof.

22 . The system of claim 1 , wherein the image acquisition component is embedded in a handheld device and has a sensitivity between around 0.3 up to around 3 lux.

23 . The system of claim 2 , wherein the light pattern projection component has an electrical power consumption between 0.05 W and 0.7 W.

24 . A non-transitory computer readable medium encoded with an information processing program for use in an information processing device, said information processing program comprising instructions which when executed by a processor in the information processing device causes the information processing device to perform the operations of claim 1 .

25 . A non-transitory computer product comprising the non-transitory computer readable medium according to claim 24 .

26 . A method of utilizing a system for estimating carbs content of food, said system comprising a mobile device with a processor and instructions which when executing cause the processor to operate accordi

Assignees

• Roche Diabetes Care Inc
• Univ Of Bern

Inventors

• Dehais Joachim
• Greenburg Alan
• Shevchick Sergey
• Soni Abhishek
• Anthimpoulos Marios
• Mougiakakou Stavroula

Classifications

• G06T7/521 Primary

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G06T7/0004 Primary

  Industrial image inspection · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06T2207/30128

  Food products · CPC title