Methods and systems for communicating sensor data on a mobile device

Patent metadata

Field	Value
Publication number	US-9313343-B2
Application number	US-201414185091-A
Country	US
Kind code	B2
Filing date	Feb 20, 2014
Priority date	Feb 20, 2014
Publication date	Apr 12, 2016
Grant date	Apr 12, 2016

Abstract

Methods and systems for communicating sensor data on a mobile device are described. An example method involves receiving, by a processor and from an inertial measurement unit (IMU), sensor data corresponding to a first timeframe, and storing the sensor data using a data buffer. The processor may also receive image data and sensor data corresponding to a second timeframe. The processor may then generate a digital image that includes at least the image data corresponding to the second timeframe and the sensor data corresponding to the first timeframe and the second timeframe. The processor may embed the stored sensor data corresponding to the first timeframe and the second timeframe in pixels of the digital image. And the processor may provide the digital image to an application processor of the mobile device.

First claim

What is claimed is:

1. A method comprising:
receiving, by a processor of a mobile device and from an inertial measurement unit (IMU) of the mobile device, sensor data corresponding to a first timeframe;
storing, by the processor, the sensor data corresponding to the first timeframe using a data buffer;
receiving, by the processor and from at least one camera of the mobile device, image data corresponding to a second timeframe, wherein the second timeframe is subsequent to the first timeframe;
receiving, by the processor and from the IMU, sensor data corresponding to the second timeframe;
generating, by the processor, a digital image that includes at least the image data corresponding to the second timeframe, the stored sensor data corresponding to the first timeframe, and the sensor data corresponding to the second timeframe, wherein the stored sensor data corresponding to the first timeframe and the sensor data corresponding to the second timeframe are embedded in pixels of the digital image; and
providing, by the processor, the digital image to an application processor of the mobile device.

2. The method of claim 1 , further comprising storing the sensor data corresponding to the second timeframe using the data buffer.

3. The method of claim 2 , further comprising:
receiving, by the processor and from the IMU, sensor data corresponding to a third timeframe, wherein the third timeframe is subsequent to the second timeframe;
receiving, by the processor and from at least one camera of the mobile device, image data corresponding to the third timeframe;
generating, by the processor, another digital image that includes at least the image data corresponding to the third timeframe, the stored sensor data corresponding to the first timeframe, the stored sensor data corresponding to the second timeframe, and the sensor data corresponding to the third timeframe, wherein the stored sensor data corresponding to the first timeframe, the stored sensor data corresponding to the second timeframe, and the sensor data corresponding to the third timeframe are embedded in pixels of the digital image; and
providing, by the processor, the other digital image to the application processor.

4. The method of claim 1 , further comprising determining time stamps associated with the first timeframe and the second timeframe respectively,
wherein the digital image further includes the determined time stamps, and
wherein the determined time stamps are embedded in additional pixels of the digital image.

5. The method of claim 4 :
wherein the sensor data corresponding to the first timeframe includes sensor data corresponding to two or more sub-timeframes within the first timeframe, and
wherein determining a time stamp associated with the first timeframe comprises determining time stamps associated with the sub-timeframes.

6. The method of claim 1 :
wherein the IMU outputs sensor data in intervals, and
wherein the sensor data corresponding to the second timeframe comprises sensor data from a present interval as well as sensor data form one or more previous intervals.

7. The method of claim 1 , wherein the image data from the at least one camera comprises first image data from a first camera and second image data from a second camera, the second camera having a field of view that is greater than 120 degrees.

8. The method of claim 1 , further comprising receiving, by the processor and from a depth processor of the mobile device, depth data corresponding to the second timeframe, wherein the digital image further comprises the depth data.

9. The method of claim 1 , further comprising:
determining computer-vision data using the image data corresponding to the first timeframe, wherein the computer-vision data comprises one or more image features of an image;
storing the computer-vision data corresponding to the first timeframe using the data buffer; and
determining computer-vision data using the image data corresponding to the second timeframe, wherein the digital image further comprises the stored computer-vision data corresponding to the first timeframe and the computer-vision data corresponding to the second timeframe.

10. The method of claim 1 , wherein the first timeframe and the second timeframe are each less than 50 milliseconds in duration.

11. The method of claim 1 , wherein the processor is configured to provide digital images to the application processor in intervals, and wherein the first timeframe corresponds to an interval immediately preceding the second timeframe.

12. The method of claim 1 , further comprising providing the digital image to the application processor using a camera bus interface.

13. A processing unit in a mobile device, the processing unit comprising:
data storage; and
instructions stored in the data storage and executable by the processing unit to perform functions comprising:
receiving from an inertial measurement unit (IMU) of the mobile device sensor data corresponding to a first timeframe,
receiving from at least one camera of the mobile device image data corresponding to the first timeframe,
accessing stored sensor data corresponding to a second timeframe, wherein the second timeframe precedes the first timeframe,
generating a digital image that includes at least the image data corresponding to the first timeframe, the stored sensor data corresponding to the second timeframe, and the sensor data corresponding to the first timeframe, wherein the stored sensor data corresponding to the second timeframe and the sensor data corresponding to the first timeframe are embedded in pixels of the digital image, and
providing the digital image to an application processor of the mobile device.

14. The processing unit of claim 13 , wherein the stored sensor data corresponding to the second timeframe is stored in a data buffer of the mobile device.

15. The processing unit of claim 14 :
wherein the functions further comprise accessing stored computer-vision data corresponding to the second timeframe, and
wherein the digital image further comprises the stored computer-vision data.

16. The processing unit of claim 13 :
wherein the functions further comprise determining time stamps associated with the first timeframe and the second timeframe respectively,
wherein the digital image further includes the determined time stamps, and
wherein the determined time stamps are embedded in additional pixels of the digital image.

17. A mobile device, comprising:
at least one camera;
an inertial measurement unit (IMU);
a first data buffer;
a second data buffer;
a processor, the processor configured to:
receive from the IMU sensor data corresponding to a first timeframe,
receive from the at least one camera image data corresponding to the first timeframe,
access, from the first data buffer, stored sensor data corresponding to a second timeframe, wherein the second timeframe precedes the first timeframe, and
generate a digital image that includes at least the image data corresponding to the first timeframe, the stored sensor data corresponding to the second timeframe, and the sensor data corresponding to the first timeframe, wherein the stored sensor data corresponding to the second timeframe and the sensor data corresponding to the first timeframe are embedded in pixels of the digital image; and
an application processor, the application processor configured to:
receive the digital image from the processor, and
store the image data corresponding to the first timeframe and the sensor data corresponding to the first timeframe in the second data buffer

Assignees

• Google Inc

Inventors

• Fung James
• Hesch Joel
• Lee Johnny

Classifications

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/57

  Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices · CPC title

• H04N1/2133 Primary

  Recording or reproducing at a specific moment, e.g. time interval or time-lapse · CPC title

• H04N2201/0084

  Digital still camera · CPC title