System and method for selective image capture on sensor floating on the open sea

Patent metadata

Field	Value
Publication number	US-11917337-B2
Application number	US-202117463433-A
Country	US
Kind code	B2
Filing date	Aug 31, 2021
Priority date	Aug 31, 2021
Publication date	Feb 27, 2024
Grant date	Feb 27, 2024

Abstract

The present specification relates to image capture. More specifically, it relates to selective image capture for sensor carrying devices or floats deployed, for example, on the open sea. In one form, data is generated on the sensor carrying devices or floats by an on-board Inertial Measurement Unit (IMU) and is used to automatically predict the wave motion of the sea. These predictions are then used to determine an acceptable set of motion parameters that are used to trigger the on-board camera(s). The camera(s) then capture images. One consideration is that images captured at or near the peak of a wave crest with minimal pitch and roll will contain fewer obstructions (such as other waves). Such images provide a view further into the horizon to, for example, monitor maritime sea traffic and other phenomenon. Therefore, the likelihood of capturing interesting objects such as ships, boats, garbage, birds, . . . etc. is increased. These images may then be further processed and/or transmitted in a variety of manners.

First claim

What is claimed is:

1. A system on a floating device deployed in open sea, the system comprising:
a camera configured to capture images;
an inertial measurement unit (IMU) configured to capture IMU data; and,
at least one processor and at least one memory having stored thereon code that when executed by the at least one processor triggers the camera to capture an image based on the IMU data and predetermined criteria, wherein the predetermined criteria comprises:
z>z _thresh× z _peak

− p _radius< p<p _radius

− r _radius< r<r _radius

where z is a current estimate of the height of the sensor carrying device or float (at, above, or below sea level), z thresh is a percentage of average predicted wave height the device or float needs to be above to allow capture, z peak is a maximum height of the device or float over c seconds (used to approximate height of wave crests), p is pitch, p radius is a pitch radius the device or float needs to be within to allow capture, r is roll, and r radius is a roll radius the device or float needs to be within to allow capture.

2. The system as set forth in claim 1 wherein the code when executed by the at least one processor executes a calibration sequence.

3. The system as set forth in claim 1 wherein the code when executed by the at least one processor triggers the camera to capture an image based on an estimate of when the sensor carrying device or float will be at a top of a wave.

4. The system as set forth in claim 3 wherein the estimate is based on the IMU data and a predictive routine.

5. The system as set forth in claim 4 wherein the predictive routine comprises implementation of a Kalman filter to predict wave behavior.

6. A system on a floating device deployed in open sea, the system comprising:
a camera configured to capture images;
an inertial measurement unit (IMU) configured to capture IMU data; and,
at least one processor and at least one memory having stored thereon code that when executed by the at least one processor triggers the camera to capture multiple images and the at least one processor to select captured images based on the IMU data, wherein the code when executed by the at least one processor synchronizes data from the inertial measurement unit with the multiple images, filters the data from the inertial measurement unit for each captured image using predetermined criteria and selects the captured images that satisfy the predetermined criteria.

7. The system as set forth in claim 6 wherein the code when executed by the at least one processor executes a calibration sequence.

8. The system as set forth in claim 6 wherein the camera randomly captures the multiple images.

9. The system as set forth in claim 6 wherein the camera captures the multiple images based on a predicted crest of a wave.

10. The system as set forth in claim 6 wherein the predetermined criteria comprises:
z>z _thresh× z _peak

− p _radius< p<p _radius

− r _radius< r<r _radius

where z is a current estimate of the height of the sensor carrying deice or float (at, above, or below sea level), z_thresh is a percentage of average predicted wave height the device or float needs to be above to allow capture, z_peak is a maximum height of the device or float over c seconds (used to approximate height of wave crests), p is pitch, p_radius is a pitch radius the device or float needs to be within to allow capture, r is roll, and r_radius is a roll radius the device or float needs to be within to allow capture.

11. A method for image capture on sensor carrying devices or floats having at least one camera and an inertial measurement unit, the method comprising:
selectively initiating an image capture routine;
executing a calibration sequence to predict a current height of the sensor carrying device or float, estimate sea level and predict a wave crest height;
determining the current height of the sensor carrying device or float using data from the inertial measurement unit;
measuring pitch and roll of the sensor carrying device or float by the inertial measurement unit; and,
capturing an image by the camera if predetermined criteria are satisfied based on the determined height, pitch and roll of the sensor carrying device or float.

12. The method as set forth in claim 11 wherein the calibration sequence comprises integrating accelerometer data of the inertial measurement unit corresponding to measured acceleration in a direction perpendicular to an ocean surface (Z) over the last w measurements to predict the current height (z) of the float or device, where w is a number of points integrated to predict current height (z), then averaging all z values to estimate the sea level, and using a maximum measured value of z over time to predict the wave crest height.

13. The method as set forth in claim 11 wherein the predetermined criteria comprises:
z>z _thresh× z _peak

− p _radius< p<p _radius

− r _radius< r<r _radius

where z is a current estimate of the height of the sensor carrying deice or float (at, above, or below sea level), z_thresh is a percentage of average predicted wave height the device or float needs to be above to allow capture, z_peak is a maximum height of the device or float over c seconds (used to approximate height of wave crests), p is pitch, p_radius is a pitch radius the device or float needs to be within to allow capture, r is roll, and r_radius is a roll radius the device or float needs to be within to allow capture.

14. A method for image capture on sensor carrying devices or floats having at least one camera and an inertial measurement unit, the method comprising:
selectively initiating an image capture routine;
executing a calibration sequence to predict a current height of the sensor carrying device or float, estimate sea level and predict a wave crest height;
estimating when the sensor carrying device or float will be at a top of a wave based on the calibration sequence and output of a predictive routine; and,
capturing an image by the camera based on the estimating.

15. The method as set forth in claim 14 wherein the calibration sequence comprises integrating accelerometer data of the inertial measurement unit corresponding to measured acceleration in a direction perpendicular to an ocean surface (Z) over the last w measurements to predict the current height (z) of the float or device, where w is a number of points integrated to predict current height (z), then averaging all z values to estimate the sea level, and using a maximum measured value of z over time to predict the wave crest height.

16. The method as set forth in claim 14 wherein the predictive routine comprises implementation of a Kalman filter to predict wave behavior.

17. A method for image capture on sensor carrying devices or floats having at least one camera and an inertial measurement unit, the method comprising:
selectively initiating an image capture routine;
executing a calibration sequence to predict a current height of the sensor carrying device or float, estimate sea level and predict a wave crest height;
capturing multiple images;
synchronizing data from the inertial measurement unit with the multiple images;
filtering the data from the inertial measurement unit for each captured image using predetermined criteria; and,
selecting the captured images that satisfy the predetermined criteria.

18. The method as set forth in claim 17 wherein the capturing comprises randomly capturing images.

19. The method as set forth in claim 17 wherein the captur

Assignees

• Palo Alto Res Ct Inc
• Xerox Corp

Inventors

• Shreve Matthew A
• Cocker Eric

Classifications

• H04N7/188 Primary

  Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253) · CPC title

• B63B79/15

  for monitoring environmental variables, e.g. wave height or weather data · CPC title

• G01C13/004

  vertical movement · CPC title

• G01C13/00 Primary

  Surveying specially adapted to open water, e.g. sea, lake, river or canal (liquid level metering G01F) · CPC title

• G08G3/00

  Traffic control systems for marine craft (marking of navigational route B63B22/16, B63B51/00) · CPC title