Signal detection, recognition and tracking with feature vector transforms

Patent metadata

Field	Value
Publication number	US-2016189381-A1
Application number	US-201514924664-A
Country	US
Kind code	A1
Filing date	Oct 27, 2015
Priority date	Oct 27, 2014
Publication date	Jun 30, 2016
Grant date	—

Abstract

A method for obtaining object surface topology in which image frames of a scene (e.g., video frames from a user passing a smartphone camera over an object) are transformed into dense feature vectors, and feature vectors are correlated to obtain high precision depth maps. Six dimensional pose is determined from the video sequence, and then used to register patches of pixels from the frames. Registered patches are aligned and then correlated to local shifts. These local shifts are converted to precision depth maps, which are used to characterize surface detail of an object. Feature vector transforms are leveraged in a signal processing method comprising several levels of interacting loops. At a first loop level, a structure from motion loop process extracts anchor features from image frames. At another level, an interacting loop process extracts surface texture, as noted. At additional levels, object forms are segmented from the images, and objects are counted and/or measured. At still a higher level, the lower level data structures providing feature extraction, 3D structure and pose estimation, and object surface registration are exploited by higher level loop processes for object identification (e.g., using machine learning classification), digital watermark or bar code reading and image recognition from the registered surfaces stored in lower level data structures.

First claim

I claim:

1 . A method of obtaining surface detail of an object from a video sequence captured by a moving camera over the object, the method comprising:
providing a camera model and the video sequence; using a processor, determining pose estimation from the video sequence using the camera model; using a processor, registering images from different frames using the pose estimation; using a processor, performing a feature vector transform on the images to produce N-dimensional feature vector per pixel of the images, the feature vector transform producing for each pixel in an array of pixels, a first vector component corresponding to plural comparisons between a center pixel and pixels at plural directions around the center pixel for a first scale, and second vector component corresponding to plural comparisons between the center pixel and pixels at plural directions around the center pixel for a second scale; using a processor, correlating the feature vector transforms of the images to obtain shift measurements between the images; and using a processor, obtaining surface height detail of the object from the shift measurements.

2 . The method of claim 1 wherein the determining of pose estimation comprises:
performing a feature vector transform on frames of the video sequence, the feature vector transform producing for each pixel in an array of pixels, a vector component corresponding to plural comparisons between a center pixel and pixels at plural directions around the center pixel;
using a processor, finding shifts between a first feature vector transformed frame and at least a second feature vector transformed frame; and
using a processor, determining the pose estimation from the shifts.

3 . The method of claim 1 wherein the plural comparisons at the first and second scales comprise quantized differences.

4 . The method of claim 3 wherein the quantized differences are encoded in arcs of a ring at the first and second scales.

5 . The method of claim 1 wherein the plural comparisons at each of the first and second scales are converted to a gradient.

6 . The method of claim 5 wherein the gradient comprises a magnitude and direction to produce at least two vector components per scale.

7 . The method of claim 1 wherein providing the video sequence comprises obtaining the video sequence from a mobile device camera, which captures the video sequence as the mobile device camera is moved over the object.

8 . The method of claim 1 wherein the processor comprises a processor in a mobile device comprising the mobile device camera.

9 . A non-transitory computer readable medium on which is stored instructions, which when executed by one or more processors, perform a method of obtaining surface detail of an object from a video sequence captured by a moving camera over the object, the method comprising:
determining pose estimation from the video sequence using a camera model; registering images from different frames using the pose estimation; performing a feature vector transform on the images to produce N-dimensional feature vector per pixel of the images, the feature vector transform producing for each pixel in an array of pixels, a first vector component corresponding to plural comparisons between a center pixel and pixels at plural directions around the center pixel for a first scale, and second vector component corresponding to plural comparisons between the center pixel and pixels at plural directions around the center pixel for a second scale; correlating the feature vector transforms of the images to obtain shift measurements between the images; and obtaining surface height detail of the object from the shift measurements.

10 . The computer readable medium of claim 9 wherein the determining of pose estimation comprises:
performing a feature vector transform on frames of the video sequence, the feature vector transform producing for each pixel in an array of pixels, a vector component corresponding to plural comparisons between a center pixel and pixels at plural directions around the center pixel;
finding shifts between a first feature vector transformed frame and at least a second feature vector transformed frame; and
determining the pose estimation from the shifts.

11 . The computer readable medium of claim 9 wherein the plural comparisons at the first and second scales comprise quantized differences.

12 . The computer readable medium of claim 11 wherein the quantized differences are encoded in arcs of a ring at the first and second scales.

13 . The computer readable medium of claim 9 wherein the plural comparisons at each of the first and second scales are converted to a gradient.

14 . The computer readable medium of claim 13 wherein the gradient comprises a magnitude and direction to produce at least two vector components per scale.

15 . A mobile device comprising:
a camera for capturing a video sequence of an object; a processor programmed with instructions the configure the processor to: determine pose estimation from the video sequence using the camera model; align images from different frames using the pose estimation; perform a feature vector transform on the images to produce N-dimensional feature vectors per pixel of the images, the feature vector transform producing for each pixel in an array of pixels, a first vector component corresponding to plural comparisons between a center pixel and pixels at plural directions around the center pixel for a first scale, and second vector component corresponding to plural comparisons between the center pixel and pixels at plural directions around the center pixel for a second scale; correlate the feature vector transforms of the images to obtain shift measurements between the images; and obtain surface height detail of the object from the shift measurements.

16 . A system for obtaining surface detail of an object from a video sequence captured by a moving camera over the object, the system comprising:
means for estimating pose of the object relative to the camera from the video sequence; means for transforming the images into dense feature vector arrays, the feature vector arrays comprising a feature vector per pixel, the feature vector having a first vector component corresponding to plural comparisons between a center pixel and pixels at plural directions around the center pixel for a first scale, and second vector component corresponding to plural comparisons between the center pixel and pixels at plural directions around the center pixel for a second scale; and means for obtaining surface height detail of the object from the dense feature vector arrays.

17 . The system of claim 16 wherein the means for estimating pose comprises a processor programmed with instructions to:
determine a coarse 6D pose from the video sequence based on a camera model;
obtain dense feature vector transforms of images in the video sequence;
aligning the feature vector transforms with the coarse 6D pose; and
determining a refined 6D pose from the aligned feature vector transforms.

18 . The system of claim 16 wherein the means for obtaining surface height detail comprises a processor programmed with instructions to:
obtain shift measurements between the images from the dense vector arrays; and
obtain surface height detail of the object from the shift measurements.

Assignees

• Digimarc Corp

Inventors

• Rhoads Geoffrey B

Classifications

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T7/579 Primary

  from motion · CPC title

• G06T7/33

  using feature-based methods · CPC title

• G06T7/0026

  Physics · mapped topic

• G06T2207/30244

  Camera pose · CPC title