Augmented reality-based hand interaction apparatus and method using image information

Patent metadata

Field	Value
Publication number	US-9881423-B2
Application number	US-201615006322-A
Country	US
Kind code	B2
Filing date	Jan 26, 2016
Priority date	Jun 15, 2015
Publication date	Jan 30, 2018
Grant date	Jan 30, 2018

Abstract

An augmented reality-based hand interaction apparatus and method using image information. The augmented reality-based hand interaction apparatus using image information includes an object recognition module configured to extract feature points from an input color image, recognize an object which is a target of interest based on the extracted feature points, and estimate a three-dimensional (3D) posture of the recognized object, and an interaction module configured to extract 3D mesh information from an input depth image, match 3D coordinates of the extracted 3D mesh information with 3D coordinates of the object recognized by the object recognition module to extract 3D position information of the recognized object in 3D space, and detect whether or not an augmented 3D object and the recognized object collide with each other based on relative positional relationship information between the recognized object and the augmented 3D object.

First claim

What is claimed is:

1. An augmented reality-based hand interaction apparatus using image information, the apparatus comprising:
a processor;
an object recognition module comprising the processor, the processor being configured to implement instructions so that the object recognition module is configured to extract feature points from an input color image, recognize an object that is a target of interest based on the extracted feature points, and estimate a three-dimensional (3D) posture of the recognized object; and
an interaction module comprising the processor, the processor being configured to implement instructions so that the interaction module configured to extract 3D mesh information from an input depth image, match 3D coordinates of the extracted 3D mesh information with 3D coordinates of the object recognized by the object recognition module to extract 3D position information of the recognized object in 3D space, and detect whether or not an augmented 3D object and the recognized object collide with each other based on relative positional relationship information between the recognized object and the augmented 3D object.

2. The augmented reality-based hand interaction apparatus of claim 1 , wherein the object recognition module tracks a movement path of the recognized object in consecutive input color image frames, extracts new feature points from an input color image when tracking of the movement path of the recognized object fails, and recursively performs object recognition, 3D posture estimation, and object tracking based on the extracted new feature points.

3. The augmented reality-based hand interaction apparatus of claim 1 , wherein the interaction module calculates 3D position coordinates of a mesh corresponding to the recognized object, a movement direction vector of the mesh, and a speed of the mesh, and calculates a direction and a position of a collision between the recognized object and the augmented 3D object using the 3D position coordinates of the mesh, the movement direction vector of the mesh, and the speed of the mesh.

4. The augmented reality-based hand interaction apparatus of claim 3 , wherein the interaction module tracks and observes the collision direction and position over time, and determines a motion state of the augmented 3D object using 3D coordinates, a movement direction vector, and a speed of the recognized object tracked and observed in real time, and the collision direction and position.

5. The augmented reality-based hand interaction apparatus of claim 4 , wherein the motion state is any one motion of the augmented 3D object among a movement motion, a rotary motion, and a complex motion in which movement and rotation are made together.

6. The augmented reality-based hand interaction apparatus of claim 4 , wherein the interaction module renders a motion or a change in a shape of the augmented object corresponding to the determined motion state and displays the rendered motion or change in the shape to a user.

7. The augmented reality-based hand interaction apparatus of claim 1 , wherein the interaction module determines a back-and-forth positional relationship between the recognized object and the augmented 3D object with respect to a camera based on the relative positional relationship information between the recognized object and the augmented 3D object, renders an object positioned forward first, and then renders a region of an object positioned back excluding a region occluded by the object positioned forward.

8. The augmented reality-based hand interaction apparatus of claim 1 ,
wherein the object recognition module comprises: a color image input unit comprising the processor, the processor being configured to implement instructions so that the color image input unit is configured to receive color image information of surroundings taken by a camera;
a feature point extractor comprising the processor, the processor being configured to implement instructions so that the feature point extractor is configured to extract the feature points that may be features of the image using pixels corresponding to an entire region or a predetermined region in the received color image information;
an object recognizer comprising the processor, the processor being configured to implement instructions so that the object recognizer is configured to recognize a specific object in the color image by matching the extracted feature points with feature points extracted from a trained image;
a posture estimator comprising the processor, the processor being configured to implement instructions so that the posture estimator is configured to calculate a rotation matrix or a translation matrix representing a relative positional relationship between the recognized object and the camera; and
an object tracker comprising the processor, the processor being configured to implement instructions so that the object tracker is configured to track the recognized object in consecutive input image frames.

9. The augmented reality-based hand interaction apparatus of claim 1 ,
wherein the interaction module comprises:
a depth image input unit comprising the processor, the processor being configured to implement instructions so that the depth image input unit is configured to receive depth image information of surroundings taken by a depth camera;
a depth image mesh generator comprising the processor, the processor being configured to implement instructions so that the depth image mesh generator is configured to convert the input depth image into mesh information usable by a 3D rendering engine;
a calibrator comprising the processor, the processor being configured to implement instructions so that the calibrator is configured to calibrate a relative positional relationship between a red, green, and blue (RGB) camera and the depth camera using initial posture information of the recognized object and match the 3D coordinates of a mesh obtained by modeling the recognized object and the 3D coordinates of the recognized object;
a manipulation processor comprising the processor, the manipulator processor being configured to detect whether or not the augmented 3D object and the recognized object collide with each other based on the relative positional relationship information between the recognized object and the augmented 3D object in the 3D space, and determine a motion state of the augmented 3D object using the 3D coordinates, a movement direction vector, and a speed of the recognized object, and a direction and position of the collision; and
a manipulation state output unit comprising the processor, the processor being configured to implement instructions so that the manipulation state output unit is configured to render a motion or a change in a shape of the augmented object corresponding to the determined motion state and display the rendered motion or change in the shape to a user.

10. The augmented reality-based hand interaction apparatus of claim 9 , wherein the manipulation state output unit determines a back-and-forth positional relationship between the recognized object and the augmented 3D object with respect to a camera based on the relative positional relationship information between the recognized object and the augmented 3D object, renders an object positioned forward first, and then renders a region of an object positioned back excluding a region occluded by the object positioned forward.

11. An augmented reality-based hand interaction method using image information, the method comprising:
extracting feature points from an input color image, recognizing an object that is a target of interest based on the extracted feature points, and estimating a three-dimensional (3D) posture of the

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Cho Hyun Woo
• Lee Young Woon
• Jung Sung Uk

Classifications

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06K9/00382

  Physics · mapped topic

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title