Method and systems for multi-view high-speed motion capture

Patent metadata

Field	Value
Publication number	US-2016140733-A1
Application number	US-201414540394-A
Country	US
Kind code	A1
Filing date	Nov 13, 2014
Priority date	Nov 13, 2014
Publication date	May 19, 2016
Grant date	—

Abstract

Systems and methods that facilitate an efficient and effective multi-view hybrid system for high-speed motion analysis are presented. The multi-view hybrid system can include a mix of relatively few high-speed cameras (e.g., 1000 fps, etc.) and a greater number of regular-speed cameras (e.g., 30 fps, etc.) that are utilized to provide more high speed views than high speed cameras included in the system. The new multi-view hybrid approach exploits spatial-temporal coherence in sampled images from both the regular-speed cameras and high-speed cameras to increase the number of high speed motion viewpoints. In one embodiment, the high-speed motion captured from one or a few high-speed cameras is “transferred” to other viewpoints to create a set of virtual high-speed camera viewpoints that provide the ability to obtain high-speed motion information (e.g., synthesized image frame sequences) from multiple viewpoints in addition to viewpoints of the physical high speed cameras in the system.

First claim

What is claimed is:

1 . A device comprising:
a processing circuit that receives input associated with a three dimension (3D) scene, wherein the input includes information associated with a first frame sequence at a first frame rate corresponding to a first camera and information associated with a second frame sequence at a second frame rate corresponding to a second camera; the first frame rate is faster than the second frame rate, wherein temporally synchronized image frames recorded in the first frame sequence and the second frame sequence at given timestamps are aligned, a three dimensional (3D) model for a dynamic scene at the first frame rate is created, and synthesized image frame sequences associated with a plurality of additional viewpoints are generated at the first frame rate; and a memory that stores information for the processing circuit.

2 . The device of claim 1 , wherein the first frame sequence and the second frame sequence are calibrated.

3 . The device of claim 2 , wherein a calibration pre-process is conducted to determine intrinsic parameters and extrinsic parameters associated with the first camera and the second camera.

4 . The device of claim 1 , wherein the 3D model is based upon 3D key frame reconstruction, constrained 3D motion estimation, and 3D key frame interpolation.

5 . The device of claim 4 , wherein 3D key frame reconstruction includes a visual-hull based method to estimate a 3D mesh model of the target objects as the 3D model of a target scene.

6 . The device of claim 4 , wherein a kinematic-based 3D motion model is used to estimate a set of motion parameters from the 3D key frames.

7 . The device of claim 4 , wherein approximate optimal kinematic-based 3D motion parameters are used to establish an intermediate interpolated 3D frame sequence based on consecutive 3D key frame sequences and the interpolated 3D frame sequence gives a dynamic 3D model at the first rate for a dynamic scene from a virtual camera viewpoint at the first rate.

8 . The device of claim 4 , wherein the calibrated frame sequences captured from multiple viewpoints are used to enhance video quality, including denoising and correlation among multiple views, and spatial-temporal coherence within and among multiple video streams are used to increase signal-to-noise ratio.

9 . A method comprising:
calibrating a first video frame sequence at a first frame rate from a first camera and a second video frame sequence at a second frame rate from a second camera, wherein the first frame rate is faster than the second frame rate; creating a three dimension (3D) model based upon 3D key frame reconstruction, constrained 3D motion estimation, and 3D key frame interpolation; and generating a plurality of synthesized image frame sequences at the first frame rate, wherein the plurality of synthesized image frame sequences are associated with a plurality of viewpoints different from a viewpoint associated with the first camera.

10 . The method of claim 9 , enabling an alternative in which the calibrating uses a plurality of intrinsic parameters and a plurality of extrinsic parameters and produces a calibrated first video frame sequence and a calibrated second video frame sequence.

11 . The method of claim 10 , enabling an alternative in which the calibrating includes at least one of the following:
correcting lens distortion for the first video frame sequence and the second video frame sequence based upon the plurality of intrinsic parameters; and enabling geometric calibration by warping the first video frame sequence and the second video frame sequence to align with a reference sequence based upon the plurality of intrinsic parameters and the plurality of extrinsic parameters.

12 . The method of claim 10 , enabling an alternative in which the plurality of intrinsic parameters and the plurality of extrinsic parameters include at least one of the following:
an intrinsic camera parameter for the first camera and an intrinsic camera parameter for the second camera; lens distortion coefficients for the first camera and the second camera; and an extrinsic 6 degree-of-freedom position and pose for the first camera and the second camera.

13 . The method of claim 9 , enabling an alternative in which the 3D key frame reconstruction includes:
reconstructing a sequence of 3D key frames based upon a calibrated second video frame sequence; ascertaining a kinematic motion parameter based upon the sequence of 3D key frames, an intrinsic parameter, and an extrinsic parameter; and establishing a sequence of interpolated 3D frames for a virtual camera viewpoint at the first frame rate based upon the sequence of 3D key frames and the kinematic motion parameter.

14 . The method of claim 9 , enabling an alternative in which the generating a plurality of synthesized image frame sequences further includes:
creating a textured 3D model sequence by texture mapping a calibrated regular frame rate image sequence to a high frame rate dynamic 3D model; and constructing a synthesized image frame sequence for a virtual camera viewpoint at the first rate based upon the textured 3D model sequence.

15 . The method of claim 9 , enabling an alternative in which the generating a plurality of synthesized image frame sequences further includes utilizing spatial-temporal coherence in multi-view video denoising.

16 . A system comprising:
a first camera that captures images at a first speed; a second camera that captures images at a second speed, the second speed is slower than the first speed; a multi-view system for simulating a plurality of viewpoints at the first speed in addition to a viewpoint of the first camera, wherein image results of the first camera capture and the second camera capture are utilized to sample respective temporal appearance discrepancies, and spatial-temporal coherence in the sampled image results are utilized to synthesis video streams at the first speed from a viewpoint different than the viewpoint of the first camera.

17 . The system of claim 16 , wherein the first camera is one of a first plurality of cameras that capture images at the first rate and the second camera is one of a second plurality of cameras that capture images at the second rate, wherein the number of cameras included in the first plurality of cameras is less than the number of cameras included in the second plurality of cameras.

18 . The system of claim 16 , wherein the first frame rate is in or above the range of hundreds of frames per second the second rate is in the range of tens of frames per second.

19 . The system of claim 16 , wherein positions and poses of the first camera and second camera can change with respect to a scene but remain fixed relative to one another during a scene capture

20 . The system of claim 16 , wherein positions and poses of the first camera and second camera change with respect to one another after a first scene capture and before a second scene capture

Assignees

• Futurewei Technologies Inc

Inventors

• Gu Jinwei
• Jiang Wei

Classifications

• G06T7/55

  from multiple images · CPC title

• G06T7/85

  Stereo camera calibration · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06T7/285 Primary

  using a sequence of stereo image pairs · CPC title

• H04N5/2627

  for providing spin image effect, 3D stop motion effect or temporal freeze effect (2D image animation in general G06T13/80) · CPC title