Adaptive sampling of training views

Patent metadata

Field	Value
Publication number	US-10769437-B2
Application number	US-201815949728-A
Country	US
Kind code	B2
Filing date	Apr 10, 2018
Priority date	Apr 10, 2018
Publication date	Sep 8, 2020
Grant date	Sep 8, 2020

Abstract

A head-mounted display, a method, and a non-transitory computer readable medium are provided. An embodiment of a method for obtaining training sample views of an object includes the step of storing, in a memory, multiple views of an object. The method also includes the step of deriving similarity scores between adjacent views and then a sampling density is varied based on the similarity scores.

First claim

What is claimed is:

1. A non-transitory computer-readable medium storing thereon a computer program that, when executed by one or more processors, performs a method comprising the steps of:
(a) receiving, in a memory, feature data sets of (i) a reference object or (ii) a 3D model, each feature data set being obtained from an image captured or generated from a primary set of views within a view range around the reference object or the 3D model,
(b) deriving similarity scores between each pair of adjacent views among the primary set of views using the feature data sets,
(c) sampling a secondary set of views disposed between each pair of the adjacent primary views that have the similarity score being equal to or less than a threshold so that sampling density varies across the view range, and
(d) generating data corresponding to (i) the primary set of views including each pair of the adjacent primary views that have the similarity score being equal to or less than the threshold and (ii) the secondary set of views.

2. The non-transitory computer-readable medium of claim 1 , further performing the steps of:
sampling at least one view from a group of the primary set of views and the secondary set of views at a coarse sampling resolution at equal angles from each other with respect to azimuth and/or elevation; and
storing the sampled views in the memory.

3. The non-transitory computer-readable medium of claim 1 , the method further comprising performing sampling at a finer sampling resolution, relative to a sampling resolution of the sampling in step (c), near adjacent views having similarity scores lower than a threshold to obtain further sampling views.

4. The non-transitory computer-readable medium of claim 3 , the method further performing the steps of:
(e) deriving similarity scores between the additional views sampled at the finer sampling resolution;
(f) performing an even finer sampling resolution near the additional views having low similarity scores; and
(g) repeating steps (e) and (f) until the similarity scores exceed a minimum threshold.

5. The non-transitory computer-readable medium of claim 1 , wherein similarity scores are based on appearance and/or shape of adjacent views of the reference object or the 3D model.

6. The non-transitory computer-readable medium of claim 1 , the method further performing the step of:
clustering images generated from adjacent views having high similarity scores to reduce a total number of images.

7. A method comprising the steps of:
(a) receiving, in a memory, feature data sets of (i) a reference object or (ii) a 3D model, each feature data set being obtained from an image captured or generated from a primary set of views within a view range around the reference object or the 3D model,
(b) deriving similarity scores between each pair of adjacent views among the primary set of views using the feature data sets,
(c) sampling a secondary set of views disposed between each pair of the adjacent primary views that have the similarity score being equal to or less than a threshold so that sampling density varies across the view range, and
(d) generating data corresponding to (i) the primary set of views including each pair of the adjacent primary views that have the similarity score being equal to or less than the threshold and (ii) the secondary set of views.

8. The method of claim 7 , wherein the step of sampling the views includes sampling views denser between a pair of adjacent views than between another pair of adjacent views, depending on the similarity scores.

9. The method of claim 7 , further comprising the steps of:
storing the sampled views in the memory.

10. The method of claim 7 , wherein the adjacent views are sampled at equal distances from a primary view.

11. The method of claim 7 , further comprising performing additional sampling at a finer sampling resolution, relative to a sampling resolution of the sampling in step (c), near adjacent views having similarity scores lower than a threshold to obtain further sampling views.

12. The method of claim 11 , further comprising the steps of:
a) deriving similarity scores between the additional views sampled at the finer sampling resolution;
b) performing an even finer sampling resolution near the additional views having low similarity scores; and
c) repeating steps (a) and (b) until the similarity scores exceed a minimum threshold.

13. The method of claim 7 , wherein the data as training data is used for detecting the object.

14. The method of claim 13 , wherein, after the views are sampled and the training data is accumulated, a head-mounted display device is configured to detect the object using the training data.

15. The method of claim 7 , wherein similarity scores are based on appearance and/or shape of the object.

16. A head-mounted display device comprising:
memory configured to store data used as training data for detecting an object, the data derived from multiple views of the object;
a processing device configured to utilize the training data to detect the object; and
a display component configured to display images of the detected object;
wherein, before object detection, the multiple views are obtained using a varying sampling density based on similarity scores of adjacent views by the steps of:
(a) receiving feature data sets of (i) a reference object or (ii) a 3D model, each feature data set being obtained from an image captured or generated from a primary set of views within a view range around the reference object or the 3D model,
(b) deriving similarity scores between each pair of adjacent views among the primary set of views using the feature data sets,
(c) sampling a secondary set of views disposed between each pair of the adjacent primary views that have the similarity score being equal to or less than a threshold so that sampling density varies across the view range, and
(d) generating the data corresponding to (i) the primary set of views including each pair of the adjacent primary views that have the similarity score being equal to or less than the threshold and (ii) the secondary set of views.

17. The head-mounted display device of claim 16 , wherein, before object detection, the multiple views are originally sampled at a coarse sampling resolution at locations surrounding the object.

18. The head-mounted display device of claim 17 , wherein, before object detection, the multiple views are sampled by:
performing a finer sampling resolution near adjacent views having low similarity scores to obtain additional views;
deriving similarity scores between the additional views sampled at the finer sampling resolution;
performing an even finer sampling resolution near the additional views having low similarity scores; and
repeating the deriving and performing steps until the similarity scores exceed a minimum threshold.

19. The head-mounted display device of claim 18 , wherein the step of sampling the views includes sampling views denser between a pair of adjacent views than between another pair of adjacent views, depending on the similarity scores.

20. The non-transitory computer-readable medium according to claim 1 , wherein the method further comprises:
receiving information identifying an object detection device including a camera for use in detecting a pose of an object in a real scene, the object corresponding to the reference object or the 3D model;
acquiring, based at least in part on the information identifying the object detection device, a camera parameter se

Assignees

• Seiko Epson Corp

Inventors

• Mukherjee Dibyendu
• Li Jia
• Brusnitsyn Mikhail
• Kezele Irina

Classifications

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title