Method, computer program product, and system for producing combined image information to provide extended vision

Patent metadata

Field	Value
Publication number	US-10593163-B2
Application number	US-201715837507-A
Country	US
Kind code	B2
Filing date	Dec 11, 2017
Priority date	Oct 15, 2014
Publication date	Mar 17, 2020
Grant date	Mar 17, 2020

Abstract

Method, computer program product, and system to provide an extended vision within an environment having a plurality of items, where the extended vision is based on a field of view of a person determined using a first visual sensor, and is further based on at least a second visual sensor disposed within the environment. Image information from the first and second visual sensors is associated to produce combined image information. Selected portions of the combined image information are displayed based on input provided through a user interface.

First claim

What is claimed is:

1. A computer-implemented method to provide an extended vision within an environment having a plurality of items available for selection by a person, the method comprising:
analyzing first image information acquired using a first visual sensor within the environment to determine a field of view of the person, wherein the first visual sensor is coupled with a first computing device that is body-worn or carried by the person;
analyzing second image information acquired using a plurality of second visual sensors disposed within the environment, wherein the plurality of second visual sensors are coupled with a second computing device;
determining that an arrangement of the plurality of second visual sensors results in at least a first deadspace area within the environment, the first deadspace area representing a first portion of the environment that is not represented in the second image information acquired using any of the plurality of second visual sensors;
determining, using stored location data describing a planned arrangement for placement of the plurality of items within the environment, one or more items of the plurality of items that are within or adjacent to the first deadspace area;
transmitting, to the first computing device, a prompt suggesting that the person should acquire image information corresponding to the first deadspace area, wherein the prompt comprises a suggestion for the one or more items;
upon determining that the field of view of the person includes the one or more items, storing at least a portion of the first image information in a memory of the second computing device; and
generating a combined image information using at least (i) the stored portion of the first image information, and (ii) a portion of the second image information.

2. The method of claim 1 , wherein generating the combined image information comprises stitching the first image information and the second image information.

3. The method of claim 1 , further comprising:
displaying, via a display device, at least a portion of the combined image information; and
displaying, via the display device, a user interface (UI) for traversing the portion of the combined image information.

4. The method of claim 3 , further comprising:
performing, in response to the input provided through the UI, at least one of reorienting and displacing one or more visual sensors of the second visual sensors.

5. The method of claim 1 , wherein the first computing device is a body-worn computing device and the first visual sensor is a forward-looking camera integrated with the first computing device.

6. The method of claim 1 , wherein the first computing device is a handheld computing device, and
wherein the person manipulates an orientation of the handheld computing device to acquire the first image information.

7. The method of claim 1 , further comprising:
determining that the arrangement of the plurality of second visual sensors results in at least a first overlap area representing a second portion of the environment acquired in the image information of two or more second visual sensors of the plurality of second visual sensors; and
analyzing, for each of the two or more second visual sensors, corresponding image information to identify a respective number of the plurality of items included therein;
assigning, based on a largest number of the identified numbers of the plurality of items, the corresponding visual sensor of the two or more second visual sensors as a primary view for the first overlap area; and
assigning one other visual sensor of the two or more second visual sensors as an auxiliary view for the first overlap area.

8. The method of claim 7 , wherein none of the identified numbers of the plurality of items meet a predetermined threshold value, the method further comprising:
upon determining that the field of view of the person overlaps at least part of the first overlap area, storing a portion of the first image information corresponding to the overlapping part in the memory of the second computing device.

9. A computer program product to provide an extended vision within an environment having a plurality of items available for selection by a person, the computer program product comprising:
a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation that includes:
analyzing first image information acquired using a first visual sensor within the environment to determine a field of view of the person, wherein the first visual sensor is coupled with a first computing device that is body-worn or carried by the person;
analyzing second image information acquired using a plurality of second visual sensors disposed within the environment, wherein the plurality of second visual sensors are coupled with a second computing device;
determining that an arrangement of the plurality of second visual sensors results in at least a first deadspace area within the environment, the first deadspace area representing a first portion of the environment that is not represented in the second image information acquired using any of the plurality of second visual sensors;
determining, using stored location data describing a planned arrangement for placement of the plurality of items within the environment, one or more items of the plurality of items that are within or adjacent to the first deadspace area;
transmitting, to the first computing device, a prompt suggesting that the person should acquire image information corresponding to the first deadspace area, wherein the prompt comprises a suggestion for the one or more items;
upon determining that the field of view of the person includes the one or more items, storing at least a portion of the first image information in a memory of the second computing device; and
generating a combined image information using at least (i) the stored portion of the first image information, and (ii) a portion of the second image information.

10. The computer program product of claim 9 , wherein generating the combined image information comprises stitching the first image information and the second image information.

11. The computer program product of claim 9 , wherein the operation further comprises:
displaying, via a display device, at least a portion of the combined image information; and
displaying, via the display device, a user interface (UI) for traversing the portion of the combined image information.

12. The computer program product of claim 9 , the operation further comprising:
determining that the arrangement of the plurality of second visual sensors results in at least a first overlap area representing a second portion of the environment acquired in the image information of two or more second visual sensors of the plurality of second visual sensors; and
analyzing, for each of the two or more second visual sensors, corresponding image information to identify a respective number of the plurality of items included therein;
assigning, based on a largest number of the identified numbers of the plurality of items, the corresponding visual sensor of the two or more second visual sensors as a primary view for the first overlap area; and
assigning one other visual sensor of the two or more second visual sensors as an auxiliary view for the first overlap area.

13. The computer program product of claim 12 , wherein none of the identified numbers of the plurality of items meet a predetermined threshold value, the operation further comprising:
upon determining that the field of

Assignees

• Toshiba Global Commerce Solutions Holdings Corp

Inventors

• Chirakansakcharoen Monsak Jason
• Herring Dean Frederick
• Singh Ankit
• Steiner David John

Classifications

• G06Q30/0251

  Targeted advertisements · CPC title

• G06Q30/0639

  Locating goods or services, e.g. based on physical position of the goods or services within a shopping facility · CPC title

• H04N7/181

  for receiving images from a plurality of remote sources · CPC title

• G06Q10/087

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G01B11/02

  for measuring length, width or thickness (G01B11/08 takes precedence) · CPC title