Graphic interface for real-time vision enhancement

Patent metadata

Field	Value
Publication number	US-11320655-B2
Application number	US-201715799404-A
Country	US
Kind code	B2
Filing date	Oct 31, 2017
Priority date	May 15, 2014
Publication date	May 3, 2022
Grant date	May 3, 2022

Abstract

Imaging systems can often gather higher quality information about a field of view than the unaided human eye. For example, telescopes may magnify very distant objects, microscopes may magnify very small objects, and high frame-rate cameras may capture fast motion. The present disclosure includes devices and methods that provide real-time vision enhancement without the delay of replaying from storage media. The disclosed devices and methods may include a live view user interface with two or more interactive features or effects that may be controllable in real-time. Specifically, the disclosed devices and methods may include a live view display and image and other information enhancements, which utilize in-line computation and constant control.

First claim

We claim:

1. A method comprising:
based on image data from a camera, displaying, on a display of a computing device, a live view representation that shows (i) a first feature at a first portion of the live view representation and (ii) a second feature at a second portion of the live view representation;
receiving, via an interface of the computing device, control input indicative of a swap effect, wherein the swap effect comprises causing (i) the first feature to be shown at the second portion of the live view representation rather than at the first portion of the live view representation and (ii) the second feature to be shown at the first portion of the live view representation rather than at the second portion of the live view representation; and
in response to receiving control input indicative of the swap effect, producing the swap effect in the live view representation in real-time.

2. The method of claim 1 ,
wherein the first feature comprises a first body part shown at the first portion of the live view representation,
wherein the second feature comprises a second body part shown at the second portion of the live view representation, and
wherein producing the swap effect causes (i) the first body part to be shown at the second portion of the live view representation rather than at the first portion of the live view representation and (ii) the second body part to be shown at the first portion of the live view representation rather than at the second portion of the live view representation.

3. The method of claim 2 , wherein the first body part is a first face, and wherein the second body part is a second face.

4. The method of claim 1 , wherein the swap effect comprises a face swap effect.

5. The method of claim 1 ,
wherein the first feature comprises a first clothing item shown at the first portion of the live view representation,
wherein the second feature comprises a second clothing item shown at the second portion of the live view representation, and
wherein producing the swap effect causes (i) the first clothing item to be shown at the second portion of the live view representation rather than at the first portion of the live view representation and (ii) the second clothing item to be shown at the first portion of the live view representation rather than at the second portion of the live view representation.

6. The method of claim 1 , wherein receiving control input indicative of a swap effect comprises receiving control input indicative of a selected icon from a plurality of icons, and wherein the selected icon corresponds to the swap effect.

7. The method of claim 6 , wherein the interface of the computing device comprises a touch-based control interface.

8. The method of claim 7 , wherein the touch-based control interface comprises one or more of the following: (i) a touch-sensitive surface and (ii) a button.

9. The method of claim 1 , further comprising:
receiving, via the interface of the computing device, control input indicative of a further effect that is different from the swap effect; and
in response to receiving control input indicative of the further effect, producing the further effect in the live view representation in real-time.

10. The method of claim 9 , wherein the swap effect and the further effect are produced concurrently in the live view representation in real-time.

11. The method of claim 9 , wherein the further effect comprises at least one of: (i) a slow-motion effect, (ii) a speed-up effect, (iii) a bokeh effect, or (iv) a high dynamic range effect.

12. The method of claim 1 , wherein producing the swap effect comprises producing the swap effect while receiving, via the interface, control input indicative of the swap effect, the method further comprising:
detecting that control input indicative of the swap effect is no longer being received via the interface; and
in response to detecting that control input indicative of the swap effect is no longer being received via the interface, stopping the producing of the swap effect in the live view representation in real-time.

13. A system comprising:
a computing device including a camera, a display, and an interface; and
a control system configured to:
based on image data from the camera, display, on the display, a live view representation that shows (i) a first feature at a first portion of the live view representation and (ii) a second feature at a second portion of the live view representation;
receive, via the interface, control input indicative of a swap effect, wherein the swap effect comprises causing (i) the first feature to be shown at the second portion of the live view representation rather than at the first portion of the live view representation and (ii) the second feature to be shown at the first portion of the live view representation rather than at the second portion of the live view representation; and
in response to receiving control input indicative of the swap effect, produce the swap effect in the live view representation in real-time.

14. The system of claim 13 ,
wherein the first feature comprises a first body part shown at the first portion of the live view representation,
wherein the second feature comprises a second body part shown at the second portion of the live view representation, and
wherein producing the swap effect causes (i) the first body part to be shown at the second portion of the live view representation rather than at the first portion of the live view representation and (ii) the second body part to be shown at the first portion of the live view representation rather than at the second portion of the live view representation.

15. The system of claim 14 , wherein the first body part is a first face, and wherein the second body part is a second face.

16. The system of claim 13 , wherein the swap effect comprises a face swap effect.

17. The system of claim 13 ,
wherein the first feature comprises a first clothing item shown at the first portion of the live view representation,
wherein the second feature comprises a second clothing item shown at the second portion of the live view representation, and
wherein producing the swap effect causes (i) the first clothing item to be shown at the second portion of the live view representation rather than at the first portion of the live view representation and (ii) the second clothing item to be shown at the first portion of the live view representation rather than at the second portion of the live view representation.

18. A non-transitory computer readable medium having stored therein instructions executable by one or more processors to cause a computing device to perform functions comprising:
based on image data from a camera, displaying, on a display of the computing device, a live view representation that shows (i) a first feature at a first portion of the live view representation and (ii) a second feature at a second portion of the live view representation;
receiving, via an interface of the computing device, control input indicative of a swap effect, wherein the swap effect comprises causing (i) the first feature to be shown at the second portion of the live view representation rather than at the first portion of the live view representation and (ii) the second feature to be shown at the first portion of the live view representation rather than at the second portion of the live view representation; and
in response to receiving control input indicative of the swap effect, producing the swap effect in the live view representation in real-time.

1

Assignees

• Google Llc

Inventors

• Zhou Changyin
• Lewkow Roman
• Levoy Marc Stewart
• Chen Jiawen

Classifications

• G06F3/0488 Primary

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06F3/03547

  Touch pads, in which fingers can move on a surface · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title