Gaze direction-based adaptive pre-filtering of video data

Patent metadata

Field	Value
Publication number	US-11295425-B2
Application number	US-202017112708-A
Country	US
Kind code	B2
Filing date	Dec 4, 2020
Priority date	Jul 21, 2017
Publication date	Apr 5, 2022
Grant date	Apr 5, 2022

Abstract

A multi-layer low-pass filter is used to filter a first frame of video data representing at least a portion of an environment of an individual. A first layer of the filter has a first filtering resolution setting for a first subset of the first frame, while a second layer of the filter has a second filtering resolution setting for a second subset. The first subset includes a data element positioned along a direction of a gaze of the individual, and the second subset of the frame surrounds the first subset. A result of the filtering is compressed and transmitted via a network to a video processing engine configured to generate a modified visual representation of the environment.

First claim

What is claimed is:

1. A method, comprising:
filtering one or more video frames using a first set of values of parameters of a multi-layer low-pass filter of a device, wherein the multi-layer low-pass filter comprises a first layer and a second layer with respective filtering resolutions, the filtering comprising:
for individual ones of the one or more video frames, applying the first layer to a first pixel subset of the video frame, and applying the second layer to a second pixel subset of the video frame;

determining one or more properties of content of the one or more video frames; and
modifying, based at least in part on the one or more properties, a value of at least one parameter of the multi-layer low-pass filter.

2. The method as recited in claim 1 , wherein the one or more properties of the content comprises a contrast property.

3. The method as recited in claim 1 , wherein the modifying comprises changing a size of a subset of a video frame to be filtered using the first layer of the multi-layer low-pass filter.

4. The method as recited in claim 1 , wherein the device comprises a wearable device.

5. The method as recited in claim 1 , wherein the device comprises a mixed reality system device.

6. The method as recited in claim 1 , wherein a perimeter of a portion of a video frame filtered using the first layer comprises one of: (a) a circle, (b) an oval, or (c) a polygon.

7. The method as recited in claim 1 , further comprising:
displaying a modified visual representation of an environment represented in the one or more video frames, wherein the modified visual representation is obtained using a video processing engine to which a result of the filtering is provided as input.

8. A system, comprising:
one or more processors; and
one or more memories;
wherein the one or more memories store program instructions that when executed on or across the one or more processors perform a method comprising:
filtering one or more video frames using a first set of values of parameters of a multi-layer low-pass filter, wherein the multi-layer low-pass filter comprises a first layer and a second layer with respective filtering resolutions, the filtering comprising:
for individual ones of the one or more video frames, applying the first layer to a first pixel subset of the video frame, and applying the second layer to a second pixel subset of the video frame;

determining one or more properties of content of the one or more video frames; and
modifying, based at least in part on the one or more properties, a value of at least one parameter of the multi-layer low-pass filter.

9. The system as recited in claim 8 , wherein the one or more properties of the content comprises a contrast property.

10. The system as recited in claim 8 , wherein the modifying comprises changing a size of a subset of a video frame to be filtered using the first layer of the multi-layer low-pass filter.

11. The system as recited in claim 8 , wherein the method further comprises:
modifying values of one or more parameters of the multi-layer low-pass filter based at least in part on feedback obtained from a user of a device at which the filtering is performed.

12. The system as recited in claim 8 , wherein a perimeter of a portion of a video frame filtered using the first layer comprises one of: (a) a circle, (b) an oval, or (c) a polygon.

13. The system as recited in claim 8 , wherein the method further comprises:
compressing a result of the filtering of the one or more video frames; and
transmitting a result of the compressing to a video processing engine configured to generate a modified visual representation of an environment represented in the one or more video frames.

14. The system as recited in claim 8 , wherein the method further comprises:
combining, using a blending function, a first filtering output produced by the first layer and a second filtering output produced by the second layer; and
transmitting a result of the combining to a video processing engine configured to generate a modified visual representation of an environment represented in the one or more video frames.

15. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to perform a method comprising:
filtering one or more video frames using a first set of values of parameters of a multi-layer low-pass filter, wherein the multi-layer low-pass filter comprises a first layer and a second layer with respective filtering resolutions, the filtering comprising:
for individual ones of the one or more video frames, applying the first layer to a first pixel subset of the video frame, and applying the second layer to a second pixel subset of the video frame;

determining one or more properties of content of the one or more video frames; and
modifying, based at least in part on the one or more properties, a value of at least one parameter of the multi-layer low-pass filter.

16. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the one or more properties of the content comprises a contrast property.

17. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the modifying comprises changing a size of a subset of a video frame to be filtered using the first layer of the multi-layer low-pass filter.

18. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the method further comprises:
modifying values of one or more parameters of the multi-layer low-pass filter based at least in part on feedback obtained from a user of a device at which the filtering is performed.

19. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein a perimeter of a first portion of a video frame filtered using the first layer has a first shape, and wherein a parameter of a second portion of the video frame filtered using the second layer has a shape which differs from the first shape.

20. The one or more non-transitory computer-accessible storage media as recited in claim 15 , wherein the method further comprises:
combining, using a blending function, a first filtering output produced by the first layer and a second filtering output produced by the second layer; and
transmitting a result of the combining to a video processing engine configured to generate a modified visual representation of an environment represented in the one or more video frames.

Assignees

• Apple Inc

Inventors

• Jin Can
• Bonnier Nicolas Pierre Marie Frederic
• Pan Hao

Classifications

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B27/017

  Head mounted · CPC title

• G06T5/20 Primary

  using local operators · CPC title

• G02B27/0172

  characterised by optical features · CPC title