Using face detection to update user interface orientation

Patent metadata

Field	Value
Publication number	US-11119638-B2
Application number	US-201916427702-A
Country	US
Kind code	B2
Filing date	May 31, 2019
Priority date	Sep 28, 2018
Publication date	Sep 14, 2021
Grant date	Sep 14, 2021

Abstract

An orientation of an application user interface (e.g., text and/or content) on a display of a device may be determined by an orientation of the device relative to gravity. In situations where the orientation of the device relative to gravity does not provide a confident or accurate orientation for the application user interface, orientation of the application user interface may be determined from face orientation data. Face orientation data may be obtained from a face detection process operating on images of the user captured during a facial recognition process or during an attention detection process.

First claim

What is claimed is:

1. A method, comprising:
detecting an orientation of a user interface on a display of a device, the device comprising a computer processor and a memory;
in response to a user generated request for the device to perform a face detection process, capturing an image of a face of a user using a camera located on the device, wherein the captured image comprises an image captured while illuminating the user's face with an infrared illuminator located on the device, and wherein the face detection process determines data for an orientation of the user's face from the captured image when the user's face is detected in the captured image;
determining an orientation of the device relative to gravity;
in response to determining the orientation of the device relative to gravity is the device is lying flat, accessing the data for the orientation of the user's face determined by the face detection process;
in response to determining the face of the user is detected in the captured image, updating the orientation of the user interface on the display of the device using the data for the orientation of the user's face determined by the face detection process; and
in response to determining the face of the user is not detected in the captured image, determining the orientation of the user interface on the display of the device remains at a previous orientation of the user interface.

2. The method of claim 1 , further comprising updating the orientation of the user interface on the display of the device using the data for the orientation of the user's face from the captured image in response to determining the orientation of the device relative to gravity is ambiguous.

3. The method of claim 1 , wherein the orientation of the user interface on the display of the device comprises one of a portrait mode or a landscape mode for displaying the user interface.

4. The method of claim 1 , wherein the face detection process comprises:
encoding the captured image to generate feature vectors for the captured image;
assessing the feature vectors to detect the face in the captured image; and
providing an output indicating that the face has been detected or has not been detected in the captured image, wherein the output includes the data for the orientation of the user's face for the output indicating that the face has been detected.

5. The method of claim 1 , wherein the updated orientation is determined using an orientation of the device relative to the orientation of the user's face in the captured image.

6. The method of claim 1 , wherein determining the orientation of the device relative to gravity comprises assessing motion of the device using at least one of an accelerometer and a gyroscope.

7. The method of claim 1 , wherein, in response to the orientation of the device relative to gravity being determined to be not lying flat, the orientation of the user interface is updated using the orientation of the device relative to gravity.

8. The method of claim 1 , wherein the face detection process is operated in response to the user generated request corresponding to either a facial recognition authentication process or an attention detection process.

9. A device, comprising:
a computer processor;
a memory;
a display;
a camera;
at least one illuminator providing infrared illumination;
circuitry coupled to the camera and the illuminator, wherein the circuitry is configured to:
detect an orientation of a user interface on the display;
in response to a user generated request for the device to perform a face detection process, capture an image of a face of a user using a camera located on the device, wherein the captured image comprises an image captured while illuminating the user's face with an infrared illuminator located on the device, and wherein the face detection process determines data for an orientation of the user's face from the captured image when the user's face is detected in the captured image;
determine an orientation of the device relative to gravity;
in response to determining the orientation of the device relative to gravity is the device is lying flat, access the data for the orientation of the user's face determined by the face detection process;
in response to determining the face of the user is detected in the captured image, update the orientation of the user interface on the display using the data for the orientation of the user's face determined by the face detection process; and
in response to determining the face of the user is not detected in the captured image, determine the orientation of the user interface on the display of the device remains at a previous orientation of the user interface.

10. The device of claim 9 , wherein the at least one illuminator comprises a flood infrared illuminator.

11. The device of claim 9 , further comprising one or more inertial sensors on the device, wherein the orientation of the device relative to gravity is determined using at least one of the inertial sensors.

12. The device of claim 11 , wherein at least one of the inertial sensors comprises an accelerometer.

13. The device of claim 11 , wherein at least one of the inertial sensors comprises a gyroscope.

14. The device of claim 9 , wherein the orientation comprises one of a portrait mode or a landscape mode for displaying the user interface.

15. A method, comprising:
capturing an image of a subject using a camera located on a device in response to a user generated request, the device comprising a computer processor, a memory, and a display, wherein the captured image comprises an image captured while illuminating the subject with an illuminator located on the device;
operating, in response to the user generated request and using the computer processor, a face detection process on the captured image, wherein the face detection process is operated to detect a face of the subject illuminated by the illuminator in the captured image, and wherein the face detection process determines an orientation of the face detected in the capture image when the user's face is detected in the captured image;
determining an orientation of the device relative to gravity based on motion of the device;
in response to the orientation of the device relative to gravity being determined to be the device is lying flat and in response to determining the face of the user is detected in the captured image, determining a selected orientation for a user interface to be displayed on the display of the device based on the determined orientation of the face detected in the captured image;
in response to determining the face of the user is not detected in the captured image, determining the selected orientation for the user interface to be displayed on the display of the device remains at a previous orientation of the user interface; and
displaying the user interface on the display of the device with the selected orientation.

16. The method of claim 15 , wherein the selected orientation comprises one of a portrait mode or a landscape mode for displaying the user interface.

17. The method of claim 15 , further comprising determining the selected orientation for the user interface based on the orientation of the face detected in the captured image in response to the orientation of the device relative to gravity being determined to be ambiguous based on motion of the device.

18. The method of claim 15 , wherein the selected orientation is determined using an orientation of the device relative to the orientation of the face detected in the captured image.

Assignees

• Apple Inc

Inventors

• Ho Kelsey Y
• Blumberg Eric J
• Biron Benjamin
• Terndrup Colin C

Classifications

• H04N23/56

  provided with illuminating means · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06F2221/2149

  Restricted operating environment · CPC title

• G06F2200/1614

  Image rotation following screen orientation, e.g. switching from landscape to portrait mode · CPC title