Image Segmentation with Touch Interaction

Patent metadata

Field	Value
Publication number	US-2019272642-A1
Application number	US-201916369750-A
Country	US
Kind code	A1
Filing date	Mar 29, 2019
Priority date	Dec 30, 2016
Publication date	Sep 5, 2019
Grant date	—

Abstract

In one embodiment, a method includes detecting one or more objects in an image, generating at least one mask for each of the detected objects, wherein each of the masks is defined by a perimeter, classifying the detected objects, receiving gesture input in relation to the image, determining whether one or more locations associated with the gesture input correlate with any of the masks, and providing feedback regarding the image in response to the gesture input. Each of the masks may include data identifying the corresponding detected object, and the perimeter of each mask may correspond to a perimeter of the corresponding detected object. The perimeter of the corresponding detected object may separate the detected object from one or more portions of the image that are distinct from the detected object.

First claim

1 - 20 . (canceled)

21 . A method comprising:
by a computing device, detecting first and second objects in an image; by the computing device, generating a first mask for the first object and a second mask for the second object, wherein each mask is defined by a corresponding perimeter; by the computing device, classifying the first and second objects, wherein the classifying identifies a first classification based on the first object and a second classification based on the second object; by the computing device, receiving first gesture input in relation to the image; by the computing device, identifying a first location associated with the first gesture input, wherein the first location corresponds to the first mask for the first object; by the computing device, identifying a relationship between the first and second objects; and by the computing device, providing first feedback regarding the image, wherein the first feedback is in response to the first gesture input and comprises a description of the relationship between the first and second objects.

22 . The method of claim 21 , wherein the first location associated with the first gesture input is within the perimeter that corresponds to the first mask.

23 . The method of claim 21 , wherein the description comprises identifications of the first and second objects based on the respective first and second classifications.

24 . The method of claim 23 , wherein the identifications of the first and second objects comprise names of the first and second objects, and the names are based on the first and second classifications.

25 . The method of claim 21 , wherein the first and second classifications are identified using an image classification algorithm.

26 . The method of claim 21 , wherein the relationship between the first and second objects is identified using an image segmentation or classification algorithm.

27 . The method of claim 26 , further comprising:
by the computing device, determining a size ratio between the first and second objects, wherein the size ratio is based on the respective first and second classifications; by the computing device, determining a relative position of the first object in relation to the second object, wherein the relative position is determined based on a comparison of the relative sizes of the first and second objects in the image to the size ratio, wherein the relative position of the first object in relation to the second object indicates whether the first object is in front of or behind the second object, and the description of the relationship between the first and second objects describes the relative position of the first object in relation to the second object.

28 . The method of claim 26 , wherein the first location associated with the first gesture input corresponds to a region in which first and second objects overlap,
wherein the detecting the first and second objects in the image is performed by the image segmentation algorithm, and the image segmentation algorithm further determines that the first object is in front of the second object, and wherein the description of the relationship between the first and second objects identifies the first and second objects and indicates that the first object is in front of the second object.

29 . The method of claim 28 , further comprising:
by the computing device, receiving second gesture input in relation to the image, wherein the second gesture input is received subsequently to the first gesture input; by the computing device, identifying a second location associated with the second gesture input, wherein the second location corresponds to the first mask for the first object; and by the computing device, providing second feedback regarding the image, wherein the second feedback is in response to the second gesture input and identifies the second object.

30 . The method of claim 29 , wherein the first and second gesture inputs comprise subsequent taps in the region in which first and second objects overlap on a touch screen of the computing device.

31 . The method of claim 26 , wherein the second object is contained within the first object,
wherein the detecting the first and second objects in the image is performed by the image segmentation algorithm, and the image segmentation algorithm further generates a list of one or more objects nested within the first object, the list of objects including the second object, and wherein the description indicates that the first object contains other objects.

32 . The method of claim 31 , wherein the description further comprises the names or types of the one or more objects nested within the first object.

33 . The method of claim 31 , wherein a size threshold is associated with one or more of the objects nested within the first object, and
wherein the description further includes one or more descriptions of one or more of the objects nested within the first object that are smaller than the size threshold.

34 . The method of claim 26 , wherein the first object is contained within the second object,
wherein the detecting the first and second objects in the image is performed by the image segmentation algorithm, and the image segmentation algorithm further generates a list of one or more objects nested within the second object, the list of objects including the first object, and wherein the description indicates that the first object is contained in the second object.

35 . The method of claim 21 , wherein each of the masks comprises data identifying the corresponding detected object, and the perimeter that defines each mask corresponds to a perimeter of the detected object for which the mask is generated.

36 . The method of claim 21 , wherein the first feedback is played as speech by an audio output component of the computing device.

37 . The method of claim 21 , wherein the first feedback is displayed as text by a display component of the computing device.

38 . The method of claim 21 , wherein the first gesture input comprises a swipe gesture across a touch screen of the computing device or a tap gesture on the touch screen, at least a portion of the gesture input is detected at the first location, and at least a portion of the image is displayed on the touch screen at the first location.

39 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
detect first and second objects in an image; generate a first mask for the first object and a second mask for the second object, wherein each mask is defined by a corresponding perimeter; classify the first and second objects, wherein the classifying identifies a first classification based on the first object and a second classification based on the second object; receive first gesture input in relation to the image; identify a first location associated with the first gesture input, wherein the first location corresponds to the first mask for the first object; identify a relationship between the first and second objects; and provide first feedback regarding the image, wherein the first feedback is in response to the first gesture input and comprises a description of the relationship between the first and second objects.

40 . A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to:
detect first and second objects in a

Assignees

• Facebook Inc

Inventors

• Cheung Vincent Charles
• Ho Connie Yeewei
• Paluri Balmanohar

Classifications

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title