Localization of a mobile device based on image and radio words

Patent metadata

Field	Value
Publication number	US-11076264-B2
Application number	US-201816648642-A
Country	US
Kind code	B2
Filing date	Sep 24, 2018
Priority date	Sep 22, 2017
Publication date	Jul 27, 2021
Grant date	Jul 27, 2021

Abstract

A mobile device that is able to locate itself in a topological map using a combination of visual and radio information is provided. The mobile device captures an image of its environment, detects receivable radio transmitters and builds an observation vector that defines visual words and radio words that are present at the current location of the mobile device. It then compares this observation vector to reference vectors corresponding to reference location and identifies, based on correlations between visual words and radio words, the location of the mobile device among reference locations.

First claim

The invention claimed is:

1. A mobile device comprising:
one or more digital image sensors onboard the mobile device;
a radio receiver;
a communication link to one or more memories storing:
a set of visual words, a visual word being representative of an object or shape found in an image;
a set of radio words, a radio word being defined at least by an identifier of a radio transmitter;
a set of reference locations, each reference location having associated therewith a reference location vector comprising binary values defining respectively the presence or absence, at the corresponding reference location, of visual words and radio words;

a communication link to a processing logic comprising:
an adaptation to acquire an image of an environment of the mobile device from said one or more digital image sensors onboard the mobile device;
an adaptation to detect receivable radio transmitters using said radio receiver;
an adaptation to create an observation vector comprising binary values defining the presence or absence of visual words in the image acquired from said one or more digital image sensors onboard the mobile device, and binary values defining the presence or absence of radio words in receivable radio transmitters;
an adaptation to determine the current location of the mobile device at least among the set of reference locations based on said observation vector and said reference locations vectors,

wherein said adaptation to determine the current location of the mobile device comprises:
an adaptation to compute, for each reference location, a likelihood of observing the observation vector, when the mobile device is in said reference location, based on said reference location vector and correlations between elements of observations and reference vectors, the correlations of elements associated to visual words and of elements associated to radio words being separately computed; and
an adaptation to compute a probability of observing the observation vector, when the mobile device is in an unknown location, based on a set of artificial reference location vectors built using an artificial list of receivable radio transmitters generated from at least the set of radio words.

2. The mobile device of claim 1 , wherein said calculation, for said reference location vector, of the likelihood of observing the observation vector when the mobile device is in the corresponding reference location is based on the observation vector, the reference location vector, and a correlation tree of elements of the observation and reference location vectors.

3. The mobile device of claim 1 , wherein the processing logic further comprises, when said adaptation to determine if a current location of the mobile device is in one of said reference locations has determined that the mobile device is in none of said reference locations, an adaptation to add a further reference location having as reference location vector the observation vector in the set of reference locations.

4. The mobile device of claim 1 , to wherein the set of radio words is built during a single training phase, comprising a detection of radio transmitters during an exploration of said environment.

5. The mobile device of claim 1 , wherein the processing logic further comprises, when said adaptation to detect receivable radio transmitters using said radio receiver has detected a receivable radio transmitter associated to none of the radio words, an adaptation to add a radio word associated to said receivable radio transmitter in said set of radio words.

6. The mobile device of claim 5 , wherein the processing logic further comprises, when said adaptation to determine if a current location of the mobile device is in one of said reference locations has determined that the mobile device is in none of said reference locations, an adaptation to add a further reference location having as reference location vector the observation vector in the set of reference locations and
wherein said adaptation to add a radio word associated to said receivable radio transmitter comprises an adaptation to modify the reference location vectors to add:
a binary value indicating the presence of said radio word in the reference vector corresponding to current location;
a binary value indicating the absence of said radio word in each other reference location.

7. The mobile device of claim 1 , wherein:
the radio words are further defined by a range of signal strength received from the corresponding radio transmitter;
said adaptation to create an observation vector comprises an adaptation to indicate that a radio word is present in the observation vector, if the radio receiver receives a signal from a corresponding radio transmitter with a corresponding signal strength.

8. The mobile device of claim 1 , wherein said adaptation to determine the current location of the mobile device further comprises adaptations to:
calculate probabilities that the mobile device is in each reference location based on said observation vector and said reference location vectors;
determine that the current location of the mobile device is the reference location having the highest probability.

9. The mobile device of claim 8 , wherein:
said adaptation to calculate probabilities that the mobile device is in each reference location furthermore comprises an adaptation to calculate a probability that the mobile device is in an unknown location;
the sum of the probabilities that the mobile device is in the reference locations, and the probability that the mobile device is in an unknown location is equal to 1.

10. A method comprising:
acquiring an image of the environment of a mobile device from one or more digital image sensors onboard the mobile device;
detecting receivable radio transmitters using a radio receiver onboard said mobile device;
accessing a memory storing:
a set of visual words, a visual word being representative of an object or shape found in an image;
a set of radio words a radio word being defined at least by an identifier of a radio transmitter; and
a set of reference locations, each reference location having associated therewith a reference location vector comprising binary values defining the presence or absence, at the corresponding reference location, of visual words and radio words;

creating an observation vector comprising binary values defining the presence or absence of visual words in the image acquired from said one or more digital image sensors onboard the mobile device, and binary values defining the presence or absence of radio words in receivable radio transmitters;
determining if a current location of the mobile device is in one of said reference locations based on said observation vector and said reference location vectors, said determining comprising:
computing, for each reference location, a likelihood of observing the observation vector, when the mobile device is in said reference location, based on said reference location vector and correlations between elements of observations and reference vectors, the correlations of elements associated to visual words and of elements associated to radio words being separately computed;
computing a probability of observing the observation vector, when the mobile device is in an unknown location, based on a set of artificial reference location vectors built using an artificial list of receivable radio transmitters generated from at least the set of radio words.

11. A non-transitory, computer-readable medium comprising instructions that when executed by a processor cause the processor to:
acquire an image of the environment of a mobile device from one or more digital image sensors onboard the mobile d

Assignees

• Softbank Robotics Europe
• Armines

Inventors

• Nowakowski Mathieu
• Joly Cyril
• Fres Scarlett
• Garcia Nicolas
• Moutarde Fabien

Classifications

• H04W4/024

  Guidance services · CPC title

• G01S5/0252

  Radio frequency fingerprinting · CPC title

• G01C21/206 Primary

  specially adapted for indoor navigation · CPC title

• H04W4/30

  Services specially adapted for particular environments, situations or purposes · CPC title

• H04W64/003

  locating network equipment · CPC title