Deep smartphone sensors fusion for indoor positioning and tracking

Patent metadata

Field	Value
Publication number	US-11927448-B2
Application number	US-202117477701-A
Country	US
Kind code	B2
Filing date	Sep 17, 2021
Priority date	Nov 13, 2020
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

A computer-implemented method of determining a position of a portable electronic device in an indoor environment includes: at a first rate, updating an absolute position of a portable electronic device within the indoor environment based on at least one of radio signal data and magnetic field data captured using the portable electronic device; at a second rate that is different than the first rate, selectively updating an estimated displacement of the portable electronic device within the indoor environment, the updating the estimated displacement comprising generating an estimated displacement, by a neural network module, based on inertial sensor data of the portable electronic device; and determining a present position of the portable electronic device within the indoor environment by updating a previous position based on at least one of (a) the estimated displacement and (b) the absolute position.

First claim

What is claimed is:

1. A computer-implemented method of determining a position of a portable electronic device in an indoor environment, the method comprising:
at a first rate, updating an absolute position of a portable electronic device within the indoor environment based on at least one of radio signal data and magnetic field data captured using the portable electronic device;
at a second rate that is different than the first rate, selectively updating an estimated displacement of the portable electronic device within the indoor environment, the updating the estimated displacement comprising generating an estimated displacement, by a neural network module, based on inertial sensor data of the portable electronic device; and
determining a present position of the portable electronic device within the indoor environment by updating a previous position based on at least one of (a) the estimated displacement and (b) the absolute position.

2. The computer-implemented method of claim 1 , wherein generating the estimated displacement based on the inertial sensor data includes converting the inertial sensor data to a graphical representation and generating the estimated displacement based on the graphical representation.

3. The computer-implemented method of claim 1 wherein the neural network module includes at least one of a convolutional neural network, a bidirectional recurrent neural network, and a neural network.

4. The computer-implemented method of claim 1 further comprising, using the neural network module, determining an activity classification based on the inertial sensor data and generating the estimated displacement further based on the activity classification.

5. The computer-implemented method of claim 4 wherein determining the activity classification includes determining the activity classification using a feed-forward network of the neural network module.

6. The computer-implemented method of claim 4 wherein the activity classification is selected from the group consisting of (a) moving and (b) not moving.

7. The computer-implemented method of claim 6 wherein generating the estimated displacement includes setting the estimated displacement to zero displacement when the activity classification is set to (b) not moving.

8. The computer-implemented method of claim 1 , wherein updating the absolute position includes generating the absolute position based on the radio signal data using a variational autoencoder trained to generate the absolute position from radio signal data.

9. The computer-implemented method of claim 1 wherein the second rate is faster than the first rate.

10. The computer-implemented method of claim 1 wherein the updating the absolute position includes updating the absolute position based on the magnetic field data.

11. The computer-implemented method of claim 10 wherein updating the absolute position includes:
generating graphical representations based on the magnetic field data, wherein the magnetic field data includes a time series of magnetic field values; and
determining the absolute position of the portable electronic device, using a neural network module, based on the graphical representations.

12. The computer-implemented method of claim 11 wherein the neural network module includes one or more convolutional layers and a multichannel input to the one or more convolutional layers.

13. The computer-implemented method of claim 11 , wherein the generating the graphical representations includes:
selecting a subset of magnetic field values of the time series of magnetic field values; and
generating one or more two-dimensional images based on the subset of magnetic field values,
wherein determining the absolute position includes determining the absolute position based on the one or more two-dimensional images.

14. The computer-implemented method of claim 11 , wherein the graphical representations include at least one of a recurrence plot, a Gramian Angular Summation Field, a Gramian Angular Difference Field, and a Markov Transition Field.

15. The computer-implemented method of claim 11 , wherein the graphical representations include at least two different ones of a recurrence plot, a Gramian Angular Summation Field, a Gramian Angular Difference Field, and a Markov Transition Field, wherein the at least two different ones are generated based on the magnetic field values.

16. The computer-implemented method of claim 12 , wherein the neural network module includes one or more recurrent layers, and wherein the one or more recurrent layers receive input from the one or more convolutional layers.

17. The computer-implemented method of claim 12 wherein the neural network module is trained based on characteristics of the magnetic field of the Earth measured within the indoor environment.

18. The computer-implemented method of claim 1 further comprising initializing the neural network module with a starting position of the portable electronic device.

19. A computer-implemented method of training a neural network module to estimate a position of a portable electronic device within an indoor environment, the computer-implemented method comprising:
training, with sensor training data, a motion classifier module of the neural network module to determine a present user activity,
wherein the sensor training data includes inertial sensor data captured while a portable electronic device moved along a path within one or more indoor environments;
training, with the sensor training data, a landmark classifier module of the neural network module to detect landmarks within one or more indoor environments;
generating labels and annotating the sensor training data with the labels, the generating the labels including generating the labels based on user activities determined by the motion classifier and landmarks detected by the landmark classifier; and
further training the neural network module based on (a) the sensor training data and (b) the labels, respectively.

20. The computer-implemented method of claim 19 further comprising by the landmark classifier module:
estimating orientation vectors from the inertial sensor data; and
detecting a landmark based on a change in the orientation vectors.

21. A computer-implemented method of determining an absolute position of a portable electronic device within an indoor environment from magnetic field data, the computer-implemented method comprising:
generating graphical representations based on magnetic field data including a time series of magnetic field values, the magnetic field data generated by a sensor of the portable electronic device within the indoor environment; and
determining an absolute position of the portable electronic device within the indoor environment based on the graphical representations using a neural network module, the neural network module including:
one or more convolutional layers; and
a multichannel input to the one or more convolutional layers.

22. The computer-implemented method of claim 21 , wherein the generating graphical representations based on the magnetic field data includes:
selecting a subset of magnetic field values of the time series of magnetic field values; and
transforming one of (a) the subset of magnetic field values and (b) a projection of the subset of magnetic field values into (c) one or more two-dimensional images; and
generating the graphical representations based on the one or more two-dimensional images.

23. The computer-im

Assignees

• Naver Corp
• Naver Labs Corp

Inventors

• Chidlovskii Boris
• Antsfeld Leonid
• Sansano Emilio

Classifications

• G01S5/02585

  at least one of the measurements being a non-radio measurement · CPC title

• G01C21/188 Primary

  for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G01V3/08

  operating with magnetic or electric fields produced or modified by objects or geological structures or by detecting devices (with electromagnetic waves G01V3/12) · CPC title