Tello localization using downward and frontal cameras, landmarks map, and a particle filter

pgminin · Jun 30, 2021

Monte Carlo Localization for Tello

Tello downward camera is used for motion estimation:
position is integrated from optical flow velocity estimation (dead-reckoning).

Tello frontal camera is used for observation estimation:
landmarks (bottles) are detected using deep learning, and drone localization from known bottle positions is made using photogrammetry.

Estimations fusion:
motion and observation estimations are combined using a particle filter.

Video References:

Code Reference:

GitHub - AtsushiSakai/PythonRobotics: Python sample codes and textbook for robotics algorithms.

Python sample codes and textbook for robotics algorithms. - AtsushiSakai/PythonRobotics

github.com

AIDrone · Jul 10, 2021

Hi,

I have been following your work with Tello & how you have been pushing the boundaries of drone with power of AI and ML. I can find so many uses cases of the Tello Localisation work you have done here.

I am a developer and have built a Drone AI Software for flying Tello from macOS and Windows. May be we can collaborate on some ideas. My software allows bunch of cool ML features. Tello Localisation implementation will be one level up.

Let me know your thoughts.

pgminin · Jul 11, 2021

AIDrone said:
Hi,

I have been following your work with Tello & how you have been pushing the boundaries of drone with power of AI and ML. I can find so many uses cases of the Tello Localisation work you have done here.

I am a developer and have built a Drone AI Software for flying Tello from macOS and Windows. May be we can collaborate on some ideas. My software allows bunch of cool ML features. Tello Localisation implementation will be one level up.

Let me know your thoughts.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Hi, thank you for your nice words. I have more ideas than time to develop all of them. Now I'm working on Simultaneous Localization and Mapping and the idea is to port those algorithms on my Android App.
At the same time, I'm evaluating the bundle adjustment problem with dense reconstruction. A lot of things and for now I don't think I've time for a collaboration. Maybe in the future!

Saemon · Jul 16, 2021

AIDrone said:
Hi Aidrone!
I am glad that I met this news on the forum) the application is interesting, I will test it. Mobile version apk to run own script will be great.Please do not stop developing.

Saemon · Jul 23, 2021

pgminin said:
Hi, thank you for your nice words. I have more ideas than time to develop all of them. Now I'm working on Simultaneous Localization and Mapping and the idea is to port those algorithms on my Android App.
At the same time, I'm evaluating the bundle adjustment problem with dense reconstruction. A lot of things and for now I don't think I've time for a collaboration. Maybe in the future!

Did you try build a 3d reconstruct? My fast experience in 3d pdf file

pgminin · Jul 24, 2021

Saemon said:
Did you try build a 3d reconstruct? My fast experience in 3d pdf file

Hi, what sw did you use? The scene.pdf seems to be blank

Saemon · Jul 24, 2021

pgminin said:
Hi, what sw did you use? The scene.pdf seems to be blank

Can use any 3d pdf reader, not just pdf reader. 3d model is not good, i dont know how use a position of drone, collaborate this coordinate with foto. This should increase the quality.
I used one flight of the drone, half a circle, split into frames and used 3d reconstruct app.

pgminin · Jul 24, 2021

Saemon said:
Can use any 3d pdf reader, not just pdf reader. 3d model is not good, i dont know how use a position of drone, collaborate this coordinate with foto. This should increase the quality.
I used one flight of the drone, half a circle, split into frames and used 3d reconstruct app.

Ok now I see it.
Overall it's not so bad, I did the same outdoor with a full orbit and the quality of my 3d was very low (worst than yours) so I did not publish the result yet.

Here is a quick screen image of the 3d model using COLMAP for 3d reconstruction from video frames of the orbit (converges only on medium quality):

Saemon · Jul 24, 2021

pgminin said:
Ok now I see it.
Overall it's not so bad, I did the same outdoor with a full orbit and the quality of my 3d was very low (worst than yours) so I did not publish the result yet.

Here is a quick screen image of the 3d model using COLMAP for 3d reconstruction from video frames of the orbit (converges only on medium quality):

View attachment 6919

I think building models is a matter of time. just using 3df zephyr, with default settings gives this effect. and this is without neural networks and any ml. now it seems to me important to associate frames from the video with the distance to the object.

pgminin · Jul 24, 2021

Saemon said:
I think building models is a matter of time. just using 3df zephyr, with default settings gives this effect. and this is without neural networks and any ml. now it seems to me important to associate frames from the video with the distance to the object.

Very interesting, I will investigate 3df zephyr performance on my orbit, I see the 50 images limitation for the free version but your result is very promising.
I think that my distance estimation and localization are pretty noisy and not so good to improve the full bundle adjustment implementation of a complete 3d sw. But I'll see

Hacky · Jul 24, 2021

Any of you who knows how to access the downward facing cam stream? This is mentioned as new feature in the new firmware update for the Tello EDU.

AIDrone · Aug 1, 2021

pgminin said:
To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Monte Carlo Localization for Tello

Tello downward camera is used for motion estimation:
position is integrated from optical flow velocity estimation (dead-reckoning).

Tello frontal camera is used for observation estimation:
landmarks (bottles) are detected using deep learning, and drone localization from known bottle positions is made using photogrammetry.

Estimations fusion:
motion and observation estimations are combined using a particle filter.

Video References:

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Code Reference:

GitHub - AtsushiSakai/PythonRobotics: Python sample codes and textbook for robotics algorithms.

Python sample codes and textbook for robotics algorithms. - AtsushiSakai/PythonRobotics

github.com

How were you able to access the downward facing camera? Can you please share your thoughts on it.

Hacky · Aug 1, 2021

@pgminin Thank you for that link to an interesting collection.

@AIDrone
As I understand pgminin's post, he is not actually accessing the video stream of the downward facing camera (I would also be interested in that...). He is using the movement information generated through the optical flow logic and accessible from the drone status.