Unity Controller for Tello 3.0 - Quest Control w/ Passthrough

VR_Dev · Oct 3, 2018

THIS IS ALPHA SOFTWARE. IT HAS NOT BEEN WELL TESTED, AND THE INPUTS HAVE NOT BEEN FINE TUNED. I TAKE NO RESPONSIBILITY FOR ANY DAMAGE THAT OCCURS TO YOUR TELLO WHILE USING THIS SOFTWARE. MAKE SURE TO VALIDATE INPUT VALUES IN FLIGHT CONTROLLER COMPONENT BEFORE TAKEOFF

Getting back into this. Quest 3 + AI. Code will be released when ready

This project would not be possible without the hard work put into these two libraries.
Krag's TelloLib : A c# library that allows the tello to be controlled via windows. It also exposes the underlying API which contains the positioning data.
comoc's TelloForUnity : Modifies TelloLib to work in Unity, including the video feed.

3D TRACKING

VR SUPPORT
FPV supported for the Rift and VIve. Rift can use touch controllers but Vive will need joystick or keyboard inupt. To use VR, you need to enable it in the project settings.

INPUT MANAGER (TUNING & JOYSTICK SUPPORT)
All inputs go through the input manager. This is where you can fine tune your selected input. This provides a way to tune your keyboard input, or set up a flight stick. Sensitivity and Deadzone are self explanitory. Gravity is only for the keyboard, and determines how quickly the value returns to 0. Feel free to suggest input values you find. Input manager can be found at Edit - Project Settings - Input. Right now the only flightstick supported is the thrustmaster 1600m and throttle. If you have a joystick you would like to use, I can let you know how to set it up.

INSTALLATION
Download Unity Editor 2018.3
Download UnityControllerForTello HERE
Open Unity and select open project. Navigate to folder you just downloaded.
Allow Unity through firewall via inbound rules Image Example

TAKEOFF
Connect to tello wifi
Press play in editor
Perform pre-flight checks
Wait for video feed
Flight instructions located top left corner

trelo · Oct 12, 2018

Dude, wow.

This looks hyper complicated.
I am amazed that you are creating smth I cannot even comprehend.

Keep up the good work.

VR_Dev · Oct 15, 2018

So I've been trying to make videos but this project is progressing pretty quickly. I'm already testing the autopilot algorithm on the real thing. I have the simulator up and running as a full test bed for development. The scene controller now has the option to switch between flying and simulating. I haven't tried the "Fly&Sim" mode lately so I would stay away from that.

First, I have craft rotation working which makes the third person view far more helpful. This video is the first attempt at flying through a replica of my office my coworker Dwight had on hand. The tracking doesn't start at the most accurate point, and it drops a lot, but you can see the 3rd person craft match the tello rotation.

Here you can see me testing headless flight via keyboard input, then Fly-To-Target Autopilot in the simulator, before giving it a try on the real thing. I have it working fully in the simulator, but the real thing needs some work. I'm pretty sure I am giving the PID controllers the wrong delta time between tello updates. I expect to have this fixed pretty quickly, and then I will be able to set a course for the tello to follow.

So not many people cloned the repo, but if you did; I deleted the whole thing and replaced it. There was something missing in the git ignore so I just started from scratch. If you want to update your repo it's probably just best to destroy it and pull again. Everyone else it's good to go. Just don't play with the autopilot yet with the real thing. Unity may seem intimidating but I'd be happy to talk anyone through it if they want to try it themselves.

trelo said:
Dude, wow.

This looks hyper complicated.
I am amazed that you are creating smth I cannot even comprehend.

Keep up the good work.

Thanks a lot. I know you wont believe me but mapping the tracking wasn't very hard. The guys who deciphered all the low level commands for position deserve most of the credit.

VR_Dev · Oct 23, 2018

Fly-To-Target autopilot testing. PID's seemed to be tuned ok, but they have a hard time fine tuning the tello's position at close range. I've even built a system to update the PID values after the tello has been in close proximity to the target for 3 seconds. When in range I crank up the I value but still not much luck. I will have to do more testing.

My tello seems to be far less stable than when I first bought it. It has taken a fair amount of crashes already, but hopefully I just need new props.

One thing that would help this project dramatically is being able to start the props and take off manually. Other libraries support this function, and the creator of aTelloPilot says this is the cmd that needs to be sent to the tello.

Code:

User Datagram Protocol, Src Port: 6525, Dst Port: 8889
TELLO_CMD
    SOP   : 0xcc
    SIZE  : 176
    CRC8  : 0x7f
    PACT  : 0x60
    DIR   :  -> TO DRONE
    CMD   : TELLO_CMD_STICK (80)
    SEQ   : 0
    DATASZ: 11
    DATA  : 6c 61 0b 5b 28 0d 06 13 21 45 02
        STICK - roll: 364, pitch: 364, thr: 364, yaw:1684, fastmode:0
    CRC16 : 0x7cec

I tried converting that Data packet to this which was also unsuccessful. I kind of just guessed at the conversion to tellolibs packet values.

C#:

public static void StartMotors()
{
    var packet = new byte[] { 0x6c, 0x61, 0x0b, 0x5b, 0x28, 0x0d, 0x06, 0x13, 0x21, 0x45, 0x02, };
    setPacketSequence(packet);
    setPacketCRCs(packet);
    client.Send(packet);
}

Update:

Took a crack at the circle flying. This is the tello following a moving target, instead of jumping from stationary target to stationary target. I think I am going to build a system where the target drone flies from point to point, and the actual tello's only job is to try and match the target drone's position.

This will give me much more control over its flightpath, and will make tuning the PIDs easier by eliminating the large range of inputs.

amandeep · Nov 20, 2018

making a drone simulation in 10 mins

VR_Dev · Nov 30, 2018

amandeep said:
making a drone simulation in 10 mins

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Thanks I'll take a look. Bought a mavik air so I've been pretty distracted with that. This weekend I'll make some progress. Before I got the air I already figured out how to start the props via tellolib, unfortunately it did not solve the takeoff offset problem, but I believe I have found a way to track accurately from takeoff which improves the overall tracking quality through an environment.

VR_Dev · Dec 5, 2018

So like I said I figured out how to start the props manually, but there is still an offset when the tello finds tracking. The tello's tracking isn't accurate until it is about half a meter off the ground. Once it reaches a certain height tracking begins, but there is a random offset applied. I use to handle this by taking off, flying to a known point, then manually enabling tracking taking into account the offset. This worked, but was only as accurate as I could fly to an imaginary point in the air.

Now I track from takeoff, but monitor the tello's tracking for the offset. Once the offset is reported, I begin tracking. By taking the known offset into account, I can keep (0,0,0) as the origin point. I also use the tello's height data to adjust for the elevation the tello was at when tracking began. The offset and height data allow me to track the tello accurately from the position tracking was established.

This is the most impressive tracking I've gotten. Unfortunately this is not a consistent result, even with these short flights. I think there are some tricks I can use to squeeze out some more accuracy though.

nahuel · Dec 8, 2018

Wow! This is amazing... who knew you could do this with suck a little "toy" drone? Amazing job!

bataurus · Dec 28, 2018

Just got a Tello and I have developed a few mobile apps and working on a 2D platformer. I just pulled down the code. Very interesting.

VR_Dev · Dec 28, 2018

bataurus said:
Just got a Tello and I have developed a few mobile apps and working on a 2D platformer. I just pulled down the code. Very interesting.

Cool thanks. Just FYI for everyone this has fallen pretty far behind with my Mavic air and Christmas. Expect a major update in the next couple days with the tracking from takeoff and autopilot updates. Also VR support.

VR_Dev · Jan 8, 2019

So I completely reworked this project from the ground up. I optimized and simplified everything, added tracking from takeoff, re-added rift support, and completely reworked the autopilot system. Instead of giving the tello a target point to fly to, the tello's only job is to try and match the a target tello's position(Blue quad in the video). It's my theory that this will be easier for the PIDs to handle because there is a smaller range of inputs to deal with.

With the tello flying itself you are free to move the target with either the gizmos, my fly-to-target system, or via a custom script. The simulator has also been updated so you can test complex autopilot features without destroying your tello or waking the neighbors.

I have spent little time tuning the autopilot PIDs, so there is a lot of room for improvement there. The autopilot loads a PID profile which can be found in the hierarchy. You can update those values or create your own PIDProfile component. You can also swap PID profiles in flight for different situations. The tellos performance in the simulator will not match the real thing, so don't use the simulator to tune the pids.

What you see in the video is the same project in the github repo. 'T' and 'L' to takeoff and land, 'P' to toggle autopilot on and off. (Any input to the flight controls will also deactivate the autopilot). With the autopilot active you can either drag transforms into the target position of the autopilot, or assign them via script. Autopilot will also automatically deactivate when tracking is lost. YOU MUST BE FOCUSED ON A GAME WINDOW FOR INPUT TO WORK

brick72 · Jan 8, 2019

Wow...!!!!
So Please make this aviable for Android too!!!!!!!!!!!!!!!!

VR_Dev · Jan 14, 2019

brick72 said:
Wow...!!!!
So Please make this aviable for Android too!!!!!!!!!!!!!!!!

Sorry man I don't know what to tell you. I could compile this to an android app but I just don't see how that would work. The point of this is kind of be a platform for people to write complex autopilot code.

It's pretty in depth and I just can't see how it could translate to mobile. If there is some specific feature you would want let me know and I'll think about it.

CapTello · Jul 16, 2019

Hi ! Thanks for your amazing work !

I'm having difficulties into making my drone fly, it starts correctly like 1 out of 4 times.
Each time it doesn't start normally I have a "Tracking Lost" log message, when it happens it cannot fly in autopilot, it can only land

Is there a specific manipulation to avoid that ? Like restarting drone/editor in a particular manner, or setting the drone in a specific way ? I don't get why I have this message, (it appears right at the take off)

Thanks a lot

VR_Dev · Jul 17, 2019

CapTello said:
Hi ! Thanks for your amazing work !

I'm having difficulties into making my drone fly, it starts correctly like 1 out of 4 times.
Each time it doesn't start normally I have a "Tracking Lost" log message, when it happens it cannot fly in autopilot, it can only land

Is there a specific manipulation to avoid that ? Like restarting drone/editor in a particular manner, or setting the drone in a specific way ? I don't get why I have this message, (it appears right at the take off)

Thanks a lot

Hey, thanks a lot for trying my project out. As you can see I haven't worked on this in a while, its definitely my loudest project, so I figured I'd give everyone in apartment a break. Also it's summer here in Chicago so I'm flying my mavic air around a bunch.

Make sure you turn off the tello after each flight, set it on the surface you want to take off from, pointing in the direction you want, then turn it on. I honestly don't remember if this is necessary anymore but that is what I did before each flight. If you are still getting tracking errors, then it is definitely from the tello. Make sure you are taking off from a well lit area, and try and have something with a pattern underneath your launch location.

So since I haven't tried this in a while, I haven't confirmed it is working correctly with any new firmware or software updates, maybe I'll give it a try soon and see. I have also been busy with my Kerbal hexapod project.

CapTello · Jul 18, 2019

VR_Dev said:
Hey, thanks a lot for trying my project out. As you can see I haven't worked on this in a while, its definitely my loudest project, so I figured I'd give everyone in apartment a break. Also it's summer here in Chicago so I'm flying my mavic air around a bunch.
View attachment 4371

Make sure you turn off the tello after each flight, set it on the surface you want to take off from, pointing in the direction you want, then turn it on. I honestly don't remember if this is necessary anymore but that is what I did before each flight. If you are still getting tracking errors, then it is definitely from the tello. Make sure you are taking off from a well lit area, and try and have something with a pattern underneath your launch location.

So since I haven't tried this in a while, I haven't confirmed it is working correctly with any new firmware or software updates, maybe I'll give it a try soon and see. I have also been busy with my Kerbal hexapod project.

Hi !

I will follow your advice and tell you if I get better results.

After checking more deeply I found that the drone returns strange posX, posY and posZ values at takeoff (just after reaching the "flying" flight state).
For example my posY value goes from a correct 1.11 (when the drone reaches the flight height) to a 30 in just a frame which causes a too big (>2) difference between positions and results in a tracking loss.
As you supposed, it may be due to the fact that I turned it on before setting it at the launch point.

Will keep you updated.

Thanks !

VR_Dev · Jul 22, 2019

CapTello said:
Hi !

I will follow your advice and tell you if I get better results.

After checking more deeply I found that the drone returns strange posX, posY and posZ values at takeoff (just after reaching the "flying" flight state).
For example my posY value goes from a correct 1.11 (when the drone reaches the flight height) to a 30 in just a frame which causes a too big (>2) difference between positions and results in a tracking loss.
As you supposed, it may be due to the fact that I turned it on before setting it at the launch point.

Will keep you updated.

Thanks !

Yeah the huge tracking offset is well known around here, I track the values from launch and as soon as I detect the huge offset I start factoring that into the tracking calculations to make it look seamless. Thats def not coming from your environment.

CapTello · Aug 20, 2019

VR_Dev said:
Yeah the huge tracking offset is well known around here, I track the values from launch and as soon as I detect the huge offset I start factoring that into the tracking calculations to make it look seamless. Thats def not coming from your environment.

Hi ! (I know it's been a while ;p)

I did the same thing, I take the offset and add it to the calculation.

One more thing, I tried to build the project on Windows and Android, but when using those the autopilot fonctionnality doesn't work. I supposed that the autopilot code is based on specific editor functionalities ? Do you have any idea about the code part that prevent it to work on build ?

Thanks!

VR_Dev · Dec 3, 2019

CapTello said:
Hi ! (I know it's been a while ;p)

I did the same thing, I take the offset and add it to the calculation.

One more thing, I tried to build the project on Windows and Android, but when using those the autopilot fonctionnality doesn't work. I supposed that the autopilot code is based on specific editor functionalities ? Do you have any idea about the code part that prevent it to work on build ?

Thanks!

I would have to try a build. As it stands right now the UI is entirely inspector based. If someone wanted to make an in game UI for this app it could be built to either mac/windows and android/ios.

The main reason I stopped developing this application is how unreliable the tello tracking is. Its pretty much a crapshoot if the tracking will be accurate or not. My fingers are crossed for a Tello 2 with far more reliable tracking.

Also Im terrible at tuning PIDS.

ChinmayNagori · Mar 1, 2020

Hey! I have been trying to use your software. All this is very new to me and I have been trying to make sense of the software.

I was able to control the drone with the keyboard. But I haven't been able to use the tracking part of this software. The tracker senses the drones movement around its axis, but not the spatial movement of the drone. I tried it using both the simulation window and the actual fly window. And when using the simulation, the drones along with the tracking camera seems to keep falling down.
If anybody can help me with this, it would be a great help.