Unofficial, very early desktop app for flights

socketbind · May 13, 2018

Hi,

I'd like to share with you the desktop application I have been working on:
socketbind/drone-control

The reason I have been doing it is that my old Nexus 5 has connection issues with the drone at the higher distances and altitudes and also as an excuse for practicing Go.

So far I got these things working in a very early, alpha quality:

Displays drone camera in a window (uses libavcodec)
Allows control with a Dualshock 4 gamepad (with hard coded button mappings for now)
Dumps video frames as H.264 NALs under the recordings/ folder. You can convert these to actual video files using ffmpeg, but writing in a proper container format should not be so hard to implement.

It works quite decently using my laptop but I urge extreme caution for anybody who would like to try it as the button mappings for your gamepad could be horribly off and cause your drone to crash. It might also be a little problematic to compile the native parts on some platforms, esp. Windows. I suggest you to try Linux or Mac OS but I might be able to look into Windows compatibility sometime soon.

Pull requests are also very much welcome. The code in the repository is the result of a weekend hacking session so the quality might not live up to the highest of standards. I intended to expand upon when time allows and would love to see some proper status indicators (battery, wifi, speed, altitude).

tashcz · May 13, 2018

Great contribution. Is there a chance you could fork this and make a binary/source/whatever that only decodes realtime video? I'm going through your source code now but you're deff more familiar with it. It's the MOST lacking thing with trying to do anything with scripting/using the API.

socketbind · May 13, 2018

I lack the time for that at the moment, sorry, but it shouldn't be too hard. Getting rid of the gamepad code in ui/main.go then anything related to commandChannel in drone/control.go should strip everything to its bare essentials. Thankfully the gobot API is easy to use and not too much complicated.

tashcz · May 13, 2018

Thank you! Will follow through and get back with a result, maybe someone else will want it too.

socketbind · May 13, 2018

As a sidenote another somewhat interesting thing I would love to have is automatically stitching panoramas using rotation controls and the onboard 5MP camera which takes kinda decent pictures. There is seemingly no API for that yet, might need to do some reverse engineering.

tashcz · May 13, 2018

There is a lot that can be done... I'm looking at gobot's examples now, your code also, everything.

What's worst is that they sell this as a "hackable" drone (or we could call it that way) but they barely provided enough functions/methods/w/e for this thing to get airborne. If the community wasn't doing 99% work we'd have zero functionality.

Must say it's a realy new product, but after all, you can't advertise it as something that it currently isn't. I mean, we can do more hacks on a atmel based controller than this... and it's open source. Wonder what's the problem of giving us the APK source or whatever they have on their side. I think at least 50% owners would want a desktop based app for controlling a drone. Eliminate the need for a range extender with a good wifi card/antenna on the field, etc...

At least they could've given us the battery info from the API lol.

tashcz · May 13, 2018

Been wondering something... I know it doesn't have IR or ultrasonic sensors anywhere except IR on the bottom, but maybe some info from the camera can do mapping of the room to avoid collisions with walls inside. Would need a neural network though, but with a 360 turn it could get info of how much and how fast the surroundings change. Could be used for mapping.

tashcz · May 14, 2018

How much video lag do you get from the PC?

socketbind · May 15, 2018

I would say I noticed no lag just some rare dropped/corrupt frames.

In the meantime I'm trying to produce some Linux binaries when time permits but I get a "Unable to initialize decoder" error on Ubuntu 18.04 without any explanation.

tashcz · May 15, 2018

I get a lot of corrupt frames even with the auto bitrate asked from the drone. Also with the 0.5MB/s setting. But I do get a lot of lag, basicly unusable. Might be my way of decoding the video, will have to try something else.

Where are you getting that? Can you commit it to a different git branch?

socketbind · May 15, 2018

On which platform are you trying to get it to work?

I originally got this working on Mac OS and now I'm attempting to do the same on Linux. Sadly, libavcodec throws some kind of generic error message that I can't decipher.

You can try it for yourself like this:

Install libavcodec-dev, libavformat-dev, libavutil-dev through apt-get or by installing it from FFmpeg source.
Then just issue "go get http://github.com.socketbind/drone-control"

It should attempt to build the native parts which will succeed as of now and puts a binary under $GOPATH/bin/drone-control. Sadly it fails like this:
gabriel@silencio:~/Desktop/drone-workspace$ bin/drone-control
2018/05/15 23:21:14 Unable to create decoder

I guess it must be something trivial. Will have another go at it. Perhaps I should do the same on Windows too.

socketbind · May 15, 2018

I admit I fooled myself. That log message comes from my source not avcodec's logging facility. I sprinkled the native code with some helpful error messages to stderr. It seemingly can't find the specified codecs which strikes me as kind of strange since Ubuntu's ffmpeg does have the libx264 encoder enabled but not h.264 decoder?

socketbind · May 15, 2018

All it needed was avcodec_register_all();

On Mac OS I had the avcodec version which ships with FFmpeg 4.0 but on Linux it was 3.4.2-2. It seems there is no need to call this method on newer versions but mandatory with 3.x series.

Anyway I'm getting corruption and lag similar to what you were describing and many more decoding errors than usual. Perhaps decoding problems are better handled in FFmpeg 4.0. I'm not too much of an expert on H.264 but I will see what I can do when I have more time for hacking on this. Outside help would be greatly appreciated too.

socketbind · May 18, 2018

Did some tests on Linux and Windows and video quality was absolutely horrendous. Mac OS works quite nicely. I'm not sure if it is the wifi adapter I'm using (TP-Link TL-WN721N) but video on these platforms is completely unusable. I will try to get my hands on a different adapter.

cbwang2016 · May 18, 2018

socketbind said:
Did some tests on Linux and Windows and video quality was absolutely horrendous. Mac OS works quite nicely. I'm not sure if it is the wifi adapter I'm using (TP-Link TL-WN721N) but video on these platforms is completely unusable. I will try to get my hands on a different adapter.

I'm on Windows using TL WN-722N Version 1, it works fine.

javaguy · May 18, 2018

Here is what I'm using for just flying the drone normally. It works incredibly well. Have never experienced any lag or stutter with the mplayer window.
Using a ps4 controller.

jjs357 · May 19, 2018

Based on the hackster tello project: Hello, Tello - Hacking Drones With Go - Hackster.io I bought a ps3 controller rather than a ps4 one.

Anyone know whether these go control programs will work with the ps3 controller?

Based on this: hybridgroup/gobot either ps3 or ps4 should work. I will try and see.

jjs357 · May 19, 2018

javaguy said:
Here is what I'm using for just flying the drone normally. It works incredibly well. Have never experienced any lag or stutter with the mplayer window.
Using a ps4 controller.

I am trying this control program.

I saved the source file to a folder. There I run:

go run ./tello_control.go

I get an error:

.# command-line-arguments

/tello_control.go:49:52: flightData.BatteryPercentage undefined (type *tello.FlightData has no field or method BatteryPercentage, but does have tello.batteryPercentage)

I tried a couple of string modifications to get around this but failed. Not much of a go programmer I'm afraid.

Any suggestions?

jjs357 · May 19, 2018

I tried to use sockebind's github source with a ps3 controller and the button presses are echoed to the go terminal but they do not cause the tello to takeoff. I have connected to the tello's network. I do see a video feed on my computer though.

I tried to see where the joystick might be set to be a ps4 controller but did not see any place in the code where this happens.

Maybe someone can enlighten me on where/how to make this change.

jjs357 · May 19, 2018

jjs357 said:
I am trying this control program.

I saved the source file to a folder. There I run:

go run ./tello_control.go

I get an error:

.# command-line-arguments /tello_control.go:49:52: flightData.BatteryPercentage undefined (type *tello.FlightData has no field or method BatteryPercentage, but does have tello.batteryPercentage)

I tried a couple of string modifications to get around this but failed. Not much of a go programmer I'm afraid.

Any suggestions?

I changed the JoyStick to a ps3 version, commented out the lines that refer to BatteryPercentage and then removed the unused strconv import and then the Tello flew for me and the video window was pretty stable as well. Would like to get the Battery Percentage value to show somewhere though, so I wonder why I can't get this part to work.