Hello Tello Pilot!
Join our DJI Tello community & remove this banner.
Sign up

Face tracking with tello and gocv

javaguy

Active member
Joined
May 1, 2018
Messages
29
Reaction score
19
Figured I'd post my OpenCV facetracking project. Sorry for no github project (just me but I hate using github. Never found a convenient way to use it)

I'm using go to do it. You will need to go through these setup steps of installing go/gobot and gocv:
Getting Started - Gobot - Golang framework for robotics, physical computing, and the Internet of Things (IoT)
Getting Started :: GoCV - Golang Computer Vision Using OpenCV 3

I'm testing on a mac, so I don't know how will this program would work on other platforms.

Im using a playstation 4 controller to control the drone. You could replace the joystick controls with keyboard ones and it should still work, but the convenience of having the drone follow you with joystick in hand to take over control is a big feature.

Run the facetracking.go file with the included proto.txt and model files.
An openCV window should open up showing the video feed. I had to make the window that small because otherwise the ffmpeg video conversion speed caused unbearable stutter.
The tracking would work much better/from farther away if I could make the window bigger, but I've not yet found any way to do it and maintain a usable video feed.

To begin tracking your face:
  • Put your face in view of the drone. You should see a green square drawn around it in the video window.
  • Press circle on the controller. This will register the size of your face, so the drone knows to move forwards or backwards if your face gets closer or farther.
  • Walk around and the drone should follow you, as long as your face remains in its view.
  • Press circle again to cancel the tracking and fly normally. The tracking is a toggle, so you can turn it on/off as many times as you want. Main thing is to make sure you first get in view of the drone at the distance you want it to maintain, and then turn tracking on.

As usual, I take no responsibility for what happens to you/your drone.
Although it hasn't happened to me (yet), there is the risk that your face will hit the drone. You could print out an image of a face or load one up on your phone and use that instead.



How it works:
  • ffmpeg execution is started with piped input and output to convert the drones video feed to something opencv can use
  • video feed is received and written to ffmpeg stdin
  • opencv image mat is created and displayed using the data from ffmpeg stdout
  • Using the code at hybridgroup/gocv, face is detected in the image and green square drawn around it
  • Once tracking begins, drone will rotate/move up and down to keep the face in the center of the screen.
  • Drone will move forwards and backwards to try and maintain the initial distance to the face from when tracking was started.

Thanks to @deadprogram for all his work with gobot and gocv, and everyone else here on these forums for all the development work gone into this.
 

Attachments

javaguy

Active member
Joined
May 1, 2018
Messages
29
Reaction score
19
To just fly the drone normally with no face tracking, I recommend you use this tello_control file.

Still uses a playstation 4 controller, but video is displayed with mplayer instead of opencv. In my experience there has been virtually no lag in the video feed.
 

Attachments

jjs357

Active member
Joined
Apr 28, 2018
Messages
30
Reaction score
7
Trying to understand the usage pattern before I try. You have to have your laptop screen (I will also be using a Mac) in view so you can see the opencv framing. You must have the tello facing you so that it's camera can see your face and cause the frame effect to happen. Then if successful, you can walk about the space where you and the tello are, and it will follow you based on the image recognition.

I wonder about seeing the screen and having tello see you at the same time.

Thanks.
 

javaguy

Active member
Joined
May 1, 2018
Messages
29
Reaction score
19
Trying to understand the usage pattern before I try. You have to have your laptop screen (I will also be using a Mac) in view so you can see the opencv framing. You must have the tello facing you so that it's camera can see your face and cause the frame effect to happen. Then if successful, you can walk about the space where you and the tello are, and it will follow you based on the image recognition.

I wonder about seeing the screen and having tello see you at the same time.

Thanks.
You don't need to be next to your computer for the tracking to work. You just need to see the opencv window the first time you run the program, to verify that it works. Point the drone at your face and if you see a green square, youre good to go.

After that, you can go anywhere (as long as you are still within wifi and bluetooth range of your computer) and have it track your face. Just take off and hover the drone a couple inches above eye level before toggling the tracking on. If it doesn't seem to have found your face, you can turn the tracking off and reposition yourself before trying again. After a couple of tries it is easy to recognize from where the drone is whether or not your face is in view.

The only way to do tracking while still seeing the opencv window on your laptop is to fly in front of your computer screen, or to cary your laptop around just as you would the playstation controller.
 

trelo

Well-known member
Joined
Apr 2, 2018
Messages
76
Reaction score
15
To just fly the drone normally with no face tracking, I recommend you use this tello_control file.

Still uses a playstation 4 controller, but video is displayed with mplayer instead of opencv. In my experience there has been virtually no lag in the video feed.
Hey There!

Is there any chance we can have face tracking in a mobile app?

Thanks for your great work.
 

javaguy

Active member
Joined
May 1, 2018
Messages
29
Reaction score
19
Can you make a simple video to show how this works?
:)
A video of the setup? Or a video of it doing the tracking.

Hey There!

Is there any chance we can have face tracking in a mobile app?

Thanks for your great work.
If @Krag allows, I should be able to add an OpenCV plugin for his android app, letting you do face tracking from there. First I've got to get his app working on my own phone though :p
 
  • Like
Reactions: adr94 and trelo

KrazeyKami

Member
Joined
May 21, 2018
Messages
5
Reaction score
2
Hi Javaguy!

First of all, thank you all for your great work!
I'm really new to this (drones AND development) but I'm eager to learn and very curious to Facetrack!

However, after following all steps, and finally running either facetracking.go or tello_control.go, I'm met with the following error:

C:\Users\jdebo\go\src\tellocontrol>go run tello_control.go
# command-line-arguments
.\tello_control.go:49:52: flightData.BatteryPercentage undefined (type *tello.FlightData has no field or method BatteryPercentage, but does have tello.batteryPercentage)

C:\Users\jdebo\go\src\facetrack>go run facetracking.go
# command-line-arguments
.\facetracking.go:231:37: flightData.BatteryPercentage undefined (type *tello.FlightData has no field or method BatteryPercentage, but does have tello.batteryPercentage)

Did I configured something incorrectly? Or can someone shed some light on this? :)

Many thanks in advance!

EDIT:
When I remove the sections from the .go files, I manage to get Tello_Control.go to work. However, when running facetracking.go, the following error is given:
exit status 3221225785
 
Last edited:

jjs357

Active member
Joined
Apr 28, 2018
Messages
30
Reaction score
7
Hi Javaguy!

First of all, thank you all for your great work!
I'm really new to this (drones AND development) but I'm eager to learn and very curious to Facetrack!

However, after following all steps, and finally running either facetracking.go or tello_control.go, I'm met with the following error:

C:\Users\jdebo\go\src\tellocontrol>go run tello_control.go
# command-line-arguments
.\tello_control.go:49:52: flightData.BatteryPercentage undefined (type *tello.FlightData has no field or method BatteryPercentage, but does have tello.batteryPercentage)

C:\Users\jdebo\go\src\facetrack>go run facetracking.go
# command-line-arguments
.\facetracking.go:231:37: flightData.BatteryPercentage undefined (type *tello.FlightData has no field or method BatteryPercentage, but does have tello.batteryPercentage)

Did I configured something incorrectly? Or can someone shed some light on this? :)

Many thanks in advance!

EDIT:
When I remove the sections from the .go files, I manage to get Tello_Control.go to work. However, when running facetracking.go, the following error is given:
exit status 3221225785
There is a reply in the Tello Development topic: Unofficial, very early desktop app for flights from Java Guy that helps with this error
 

javaguy

Active member
Joined
May 1, 2018
Messages
29
Reaction score
19
Hi Javaguy!

First of all, thank you all for your great work!
I'm really new to this (drones AND development) but I'm eager to learn and very curious to Facetrack!

However, after following all steps, and finally running either facetracking.go or tello_control.go, I'm met with the following error:

C:\Users\jdebo\go\src\tellocontrol>go run tello_control.go
# command-line-arguments
.\tello_control.go:49:52: flightData.BatteryPercentage undefined (type *tello.FlightData has no field or method BatteryPercentage, but does have tello.batteryPercentage)

C:\Users\jdebo\go\src\facetrack>go run facetracking.go
# command-line-arguments
.\facetracking.go:231:37: flightData.BatteryPercentage undefined (type *tello.FlightData has no field or method BatteryPercentage, but does have tello.batteryPercentage)

Did I configured something incorrectly? Or can someone shed some light on this? :)

Many thanks in advance!

EDIT:
When I remove the sections from the .go files, I manage to get Tello_Control.go to work. However, when running facetracking.go, the following error is given:
exit status 3221225785
Hi KrazeyKami,
The tello driver files from the main gobot branch don't have exported fields for flight data, meaning things like battery percentage cannot be viewed from outside files. You will have to swap over to the dev branch by executing the following terminal commands:
Code:
cd $GOPATH/src/gobot.io/x/gobot
git checkout dev
git fetch
I'm not sure what that exit status code is. Run the above bash commands and let me know if theres any change.
 

KrazeyKami

Member
Joined
May 21, 2018
Messages
5
Reaction score
2
Hi guys!

When switching to DEV, it works without commenting out the battery sections, thanks for that :)
Unfortunately, I still receive "exit status 3221225785" on running facetracking.go.

According to Windows 10: Fresh Install - exit status 3221225785 · Issue #51 · hybridgroup/gocv, this issue was earlier due to a misconfig in CGO_LDFLAGS. However, I double-checked my settings in ENV.CMD, and they seem to be correct:

set CGO_CFLAGS=-IC:\Users\jdebo\go\src\github.com\veandco\go-sdl2\.go-sdl2-libs\include
set CGO_CPPFLAGS=-IC:\opencv\build\install\include
set CGO_LDFLAGS=-LC:\opencv\build\install\x64\mingw\lib -lopencv_core341 -lopencv_face341 -lopencv_videoio341 -lopencv_imgproc341 -lopencv_highgui341 -lopencv_imgcodecs341 -lopencv_objdetect341 -lopencv_features2d341 -lopencv_video341 -lopencv_dnn341 -lopencv_xfeatures2d341 -lopencv_plot341 -lopencv_tracking341

I'll continue to fiddle with this, see if I can find what's wrong.

Thanks so far! :)
 

javaguy

Active member
Joined
May 1, 2018
Messages
29
Reaction score
19
Hi guys!

When switching to DEV, it works without commenting out the battery sections, thanks for that :)
Unfortunately, I still receive "exit status 3221225785" on running facetracking.go.

According to Windows 10: Fresh Install - exit status 3221225785 · Issue #51 · hybridgroup/gocv, this issue was earlier due to a misconfig in CGO_LDFLAGS. However, I double-checked my settings in ENV.CMD, and they seem to be correct:

set CGO_CFLAGS=-IC:\Users\jdebo\go\src\github.com\veandco\go-sdl2\.go-sdl2-libs\include
set CGO_CPPFLAGS=-IC:\opencv\build\install\include
set CGO_LDFLAGS=-LC:\opencv\build\install\x64\mingw\lib -lopencv_core341 -lopencv_face341 -lopencv_videoio341 -lopencv_imgproc341 -lopencv_highgui341 -lopencv_imgcodecs341 -lopencv_objdetect341 -lopencv_features2d341 -lopencv_video341 -lopencv_dnn341 -lopencv_xfeatures2d341 -lopencv_plot341 -lopencv_tracking341

I'll continue to fiddle with this, see if I can find what's wrong.

Thanks so far! :)
hybridgroup/gocv
Are you running this script in the same window you are running the tello program? The script has to be run for every new command prompt session.
 

KrazeyKami

Member
Joined
May 21, 2018
Messages
5
Reaction score
2
Hi Javaguy!

Yep, I do. I've found the issue however; I messed up the installation for SDL2. So I re-installed everything, took the time to grasp and find out where to extract the files etc.
It works great now! :)

Thanks for your help!! :D
 

javaguy

Active member
Joined
May 1, 2018
Messages
29
Reaction score
19
Hi Javaguy!

Yep, I do. I've found the issue however; I messed up the installation for SDL2. So I re-installed everything, took the time to grasp and find out where to extract the files etc.
It works great now! :)

Thanks for your help!! :D
Glad to hear.
 
  • Like
Reactions: KrazeyKami

Edvard5

New member
Joined
May 29, 2018
Messages
2
Reaction score
0
Hi,
been messing around for almost 6 hours, still opencv not working correctly, install failed at 48%, SDL2, not install either.. Don't know what I'm doing wrong. Hope someone make noob friendly guide lol..
 

javaguy

Active member
Joined
May 1, 2018
Messages
29
Reaction score
19
Hi,
been messing around for almost 6 hours, still opencv not working correctly, install failed at 48%, SDL2, not install either.. Don't know what I'm doing wrong. Hope someone make noob friendly guide lol..
What OS are you on?
 

KrazeyKami

Member
Joined
May 21, 2018
Messages
5
Reaction score
2
Hi,
been messing around for almost 6 hours, still opencv not working correctly, install failed at 48%, SDL2, not install either.. Don't know what I'm doing wrong. Hope someone make noob friendly guide lol..
I've created a very rudimentary step by step note, more as a reminder to myself how to install this from scratch. Not all download URL's are mentioned like for SDL2, so you might wanna google those. Following this guide I had no issues getting this to work on Win10.

Hope it helps you.

Kami.
 

Attachments

  • Like
Reactions: Edvard5

KrazeyKami

Member
Joined
May 21, 2018
Messages
5
Reaction score
2
Found the downloads for SDL2. Don't forget to run all commands in elevated CMD prompts, and run the ENV cmd before trying to start a project.

Downloads
1.Mingw-builds Mingw-builds [mingw-w64] ?Version: latest (at time of writing 6.3.0)
?Architecture: x86_x64
?Threads: I selected win32 and it worked
?Exception: I selected seh and it worked
?Build revision: I selected 1 and it worked
?Destination Folder: Select a Folder that your Windows User owns, like something under %USERPROFILE%, otherwise you'll have plenty of UAC popups / run everything as admin

2.SDL2 Simple DirectMedia Layer - SDL version 2.0.8 (stable) ?Current Runtime Binaries for win32-x64
?Current Development Libraries for mingw

3.SDL_image, mixer and ttf Index of /projects ?Current Runtime Binaries for win32-x64
?Current Development Libraries for mingw

4.(Optional) Nice tool to edit environment variables About - Rapid Environment Editor
5.(Optional) 7-zip 7-Zip.de
6.(Optional) Super cool cmd alternative ConEmu - Handy Windows Terminal

Building go-sdl2
1.Extract the Runtime Binaries from 2. and 3. to the lib-Folder inside your mingw64-Folder from 1.
2.Extract the x86_64-w64-mingw32-Folder from the Development Libraries into your mingw64-Folder, everything from x86_64-w64-mingw32/bin needs to get into mingw64\bin, etc.
3.Add mingw64/bin and mingw64/lib to your path
4.Run conemu or cmd
?Set the CGO_CFLAGS env variable, so gcc finds the SDL2 headers with set: set CGO_CFLAGS=-IC:\mingw-w64\x86_64-7.3.0-posix-seh-rt_v5-rev0\mingw64\include
?go get go-sdl2 -> go get -v github.com/veandco/go-sdl2/sdl
?go get -v github.com/veandco/go-sdl2/sdl_mixer
?go get -v github.com/veandco/go-sdl2/sdl_ttf
?go get -v github.com/veandco/go-sdl2/sdl_image
 
  • Like
Reactions: JoeBaldy

dopestar667

New member
Joined
Jun 5, 2018
Messages
1
Reaction score
0
Any reason this could not be expanded to whole body tracking? A face or a body is just an image for the app to recognize, right?
 

New Threads

Members online

Forum statistics

Threads
3,439
Messages
26,671
Members
8,932
Latest member
Ula78