Hello Tello Pilot!
Join our DJI Tello community & remove this banner.
Sign up

The secret of showing video with OpenCV is...

robagar

Active member
Joined
Apr 17, 2021
Messages
27
Reaction score
12
...threading :D

Basically the OpenCV GUI *has* to run on the main thread, and does not share well. So the trick is to do the drone control on a separate worker thread:
(Using my new tello-asyncio library here, but the principle would be the same for DJITelloPy)

Python:
#!/usr/bin/env python3

import asyncio
from threading import Thread

import cv2  # requires python-opencv

from tello_asyncio import Tello, VIDEO_URL

##############################################################################
# drone control in worker thread

def fly():
    async def main():
        drone = Tello()
        try:
            await drone.connect()
            await drone.start_video()
            await drone.takeoff()
            await drone.turn_clockwise(360)
            await drone.land()
        finally:
            await drone.stop_video()
            await drone.disconnect()

    asyncio.run(main())

fly_thread = Thread(target=fly, daemon=True)
fly_thread.start()

##############################################################################
# Video capture and GUI in main thread

capture = cv2.VideoCapture(VIDEO_URL)
capture.open(VIDEO_URL)

while True:
    grabbed, frame = capture.read()
    if grabbed:
        cv2.imshow('tello-asyncio', frame)
    if cv2.waitKey(1) != -1:
        break

capture.release()
cv2.destroyAllWindows()
 

Hacky

Member
Joined
Jul 19, 2021
Messages
5
Reaction score
0
@robagar Thank you for your example and the useful asyncio implementation.

In order to be able to play around with OpenCV with TELLO sitting on the ground without overheating, I modified your example to add some temperature control and battery monitoring.

It makes use of the new API 3.0 functions "motoron" and "motoroff". When board temperature reaches 85°C (configurable), the motors are switched on and below that threshold they are switched off again. When battery level falls below a certain "critical" level, the program stops motors and video streaming and terminates automatically.

Perhaps you may find it helpful as a basis for further OpenCV experiments.

Python:
#!/usr/bin/env python3

##############################################################################
#
# Allows experimenting with video stream in OpenCV under temperature control
# and battery monitoring.
#
# Adapted from: https://github.com/robagar/tello-asyncio/blob/main/examples/video_opencv.py
# Modified by Helge Hackbarth and provided under the same conditions as the
# original file.
#
# The OpenCV UI must run in the main thread, so the drone control runs in a
# worker thread with its own asyncio event loop.
#
# Please note:
#   - If OpenCV fails to capture any video it gives up without showing the
#     window
#   - The video may lag a few seconds behind the live action, but it should
#     keep up with the live stream after some seconds
#
##############################################################################

import asyncio
import time
from threading import Thread

import cv2  # requires python-opencv

from tello_asyncio import Tello, VIDEO_URL

# constants
TEMP_THRESHOLD = 85
BATTERY_CRITICAL_PERCENT = 20
# global vars
video_started = False
battery_critical = False

print("[main thread] START")

##############################################################################
# drone control in worker thread


def fly():
    print("[fly thread] START")

    async def main():
        global video_started, battery_critical, TEMP_THRESHOLD, BATTERY_CRITICAL_PERCENT
        drone = Tello()
        motors_running = False
        try:
            await asyncio.sleep(1)
            await drone.wifi_wait_for_network(prompt=True)
            await drone.connect()
            await drone.start_video(connect=False)
            video_started = True
            while video_started and not battery_critical:
                if int(drone.temperature.high) >= TEMP_THRESHOLD and not motors_running:
                    # cool down by activating motors
                    try:
                        await drone.motor_on()
                        motors_running = True
                    except:
                        # motors were propably already on
                        pass
                if int(drone.temperature.high) < TEMP_THRESHOLD and motors_running:
                    # turn off motors
                    try:
                        await drone.motor_off()
                        motors_running = False
                    except:
                        # motors were propably already on
                        pass
                if video_started:
                    await asyncio.sleep(10)
                battery = await drone.query_battery()
                print(f"battery: {battery}%")
                battery_critical = int(battery) < BATTERY_CRITICAL_PERCENT
                print(
                    f"temperature: {drone.temperature.low}-{drone.temperature.high}°C"
                )
        finally:
            await drone.stop_video()
            video_started = False
            try:
                if motors_running:
                    await drone.motor_off()
                    motors_running = False
            except:
                # motors were propably already on
                pass
            await drone.disconnect()

    # Python 3.7+
    # asyncio.run(main())
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(main())

    print("[fly thread] END")


# needed for drone.wifi_wait_for_network() in worker thread in Python < 3.8
try:
    asyncio.get_child_watcher()
except NotImplementedError:
    pass

fly_thread = Thread(target=fly, daemon=True)
fly_thread.start()

##############################################################################
# Video capture and GUI in main thread


capture = None
try:
    while not video_started and fly_thread.is_alive():
        time.sleep(0.1)
    print(f"[main thread] OpenCV capturing video from {VIDEO_URL}")
    print(
        f"[main thread] Press Ctrl-C or any key with the OpenCV window focussed to exit (the OpenCV window may take some time to close)"
    )
    capture = cv2.VideoCapture(VIDEO_URL)
    # capture.open(VIDEO_URL)

    while not battery_critical:
        # grab and show video frame in OpenCV window
        grabbed, frame = capture.read()
        if grabbed:
            cv2.imshow("tello-asyncio", frame)

        # process OpenCV events and exit if any key is pressed
        if cv2.waitKey(1) != -1:
            video_started = False
            if not fly_thread is None and fly_thread.is_alive():
                # wait for fly thread to finish
                print("[main thread] waiting for fly_thread to finish...")
                fly_thread.join()
            break
except KeyboardInterrupt:
    pass
finally:
    # tidy up
    if capture:
        capture.release()
    cv2.destroyAllWindows()

print("[main thread] END")
 

robagar

Active member
Joined
Apr 17, 2021
Messages
27
Reaction score
12
Very nice! I really need to update my drone to SDK 3.0 just to get the motor control. It's always overheating while I tinker...

Glad you like the asyncio library!
 

Hacky

Member
Joined
Jul 19, 2021
Messages
5
Reaction score
0
@robagar
Do you have some detailed insights about the external communication with the RMTT module?

Your documentation mentions sending "EXT" commands here: tello_asyncio.state — tello-asyncio 2.1.1 documentation

To be honest, I do not see much benefit from the bulky RMTT device. But I would like to use this mechanism to communicate with my own external devices.

As far as I see from the RMTT_Libs (GitHub - RoboMaster/RMTT_Libs: RoboMaster TelloTalent Arduino Support Package), the ESP32 communicates via the USB (serial) connector of the Tello drone with 1000000 bps 8N1.

I did not dig deeper into these libraries but it should be possible to use them with some adaptions also with other devices capable of USB serial commication.

If it would be possible to just forward any kind of command to that interface using the "EXT ..." command and give back the reply, this would open a completely new world for 3rd party extensions...
 

New Threads

Members online

No members online now.

Forum statistics

Threads
5,108
Messages
36,932
Members
13,804
Latest member
Dnexus