Hello Tello Pilot!
Join our DJI Tello community & remove this banner.
Sign up

New Python library for Tello AI - jetson-tello

robagar

Active member
Joined
Apr 17, 2021
Messages
27
Reaction score
12
Announcing jetson-tello - a new Python library for connecting the Tello with the AI capability provided by NVIDIA's Jetson hardware

Makes it as simple as possible to feed drone video into a neural network for AI processing

Here's an example - detecting faces and recognising objects in view of a flying drone -

Python:
#!/usr/bin/env python3

import asyncio
import jetson.inference
from jetson_tello import h264_frame_to_cuda, FrameDecodeError
from tello_asyncio import Tello

face_detector = jetson.inference.detectNet("facenet", threshold=0.5)
object_detector = jetson.inference.detectNet("ssd-mobilenet-v2", threshold=0.5)


async def process_frame(frame):
    try:
        cuda, width, height = h264_frame_to_cuda(frame)

        face_detections = face_detector.Detect(cuda)
        object_detections = object_detector.Detect(cuda)

        print('faces:')
        for d in face_detections:
            print(d)

        print('objects:')
        for d in object_detections:
            print(d)

    except FrameDecodeError:
        pass   

async def main():
    global next_frame

    drone = Tello()

    await drone.wifi_wait_for_network()
    await drone.connect()
    await drone.start_video()

    async def fly():
        await drone.takeoff()

    async def process_video():
        async for frame in drone.video_stream:
            await process_frame(frame)

    try:
        await asyncio.wait([fly(), process_video()])
    finally:
        await drone.stop_video()
        await drone.disconnect()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
 
  • Like
Reactions: pgminin

pgminin

Well-known member
Joined
Jan 7, 2019
Messages
96
Reaction score
95
Announcing jetson-tello - a new Python library for connecting the Tello with the AI capability provided by NVIDIA's Jetson hardware

Makes it as simple as possible to feed drone video into a neural network for AI processing

Here's an example - detecting faces and recognising objects in view of a flying drone -

Python:
#!/usr/bin/env python3

import asyncio
import jetson.inference
from jetson_tello import h264_frame_to_cuda, FrameDecodeError
from tello_asyncio import Tello

face_detector = jetson.inference.detectNet("facenet", threshold=0.5)
object_detector = jetson.inference.detectNet("ssd-mobilenet-v2", threshold=0.5)


async def process_frame(frame):
    try:
        cuda, width, height = h264_frame_to_cuda(frame)

        face_detections = face_detector.Detect(cuda)
        object_detections = object_detector.Detect(cuda)

        print('faces:')
        for d in face_detections:
            print(d)

        print('objects:')
        for d in object_detections:
            print(d)

    except FrameDecodeError:
        pass

async def main():
    global next_frame

    drone = Tello()

    await drone.wifi_wait_for_network()
    await drone.connect()
    await drone.start_video()

    async def fly():
        await drone.takeoff()

    async def process_video():
        async for frame in drone.video_stream:
            await process_frame(frame)

    try:
        await asyncio.wait([fly(), process_video()])
    finally:
        await drone.stop_video()
        await drone.disconnect()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Hi, please can you provide some numbers on inference time on the jetson, in particular for the mobilnet model?
 

robagar

Active member
Joined
Apr 17, 2021
Messages
27
Reaction score
12
sure, I'll need to add some timing code.. It should also skip frames to keep up with the video, but I haven't verified that it actually is yet
 
  • Like
Reactions: pgminin

robagar

Active member
Joined
Apr 17, 2021
Messages
27
Reaction score
12
ok, here's the detection times for frames captured from the drone, but loaded from file in the object detection example

Code:
detection times (seconds):
  1: 2.303
  2: 0.1095
  3: 0.05508
  4: 0.06146
  5: 0.05088
  6: 0.05064
  7: 0.05104

The first frame is slow, but after that its ~20fps :)

I'll do another experiment with live video from a flying drone & see if makes any difference..
 
  • Like
Reactions: pgminin

robagar

Active member
Joined
Apr 17, 2021
Messages
27
Reaction score
12
Some results from a flying drone:

Code:
  frame #93, detection time 0.04997s, dining table (0.95), chair (0.78), person (0.68), person (0.94)
  frame #95, detection time 0.04945s, dining table (0.96), chair (0.74), person (0.62), person (0.94)
  frame #97, detection time 0.04964s, dining table (0.96), chair (0.78), person (0.74), person (0.94)
  frame #99, detection time 0.04992s, dining table (0.94), chair (0.72), person (0.78), person (0.91)
  frame #101, detection time 0.04975s, dining table (0.97), chair (0.64), person (0.8), person (0.91)
  frame #103, detection time 0.05076s, dining table (0.98), chair (0.57), person (0.66), person (0.91)
  frame #105, detection time 0.05089s, dining table (0.98), chair (0.55), person (0.56), person (0.91)
  frame #107, detection time 0.0501s, dining table (0.97), chair (0.66), person (0.59), person (0.93)
  frame #109, detection time 0.05065s, dining table (0.97), chair (0.63), person (0.6), person (0.93)
  frame #111, detection time 0.05063s, dining table (0.97), chair (0.61), person (0.6), person (0.91)
  frame #114, detection time 0.05047s, dining table (0.88), chair (0.82), person (0.58), person (0.85)

Pretty good, apart from seeing a ghost 👻
 

robagar

Active member
Joined
Apr 17, 2021
Messages
27
Reaction score
12
jetson-tello 1.1.8

Mostly a tidy up, but also added a utility function run_jetson_tello_app which makes it very easy to fly a drone and CUDA process video frames, without falling behind the live video stream.

Here's an example - take off, look around and find faces...

Python:
#!/usr/bin/env python3

import asyncio
import jetson.inference
from jetson_tello import run_jetson_tello_app


face_detector = jetson.inference.detectNet("facenet", threshold=0.5)


def detect_faces(drone, frame, cuda):
    face_detections = face_detector.Detect(cuda)

    for d in face_detections:
        print(d)


async def fly(drone):
    await drone.takeoff()
    for i in range(4):
        await drone.turn_clockwise(90)
        await asyncio.sleep(3)
    await drone.land()


run_jetson_tello_app(fly, process_frame=detect_faces)
 
  • Like
Reactions: AIDrone

New Threads

Members online

Forum statistics

Threads
5,108
Messages
36,932
Members
13,804
Latest member
Dnexus