Tello. Whats possible?

[RawrGuthlaf]

Hi, I decompiled the Android APK and began trying to interpret how the application works. I found a file DreamDroneParser.so in the project structure so decided to upload this to AVG's decompiler to see what the structure looks like. The actual decompiled code is nearly gibberish to me but this call chart kind of makes sense.

Unfortunately I have to re-upload the .so file to get the full decompiled code again as that is a limitation of the AVG's web service session. But this chart should remain available for now.

Anyway, if this is what actually handles the video feed, and not things like object recognition etc, then its possible they use some variation of OpenCV, hinted by the yuv420spToyuv420p, and yuv420pToyuv420sp methods. Theres also some references to h264 in here. Anyone who is more familiar with video formats then me may find this helpful in interpretting the tcpdumps above perhaps?

 

[Krag]

I got tcpdump working on Nox and I can now dump all the traffic (I think). I see nothing but straight UDP traffic. So I don't think ssl is really a factor.

But I don't see any video traffic to port 6038. That might not be surprising since Nox isn't showing video. I though it was because it didn't support the codec but now I am not sure.

Rlouro your capture didn't have any video traffic either. Was it showing video when you did the capture?

 

[Krag]

I think I got it!
-Open port udp port 6038 on computer. The video will be received here.
-Open udp port 6525 on computer. For Tello messages.
-Send from UDP 6525 to 192.168.10.1:8889 the message "conn_req:" plus the video port in binary hex(6038 = 0x1796). See below.
-??
-Profit!!

 

[rjlouro]

Cool! So it seems it's not a video stream that the phone is accessing, but video stream sent from Tello to the device.

It also seems all we need is to send a simple UDP packet to Tello to start receiving the stream.

I did got UDP packets for the video, but they are too large to upload here (15MB). I also tried to save them in a separate .pcap and load that pcap into vlc, but it does not play any video.

 

[deadprogram]

I tried using VLC's "udp://@6038" to supposedly be able to open nearly any format, but it was unable to decode anything.

One note: once you send the "command" command via UDP you can no longer send the "conn_req:" command. Appears that "command" puts the Tello into a specific mode. I have not found any command aka "exit" "quit" etc. that leaves command mode.

Also, the app does not appear to use the "command" mode, but rather to use a binary packet format to send drone commands, as of yet undocumented.

 

[rjlouro]

That's correct, although I did not try to send commands to tello but I believe they will be UDP packets with an yet unknown format.

I'm still puzzled by the video, we not have everything to get the stream but we are unable to decode it. Either the first packets are headers, or it's a proprietary codec. finger crossed for the first option.

 

[deadprogram]

Looking at the video packets, I see the following data:

Frame 274: 1502 bytes on wire (12016 bits), 1502 bytes captured (12016 bits) on interface 0
Ethernet II, Src: SzDjiTec_aa:56:20 (60:60:1f:aa:56:20), Dst: RivetNet_f5:0a:d1 (9c:b6:d0:f5:0a:d1)
Internet Protocol Version 4, Src: 192.168.10.1, Dst: 192.168.10.2
User Datagram Protocol, Src Port: 62513, Dst Port: 6038
Data (1460 bytes)
Data: 75000000000141e80215776c234ed352f1ce4f5afdc561b3...
Text: u
[Length: 1460]

I will surmise to guess the following structure:

The first byte 75 indicates the "video frame number" or something like that.
The second byte 00 indicates the sequence number of the packet within the frame.

The next packet of a frame has the first byte set to "75" with with the second byte part set to "01".
The last packet of a frame has an additional bit set to indicate that it is the last sequence in that frame. For example "84" indicates that sequence "4" is the last packet in that frame.

Anyone have any thoughts on this?

 

[rjlouro]

Mine are a bit different:

Frame 25: 1502 bytes on wire (12016 bits), 1502 bytes captured (12016 bits)
Ethernet II, Src: SzDjiTec_aa:53:e6 (60:60:1f:aa:53:e6), Dst: AsustekC_b9:6b:18 (14:dd:a9:b9:6b:18)
Internet Protocol Version 4, Src: 192.168.10.1, Dst: 192.168.10.3
User Datagram Protocol, Src Port: 62513, Dst Port: 6038
Data (1460 bytes)
Data: 46000000000141e242156b122e4c505f482f0dec4c8c1489...
[Length: 1460]

There's a pattern, every 10 packets the Data starting byte is incremented. Sometimes it's on the 11th package, so we see 9 packets with 1502 bit length and a 10th that is smaller. After that the data first byte is incremented.

I'm not familiar with video transmission or codecs, but I find it strange that they would be developing a proprietary codec.

 

[RawrGuthlaf]

I tried using VLC's "udp://@6038" to supposedly be able to open nearly any format, but it was unable to decode anything.

One note: once you send the "command" command via UDP you can no longer send the "conn_req:" command. Appears that "command" puts the Tello into a specific mode. I have not found any command aka "exit" "quit" etc. that leaves command mode.

Also, the app does not appear to use the "command" mode, but rather to use a binary packet format to send drone commands, as of yet undocumented.

Try some of the different methods seen here. (i.e. remove the @, try with 0.0.0.0 etc).

I would also try RTP, as accessing by UDP may only deliver the raw data with no header info.

And also, perhaps try starting the VLC network stream before initializing the UDP connection as it may contain header info on initialization which VLC would need.

I read this info to try to understand what would happen in a typical H264 stream, but I'm not able to test anything at the moment so just passing this along. Hopefully I'll have time over the weekend to hack away at it more myself.

 

[deadprogram]

Try some of the different methods seen here. (i.e. remove the @, try with 0.0.0.0 etc).

I would also try RTP, as accessing by UDP may only deliver the raw data with no header info.

And also, perhaps try starting the VLC network stream before initializing the UDP connection as it may contain header info on initialization which VLC would need.

I read this info to try to understand what would happen in a typical H264 stream, but I'm not able to test anything at the moment so just passing this along. Hopefully I'll have time over the weekend to hack away at it more myself.

I tried with all those variations, but was not successful.

 

[Krag]

I'm looking at the messages that Tello sends back to 6525.
Byte 0 is always 0xCC.
Bytes 1-3 are unknown.
Byte 4 contains a bit for "pacType". I think this is for multi part packets.
Bytes 5-6 are the "cmdId". This corresponds to code in the APK that can help figure out what each message is and what it contains.
Bytes 7-8 are the message sequence number.
Rest is payload except the last two bytes in the packet are unknown. They might be a checksum.

Of particular interest is cmdId 4182,4183 and 4185. They contain a large amount of uncompressed data. Several K per second. And as far as I can tell it is just saved into the flightlogs .dat files. It might be wishful thinking but it almost looks like image data to me. Maybe from the down facing camera? Wouldn't that be cool.

 

[Krag]

For video you should look at ffmpeg and ffplay. They should be able to handle stuff than even vlc wont and are good at making sense of broken streams.

I tried running it through ffplay and got errors "Unsupported RTP version packet received". Looking that up indicates the solution might be using a .sdp file that describes the incoming video format.

 

[deadprogram]

I'm looking at the messages that Tello sends back to 6525.
Byte 0 is always 0xCC.
Bytes 1-3 are unknown.
Byte 4 contains a bit for "pacType". I think this is for multi part packets.
Bytes 5-6 are the "cmdId". This corresponds to code in the APK that can help figure out what each message is and what it contains.
Bytes 7-8 are the message sequence number.
Rest is payload except the last two bytes in the packet are unknown. They might be a checksum.

Of particular interest is cmdId 4182,4183 and 4185. They contain a large amount of uncompressed data. Several K per second. And as far as I can tell it is just saved into the flightlogs .dat files. It might be wishful thinking but it almost looks like image data to me. Maybe from the down facing camera? Wouldn't that be cool.

Were you able to disassemble the cmdid's from the APK?

 

[Krag]

Not as such. What I do is look at the code where it says if(cmdId == 86) do something. Then I look at the "something" to figure out what it is. For example 86 (0x56) is the packed FlyControllerEntity struct. See post #5 in this thread. That contains pretty much everything about the state of the aircraft.

 

[deadprogram]

Not as such. What I do is look at the code where it says if(cmdId == 86) do something. Then I look at the "something" to figure out what it is. For example 86 (0x56) is the packed FlyControllerEntity struct. See post #5 in this thread. That contains pretty much everything about the state of the aircraft.

There are 37 int values in that struct you mention from your previous post.

However the packet with header byte 0x56 has only 24 bytes that you suggest could be used for data.

 

[Krag]

There are 37 int values in that struct you mention from your previous post.

However the packet with header byte 0x56 has only 24 bytes that you suggest could be used for data.

Its all packed into bits.

  public void setFlyData(byte[] paramArrayOfByte)
  {
    this.height = b.a(b.a(paramArrayOfByte, 0, 2));
    this.northSpeed = b.a(b.a(paramArrayOfByte, 2, 2));
    this.eastSpeed = b.a(b.a(paramArrayOfByte, 4, 2));
    this.flySpeed = ((int)Math.sqrt(Math.pow(this.northSpeed, 2.0D) + Math.pow(this.eastSpeed, 2.0D)));
    this.groundSpeed = ah.a(paramArrayOfByte[6], paramArrayOfByte[7]);
    this.flyTime = ah.a(paramArrayOfByte[8], paramArrayOfByte[9]);
    this.imuState = (paramArrayOfByte[10] >> 0 & 0x1);
    this.pressureState = (paramArrayOfByte[10] >> 1 & 0x1);
    this.downVisualState = (paramArrayOfByte[10] >> 2 & 0x1);
    this.powerState = (paramArrayOfByte[10] >> 3 & 0x1);
    this.batteryState = (paramArrayOfByte[10] >> 4 & 0x1);
    this.gravityState = (paramArrayOfByte[10] >> 5 & 0x1);
    this.windState = (paramArrayOfByte[10] >> 7 & 0x1);
    if (paramArrayOfByte.length < 19) {}
    do
    {
      return;
      this.imuCalibrationState = paramArrayOfByte[11];
      this.batteryPercentage = paramArrayOfByte[12];
      this.droneFlyTimeLeft = b.a(b.a(paramArrayOfByte, 13, 2));
      this.droneBatteryLeft = b.a(b.a(paramArrayOfByte, 15, 2));
      this.eMSky = (paramArrayOfByte[17] >> 0 & 0x1);
      this.eMgroud = (paramArrayOfByte[17] >> 1 & 0x1);
      this.eMOpen = (paramArrayOfByte[17] >> 2 & 0x1);
      this.droneHover = (paramArrayOfByte[17] >> 3 & 0x1);
      this.outageRecording = (paramArrayOfByte[17] >> 4 & 0x1);
      this.batteryLow = (paramArrayOfByte[17] >> 5 & 0x1);
      this.batteryLower = (paramArrayOfByte[17] >> 6 & 0x1);
      this.factoryMode = (paramArrayOfByte[17] >> 7 & 0x1);
      this.flyMode = paramArrayOfByte[18];
      this.throwFlyTimer = paramArrayOfByte[19];
      EventBus.getDefault().post(new EventCenter(248, Integer.valueOf(this.throwFlyTimer)));
      this.cameraState = paramArrayOfByte[20];
      if (paramArrayOfByte.length >= 22) {
        this.electricalMachineryState = (paramArrayOfByte[21] & 0xFF);
      }
      if (paramArrayOfByte.length >= 23)
      {
        this.frontIn = (paramArrayOfByte[22] >> 0 & 0x1);
        this.frontOut = (paramArrayOfByte[22] >> 1 & 0x1);
        this.frontLSC = (paramArrayOfByte[22] >> 2 & 0x1);
      }
    } while (paramArrayOfByte.length < 24);
    this.temperatureHeight = (paramArrayOfByte[23] >> 0 & 0x1);
  }

 

[Krag]

Mine are a bit different:

There's a pattern, every 10 packets the Data starting byte is incremented. Sometimes it's on the 11th package, so we see 9 packets with 1502 bit length and a 10th that is smaller. After that the data first byte is incremented.

I'm not familiar with video transmission or codecs, but I find it strange that they would be developing a proprietary codec.

I think I know what is going on here. UDP has a limited packet size so they are breaking a video frame up into multiple chunks. You probably have to remove the extra packet bytes, assemble them (and potentially reorder them) before sending them to the RTP decoder.

 

[Krag]

Here is what I can figure out of the commands based on the code. The comments are from the logs and were translated from Chinese.

//DroneLog
DroneLog.java:46:    cmdId:4176//Write file header
DroneLog.java:51:    cmdId:4177//Write data
DroneLog.java:56:    cmdId:4178//Write configuration
    
//FlyData   
FlyData.java:193:    cmdId:86//FlyData
FlyData.java:198:    cmdId:69//Answer - see the version number
FlyData.java:203:    cmdId:73//answer-loader version
FlyData.java:208:    cmdId:40//Answer-Get Rate Rate Parameters
FlyData.java:213:    cmdId:4182//Acknowledge - Get Height Limit Parameters
FlyData.java:218:    cmdId:4183//Answer - Get low-power parameter
FlyData.java:223:    cmdId:4185//Answer - Get attitude angle
FlyData.java:228:    cmdId:71//Answer - Aircraft Activation Time
FlyData.java:233:    cmdId:26//?? this.d, this.e
FlyData.java:238:    cmdId:53//?? this.c

//Wifi
Wifi.java:132:    cmdId:16//??
Wifi.java:137:    cmdId:17//Answer-ssid Get
Wifi.java:142:    cmdId:18//answer-ssid settings
Wifi.java:147:    cmdId:19//answer-wifi password retrieval
Wifi.java:152:    cmdId:20//answer-wifi password settings
Wifi.java:157:    cmdId:21//Answer - Get Country Code
Wifi.java:162:    cmdId:22//Answer - Set Country Code

//
ryzerobotics\tello\gcs\core\cmd\h.java:16:    cmdId:70//??

//??
ryzerobotics\tello\gcs\core\cmd\j.java:171:    cmdId:84//Answer - One Click Off
ryzerobotics\tello\gcs\core\cmd\j.java:176:    cmdId:85//Answer-OneKeyDown
ryzerobotics\tello\gcs\core\cmd\j.java:181:    cmdId:82//l
ryzerobotics\tello\gcs\core\cmd\j.java:186:    cmdId:83//m
ryzerobotics\tello\gcs\core\cmd\j.java:191:    cmdId:88//g
ryzerobotics\tello\gcs\core\cmd\j.java:196:    cmdId:4181//Answer-Low Battery Threshold
ryzerobotics\tello\gcs\core\cmd\j.java:201:    cmdId:4184//Answer-set attitude angle
ryzerobotics\tello\gcs\core\cmd\j.java:206:    cmdId:89//f
ryzerobotics\tello\gcs\core\cmd\j.java:211:    cmdId:92//Answer-turnover
ryzerobotics\tello\gcs\core\cmd\j.java:216:    cmdId:93//Answer-throw fly
ryzerobotics\tello\gcs\core\cmd\j.java:221:    cmdId:94//answer-palm landing
ryzerobotics\tello\gcs\core\cmd\j.java:226:    cmdId:4180//Answer-center-of-gravity/calibration plane calibration
ryzerobotics\tello\gcs\core\cmd\j.java:231:    cmdId:55//Answer-toggle JPEG photo quality

//??
ryzerobotics\tello\gcs\core\cmd\k.java:25:    cmdId:68//??
ryzerobotics\tello\gcs\core\cmd\k.java:30:    cmdId:67//??

//Firmware
Firmware.java:247:    cmdId:97 
Firmware.java:253:      cmdId:98
Firmware.java:263:      cmdId:99
Firmware.java:274:    cmdId:101

//FirmwareLog
FirmwareLog.java:177:    cmdId:112
FirmwareLog.java:182:    cmdId:113
FirmwareLog.java:187:    cmdId:114
FirmwareLog.java:192:    cmdId:116
FirmwareLog.java:197:    cmdId:117

//IMU?
IMU.java:36:    cmdId:90//handleIMUStart
IMU.java:41:    cmdId:91//??

//??
ryzerobotics\tello\gcs\core\cmd\o.java:69:    cmdId:32//Answer - set coding rate
ryzerobotics\tello\gcs\core\cmd\o.java:74:    cmdId:33//??
ryzerobotics\tello\gcs\core\cmd\o.java:79:    cmdId:34//??
ryzerobotics\tello\gcs\core\cmd\o.java:84:    cmdId:35//??
ryzerobotics\tello\gcs\core\cmd\o.java:89:    cmdId:36//Answer-EIS settings

//Aircraft setting??
ryzerobotics\tello\gcs\core\cmd\p.java:75:    cmdId:37
ryzerobotics\tello\gcs\core\cmd\p.java:80:    cmdId:48
ryzerobotics\tello\gcs\core\cmd\p.java:85:    cmdId:49
ryzerobotics\tello\gcs\core\cmd\p.java:90:    cmdId:50
ryzerobotics\tello\gcs\core\cmd\p.java:95:    cmdId:51
ryzerobotics\tello\gcs\core\cmd\p.java:100:    cmdId:52

//??
ryzerobotics\tello\gcs\core\cmd\q.java:75:    cmdId:4179//Response - Xiao Huangren
ryzerobotics\tello\gcs\core\cmd\q.java:80:    cmdId:128//Answer - Wisdom Start/End
ryzerobotics\tello\gcs\core\cmd\q.java:85:    cmdId:129//Answer - Wisdom Entry/Exit

//??
ryzerobotics\tello\gcs\core\cmd\s.java:15:    cmdId:65//??

 

[Krag]

The command packet actually contains two crc's. the first one is called uCRC and is for the first 3 bytes of the packet and is stored in the 4th byte.
The second one is a checksum called Frame Checking Sequence 16 and it is stored in the last 2 bytes of the packet. Here is the code to calculate both the uCRC and the FCS CRC for a packet. We should be able to create our own command packets now.

Calc uCRC and FCS16 in C# | .NET Fiddle

 

[deadprogram]

Its all packed into bits.

  public void setFlyData(byte[] paramArrayOfByte)
  {
    this.height = b.a(b.a(paramArrayOfByte, 0, 2));
    this.northSpeed = b.a(b.a(paramArrayOfByte, 2, 2));
    this.eastSpeed = b.a(b.a(paramArrayOfByte, 4, 2));
    this.flySpeed = ((int)Math.sqrt(Math.pow(this.northSpeed, 2.0D) + Math.pow(this.eastSpeed, 2.0D)));
    this.groundSpeed = ah.a(paramArrayOfByte[6], paramArrayOfByte[7]);
    this.flyTime = ah.a(paramArrayOfByte[8], paramArrayOfByte[9]);
    this.imuState = (paramArrayOfByte[10] >> 0 & 0x1);
    this.pressureState = (paramArrayOfByte[10] >> 1 & 0x1);
    this.downVisualState = (paramArrayOfByte[10] >> 2 & 0x1);
    this.powerState = (paramArrayOfByte[10] >> 3 & 0x1);
    this.batteryState = (paramArrayOfByte[10] >> 4 & 0x1);
    this.gravityState = (paramArrayOfByte[10] >> 5 & 0x1);
    this.windState = (paramArrayOfByte[10] >> 7 & 0x1);
    if (paramArrayOfByte.length < 19) {}
    do
    {
      return;
      this.imuCalibrationState = paramArrayOfByte[11];
      this.batteryPercentage = paramArrayOfByte[12];
      this.droneFlyTimeLeft = b.a(b.a(paramArrayOfByte, 13, 2));
      this.droneBatteryLeft = b.a(b.a(paramArrayOfByte, 15, 2));
      this.eMSky = (paramArrayOfByte[17] >> 0 & 0x1);
      this.eMgroud = (paramArrayOfByte[17] >> 1 & 0x1);
      this.eMOpen = (paramArrayOfByte[17] >> 2 & 0x1);
      this.droneHover = (paramArrayOfByte[17] >> 3 & 0x1);
      this.outageRecording = (paramArrayOfByte[17] >> 4 & 0x1);
      this.batteryLow = (paramArrayOfByte[17] >> 5 & 0x1);
      this.batteryLower = (paramArrayOfByte[17] >> 6 & 0x1);
      this.factoryMode = (paramArrayOfByte[17] >> 7 & 0x1);
      this.flyMode = paramArrayOfByte[18];
      this.throwFlyTimer = paramArrayOfByte[19];
      EventBus.getDefault().post(new EventCenter(248, Integer.valueOf(this.throwFlyTimer)));
      this.cameraState = paramArrayOfByte[20];
      if (paramArrayOfByte.length >= 22) {
        this.electricalMachineryState = (paramArrayOfByte[21] & 0xFF);
      }
      if (paramArrayOfByte.length >= 23)
      {
        this.frontIn = (paramArrayOfByte[22] >> 0 & 0x1);
        this.frontOut = (paramArrayOfByte[22] >> 1 & 0x1);
        this.frontLSC = (paramArrayOfByte[22] >> 2 & 0x1);
      }
    } while (paramArrayOfByte.length < 24);
    this.temperatureHeight = (paramArrayOfByte[23] >> 0 & 0x1);
  }

Very useful thanks Krag! I can now parse the flight data packets in Gobot:

hybridgroup/gobot

Seems to be working, though most of the data coming from drone are zeros:

2018/04/07 21:20:17 Starting work...
&{batteryLow:0 batteryLower:0 batteryPercentage:86 batteryState:0 cameraState:0 downVisualState:0 droneBatteryLeft:4153
droneFlyTimeLeft:0 droneHover:0 eMOpen:0 eMSky:0 eMgroud:0 eastSpeed:0 electricalMachineryState:0 factoryMode:0 flyMod
e:1 flySpeed:0 flyTime:0 frontIn:0 frontLSC:0 frontOut:0 gravityState:0 groundSpeed:0 height:0 imuCalibrationState:0 im
uState:0 lightStrength:0 northSpeed:0 outageRecording:0 powerState:0 pressureState:0 smartVideoExitMode:0 temperatureHe
ight:0 throwFlyTimer:0 wifiDisturb:0 wifiStrength:0 windState:0}