ESP8266 Video Streaming

I just finished the first version of a library + firmware that allows streaming of video data over wifi using packet injection and monitor mode oon the esp8266.

The project is hosted on github:  https://github.com/jeanleflambeur/esp8266_bridge_broadcast

I plan to use it to replace the finicky WN721 wifi cards that I’m using now with something I have more control over.

I’m testing using NodeMCU ESP12F modules and the results are very good.

 

From the Readme:

This lib + firmware allows you to inject and receive packets using an esp8266 module. It’s meant for streaming data – like video – similarly to the wifibroadcast project, but instead of using of the shelf wifi dongles with patched firmwares, it uses the esp8266.

The advantages over standard wifi dongles (like the WN721N) are:

• Ability to control the rate (from 1Mbps to 56Mbps) and modulation (CCK with and w/o short preamble and ODFM)
• Cheaper and readily available
• Very short stack so less points of failure. It doesn’t have to go through the kernel, 802.11 stack, rate control, firmware etc
• Easy to add new features in the firmware
• Very good sensitivity and power: Up to -92dBi and 22.5dBm
• SPI connection so no USB issues on the Raspberry Pi

Disadvantages:

• More complicated connectivity. The module is connected through SPI to the host device which is a bit more complicated than just plugging a USB dongle
• Limited bandwidth. With PIGPIO, 12Mhz SPI speed and 10us delay you can get ~8Mbps throughput. Recommended settings are 10Mhz and 20us delay which results in 5-6Mbps

There are 2 helper classes in the project:

• A Phy which talks to the esp firmware. It supports:
  • Sending and receiving data packets up to 1376K. Data is sent through the SPI bus in packets of 64 bytes. When receiving you get the RSSI as well, per packet.
  • Changing the power settings, in dBm from 0 to 20.5
  • Changing the rate & modulation. These are the supported ones: 0: 802.11b 1Mbps, CCK modulation 1: 802.11b 2Mbps, CCK modulation 2: 802.11b 2Mbps, Short Preamble, CCK modulation 3: 802.11b 5.5Mbps, CCK modulation 4: 802.11b 5.5Mbps, Short Preamble, CCK modulation 5: 802.11b 11Mbps, CCK modulation 6: 802.11b 11Mbps, Short Preamble, CCK modulation 7: 802.11g 6Mbps, ODFM modulation 8: 802.11g 9Mbps, ODFM modulation 9: 802.11g 12Mbps, ODFM modulation 10: 802.11g 18Mbps, ODFM modulation 11: 802.11g 24Mbps, ODFM modulation 12: 802.11g 36Mbps, ODFM modulation 13: 802.11g 48Mbps, ODFM modulation 13: 802.11g 56Mbps, ODFM modulation
  • Changing the channel.*This is broken for now as the radio doesn’t seem to react to this setting for some reason.
  • Getting stats from the esp module – like data transfered, packets dropped etc.
• A FEC_Encoder that does… fec encoding. It allows settings as the K & N parameters (up to 16 and 32 respectively), timeout parameters so in case of packet loss the decoder doesn’t get stuck, blocking and non blocking operation.

Both classes can be used independently in other projects.

Test app

There is also a test app (esp8266_app) that uses them and sends whatever is presented in its stdin and outputs to stdout whatever it received. You can configure the fec params, the spi speeds and the phy rates/power/channel.

Firmware

The firmware is done using the Arduino IDE and can be compiled with the 2.3.0 or 2.4.0 sdk. It does require some patching of the Arduino make command to allow access to an internal function: After downloading the board in arduino, go to packages/esp8266/hardware/2.3.0/platform.txt and locate the compiler.c.elf.flags line:

compiler.c.elf.flags={compiler.warning_flags} -O3 -nostdlib -Wl,--no-check-sections -u call_user_start -u _printf_float -u _scanf_float -Wl,-static "-L{compiler.sdk.path}/lib" "-L{compiler.sdk.path}/ld" "-L{compiler.libc.path}/lib" "-T{build.flash_ld}" -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read

Then add this at the end of that line: -Wl,-wrap=ppEnqueueRxq

It should read this:

compiler.c.elf.flags={compiler.warning_flags} -O3 -nostdlib -Wl,--no-check-sections -u call_user_start -u _printf_float -u _scanf_float -Wl,-static "-L{compiler.sdk.path}/lib" "-L{compiler.sdk.path}/ld" "-L{compiler.libc.path}/lib" "-T{build.flash_ld}" -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read **-Wl,-wrap=ppEnqueueRxq**

si4463 FPV #2

In my last post I talked about using the si4463 chip to send video, telemetry and RC data to the quadcopter. I calculated the bandwidth and it seemed that 500kbps video with 5/7 FEC coding should be possible.

I wrote the code, linked everything together and it kind of worked. The video was stable, not many packets lost but the latency was pretty bad. It got up to ~200ms from the current 100-130ms but worse than this, the video was very choppy. I managed to narrow it down to the H264 codec in the raspberry pi: the bitrate you configure in the codec is an average bitrate, per second. Each frame can vary a lot in size as long as the average is preserved (with some allowed over and undershoot). I got I-frames of 12-16KB and P-frames of 400-500 bytes (at 30 FPS). The I-frames took way longer to send than the P-frames and this resulted in a choppy video. I tried to play around with settings – like disabling CABAC and activating CBR but nothing made the bitstream uniform enough.

The final, biggest problem was actually caused by the RF4463F30 module – and it’s the same problem I had months ago when I tried to use them: they introduce a LOT of noise in the power line, enough to cause all the I2C chips on the quad to fail.

I tried all kinds of capacitors to decouple the module and reduced the noise a lot but there seem to always be some capacitive/inductive coupled noise persisting.

In the end I just gave up and went back to the RFM22B chip which just works. For video I went back to the monitor mode/injection system but did change the modulation to CCK, 5.5Mb to hopefully reduce the possibility of interference with other 2.4Ghz RC systems. CCK is spread spectrum similar to DSSS and should be able to coexist with RC systems around.

 

So yeah, FPV through a very constrained channel with a temperamental H264 encoder and a very temperamental transceiver module is not fun…

 

Video Latency

Just did some latency tests using RUDP through wifi, 640×480@30fps, 2Mbps.

Both the laptop and the quadcopter are in the same room but they go through a router 3 walls away. Signal strengths are (as reported by iwconfig):

Quad:
Link Quality=70/70  Signal level=-37 dBm

Laptop:
Link Quality=58/70  Signal level=-52 dBm

Ping reports these RTTs:
64 bytes from 192.168.1.110: icmp_seq=1 ttl=64 time=137 ms
64 bytes from 192.168.1.110: icmp_seq=2 ttl=64 time=160 ms
64 bytes from 192.168.1.110: icmp_seq=3 ttl=64 time=85.4 ms
64 bytes from 192.168.1.110: icmp_seq=4 ttl=64 time=108 ms
64 bytes from 192.168.1.110: icmp_seq=5 ttl=64 time=125 ms
64 bytes from 192.168.1.110: icmp_seq=6 ttl=64 time=149 ms
64 bytes from 192.168.1.110: icmp_seq=7 ttl=64 time=73.6 ms
64 bytes from 192.168.1.110: icmp_seq=9 ttl=64 time=119 ms

The quadcopter uses the more sensitive alfa card while the laptop has its own crappy RTL8723be card that has many issues under linux…

I happen to live in a building with a very noisy wifi situation so SNR is not good at all.

Average latency is around 100-160ms with random spikes of 3-400 ms every 10-20 or so seconds.

[Edit – just realized that both the brain and the GS are debug versions…]

To measure I pointed the raspicam at my phone’s stopwatch app and then took photos of the phone and the screen at the same time. Here are some of them: