ESP8266 Video Streaming

I just finished the first version of a library + firmware that allows streaming of video data over wifi using packet injection and monitor mode oon the esp8266.

The project is hosted on github:  https://github.com/jeanleflambeur/esp8266_bridge_broadcast

I plan to use it to replace the finicky WN721 wifi cards that I’m using now with something I have more control over.

I’m testing using NodeMCU ESP12F modules and the results are very good.

 

From the Readme:

This lib + firmware allows you to inject and receive packets using an esp8266 module. It’s meant for streaming data – like video – similarly to the wifibroadcast project, but instead of using of the shelf wifi dongles with patched firmwares, it uses the esp8266.

The advantages over standard wifi dongles (like the WN721N) are:

• Ability to control the rate (from 1Mbps to 56Mbps) and modulation (CCK with and w/o short preamble and ODFM)
• Cheaper and readily available
• Very short stack so less points of failure. It doesn’t have to go through the kernel, 802.11 stack, rate control, firmware etc
• Easy to add new features in the firmware
• Very good sensitivity and power: Up to -92dBi and 22.5dBm
• SPI connection so no USB issues on the Raspberry Pi

Disadvantages:

• More complicated connectivity. The module is connected through SPI to the host device which is a bit more complicated than just plugging a USB dongle
• Limited bandwidth. With PIGPIO, 12Mhz SPI speed and 10us delay you can get ~8Mbps throughput. Recommended settings are 10Mhz and 20us delay which results in 5-6Mbps

There are 2 helper classes in the project:

• A Phy which talks to the esp firmware. It supports:
  • Sending and receiving data packets up to 1376K. Data is sent through the SPI bus in packets of 64 bytes. When receiving you get the RSSI as well, per packet.
  • Changing the power settings, in dBm from 0 to 20.5
  • Changing the rate & modulation. These are the supported ones: 0: 802.11b 1Mbps, CCK modulation 1: 802.11b 2Mbps, CCK modulation 2: 802.11b 2Mbps, Short Preamble, CCK modulation 3: 802.11b 5.5Mbps, CCK modulation 4: 802.11b 5.5Mbps, Short Preamble, CCK modulation 5: 802.11b 11Mbps, CCK modulation 6: 802.11b 11Mbps, Short Preamble, CCK modulation 7: 802.11g 6Mbps, ODFM modulation 8: 802.11g 9Mbps, ODFM modulation 9: 802.11g 12Mbps, ODFM modulation 10: 802.11g 18Mbps, ODFM modulation 11: 802.11g 24Mbps, ODFM modulation 12: 802.11g 36Mbps, ODFM modulation 13: 802.11g 48Mbps, ODFM modulation 13: 802.11g 56Mbps, ODFM modulation
  • Changing the channel.*This is broken for now as the radio doesn’t seem to react to this setting for some reason.
  • Getting stats from the esp module – like data transfered, packets dropped etc.
• A FEC_Encoder that does… fec encoding. It allows settings as the K & N parameters (up to 16 and 32 respectively), timeout parameters so in case of packet loss the decoder doesn’t get stuck, blocking and non blocking operation.

Both classes can be used independently in other projects.

Test app

There is also a test app (esp8266_app) that uses them and sends whatever is presented in its stdin and outputs to stdout whatever it received. You can configure the fec params, the spi speeds and the phy rates/power/channel.

Firmware

The firmware is done using the Arduino IDE and can be compiled with the 2.3.0 or 2.4.0 sdk. It does require some patching of the Arduino make command to allow access to an internal function: After downloading the board in arduino, go to packages/esp8266/hardware/2.3.0/platform.txt and locate the compiler.c.elf.flags line:

compiler.c.elf.flags={compiler.warning_flags} -O3 -nostdlib -Wl,--no-check-sections -u call_user_start -u _printf_float -u _scanf_float -Wl,-static "-L{compiler.sdk.path}/lib" "-L{compiler.sdk.path}/ld" "-L{compiler.libc.path}/lib" "-T{build.flash_ld}" -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read

Then add this at the end of that line: -Wl,-wrap=ppEnqueueRxq

It should read this:

compiler.c.elf.flags={compiler.warning_flags} -O3 -nostdlib -Wl,--no-check-sections -u call_user_start -u _printf_float -u _scanf_float -Wl,-static "-L{compiler.sdk.path}/lib" "-L{compiler.sdk.path}/ld" "-L{compiler.libc.path}/lib" "-T{build.flash_ld}" -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read **-Wl,-wrap=ppEnqueueRxq**

RCP (the new RUDP)

Some months ago I found this awesome blog with research and a usable implementation of rfmon for uni-directional wireless communication. It uses libpcap and a modified firmware for the TP 721N dongle to send encoded video packets and telemetry to the GS. It’s unidirectional and this has some advantages. The biggest advantage for me was that it doesn’t require pairing nor connection handshake. Perfect for a quad!

So I included rfmon in RUDP and renamed the result to RCP (Reliable Comms Protocol…). It’s bidirectional in my case as I need to send control data to the quad as well but the advantages still apply.

Performance is great, I can send 1024×768 video @30 FPS, 2Mbps using around 4-4.5Mbps with very little CPU usage.

The problem I’ve hit is this: the more data I send, the more packets get lost due to interference, etc. Having a fixed retransmit rate – like every packet 3 times – requires a fixed amount of bandwidth but doesn’t scale at all to low-bandwidth situations. So I need a way to use the least amount of bandwidth possible while still ensuring that frames arrive at the other end.
One solution is to add ACK packets to reduce retransmission – which is what RCP does – but this also has an issue. Every confirmation packet keeps the channel busy for a little while – in turn dropping the total bandwidth of the system.
Current solution is to gather many confirmations and send them with low frequency – around MAX_RETRANSMIT_TIME / 2 (currently every 10ms). So fast enough to avoid retransmission, but not as fast as to keep the channel busy unnecessarily. So far this works beautifully.
Code time!

The brain on the UAV uses RCP setup with a RFMON socket:

 

auto s = new util::RCP_RFMON_Socket("mon0", 5);//5 is the end-point ID
m_socket.reset(s);
m_rcp.reset(new util::RCP);

util::RCP::Socket_Handle handle = m_rcp->add_socket(s);
if (handle >= 0)
{
    m_rcp->set_internal_socket_handle(handle);
    m_rcp->set_socket_handle(SETUP_CHANNEL, handle);
    m_rcp->set_socket_handle(PILOT_CHANNEL, handle);
    m_rcp->set_socket_handle(VIDEO_CHANNEL, handle);
    m_rcp->set_socket_handle(TELEMETRY_CHANNEL, handle);
}

 

So – this code adds a socket to the RCP instance and then instructs all channels to go through this socket. Same for the internal data – which represents the ACKs, pings (for RTT estimation) and connection requests.

The reason for this indirection is to allow different sockets for different channels. For example – use RFMON for unidirectional video streaming and another socket over a 433Mhz radio (like this one) for all other comms and ACKs.

In the near future I’ll try this – sending all channels except video through the RFM22B socket, as this should give me better range & penetration compared to 2.4Ghz.