Inspiration
Remote Desktop solutions are terrible.
The old champions like VNC don't scale at all with higher bandwidths and the others are mostly proprietary paid services.
In recent years we've seen the explosion of video streaming as an entertainment medium - both through playing (NVIDIA's GameStream, Steam In-Home Streaming, OnLive...) and watching (Twitch, YouTube)
Our project approaches the first kind, and implements remote desktop without needing specific brand hardware or proprietary software.
What it does
FastMirror is a screen streaming software. It wraps around the complicated fuss of low latency, high bandwidth streaming using experimental, poorly tested ffmpeg tricks.
It needs to get a PC's screen, encode it into a video, push it through the network, and play it.
All within <100ms, or it will be unusable, while still being orders of magnitude faster than the alternatives.
How we built it
The sever (PC that sends the video stream) relays the screen to a modified nginx with RTMP support running on localhost. Clients connect to the RTMP stream and display it with MPlayer.
Keyboard and mouse handling is done via python and xdotool. A preliminary unified approach to these two parts of the program should be present.
With proper recent hardware able to encode webm and a cable connection the system is able to push lossless display feeds with sub 0.5s latency through Ethernet.
On constrained WiFi or 10 year old CPUs, it's able to stream the computer's display with reasonable quality, which is more than we ever imagined possible.
Challenges we ran into
Streaming low latency video in 2016 through the web is sadly impossible.
The challenges were many and apparently unsolvable according to the status quo.
But we tried anyway.
Our first goal was to minimize encoding delay. The video taken from the screen had to be compressed, and that takes time. ffmpeg is probably the most undocumented piece of black magic ever known to mankind, and getting through it used up our mind the entire 2 days of the hack.
Gathering configuration options and endless trial and error succeeded into creating the perfect ffmpeg command to get a screen and push it. On Core2Duo systems usually incapable of encoding at more than 3 FPS.
We wanted to make the GUI for the app in Electron, however we found out the hard truth then. Using web technologies, the only way to stream is to use webm (vp8/vp9) or a pure RTMP stream in the old, dead flash player.
The first proved challenging considering our computers didn't have hardware encoders for webm (and nobody does, since those first started appearing last year!) Direct PC to stream proved impossible, so a middleman had to happen.
Software encoding webm on a system that was already running under load was ridiculous. Even when unconstrained, it would add massive input lag.
The grasp of Adobe into streaming was so large streaming can't be conceived without them. RTMP was made to be a protocol used by them and Flash Player, but many open source implementations of it have started to appear. Browsers, however, don't support it. We tried many different implementations and approaches for an entire day, but in the end, without any other options, we moved onto a native display.
It'd be easy to think this would solve all of our problems, but truth is, they hadn't even started.
MPlayer is the only video player capable of streaming low latency video, period. No other player can, we are unsure why we as a species got to that point, but that's how things are. It also needs launch parameters that tell it to disable most of its functionality, and lose some quality. All in the name of low input lag.
Better, newer hardware would've made things easier for us, as our computers were simply not powerful enough to be considered worthy of streaming anything. But in the end, it worked.
Maybe a little too close to the end because we ran out of time!
The python communication of keystrokes is not completely finished, mainly because it's our third one (the first one ran on nodejs for electron, then we had another one in regular js!) thanks to the constant scrapping of platforms. Thanks to streaming not being in HTML5, or anywhere in the web.
But how do Twitch and YouTube do it?
They have massive encoding lag. Twitch moved recently from pure RTMP to mp4 streams and the suffer now from almost minute long delays. YouTube is lower, but it has massive server farms powering it hundreds of times more powerful than regular laptops.
The other similar services for screen streaming locally have similar tricks in the ffmpeg bag, adding custom silicon for decoding/encoding to further minimize lag.
Accomplishments that we are proud of
We got within an inch of them in two days, which is awesome!
Seeing how we got stream delay from 20 seconds to .2 was amazing, and so was watching my laptop's screen on my friend's computer.
We actually got to learn lots of useful languages, and techniques. Some of them were a bit niche, but actually quite useful.
What We learned
We learned the intricacies of ffmpeg, video streaming, python, pygtk, javascript, some more nodejs for server usage, keyboard and mouse capture... we scripted them with xdotool, even programmed a bit of Junction Plays Pokemon to be demoed if the time came. We had a lot of fun doing all of this.
For some reason Ruben's laptop kept nuking the HDD's MBR and it required GRUB to be reinstalled every boot. We tried not to reboot it.
What's next for FastMirror
Finish the mouse handling! It needs to be able to actually play FPS games, not just RPGs. It will also get proper per-platform hardware encoding and decoding once we get back to our main rigs.
(Let's not talk about 6AM ssh'ing into your home 2000km away trying to get vdpau parameters)
Log in or sign up for Devpost to join the conversation.