12 comments

  • kwindla 1801 days ago
    This looks to be nicely architected. Looking forward to digging in.

    In answer to the questions about TURN, this approach won't be lower latency than TURN but will scale better. TURN servers are just (nearly) passive relays, so the sending client needs to set up as many outbound streams as there are receiving clients in the session.

    The advantage to the TURN approach is that you can do end-to-end encryption.

    The server this post is about "forwards" streams. It's in a class of infrastructure traditionally called a "Selective Forwarding Unit." So the sending client can send one stream, and the SFU copies the stream and sends it to the receiving clients. The server needs more CPU and outgoing bandwidth as the receiver count grows, but the sending client doesn't need much of either.

    Once you have streams piping through a server, you can do lots of other things, of course. This server can transcode to several formats, including HLS and RTMP, and can copy streams internally across a cluster to scale more than a single machine can.

    Media servers are really fun to work on: lots of small(-ish) hard problems that touch low-level network protocols, memory and cpu optimization, architecture (because eventually you want a clean plugins interface, etc.).

    If you're interested in this stuff, check out Mediasoup, an open source WebRTC SFU with a very nice design in which all the low-level stuff is c++ and all the high-level interfaces are exposed as nodejs objects.[0]

    [0]-https://mediasoup.org/

    • tpetry 1801 days ago
      The sad part about selective forwarding units (SFU) is you loose end-to-end-encryption. So you only have encryption between you and the sfu and sfu to your participant. For hosted solutions this means a server could record all your conferences.

      Therefore i am waiting for the PERC standard. It will allow to send the audio/video to server only once and the server will retransmit to all peers - everything still end-to-end encrpted.

      • pjc50 1800 days ago
        How does the key distribution work for that?
    • sansnomme 1801 days ago
      How's the performance and features of mediasoup compared to this? I have been looking for an easy to use DFU.
      • kwindla 1800 days ago
        Haven't benchmarked ant media server, but mediaSoup performance is very good.

        "Easy to use" is in the eye of the beholder. Mediasoup isn't a packaged solution; you need to write some javascript to get a server up and running. From this perspective it's a framework that sits at the same level as, say, Express. But the API is flexible and powerful. It's exactly what you want if you're building an SFU cluster for a specific use case and want to decide on the architecture yourself. Here's excellent "1:N broadcast" sample code: [0]

        With that sample code, you can broadcast a single WebRTC stream to ~500 clients on one vCPU of an AWS c5.large instance. To scale to more clients you need to use more than one vCPU. The new mediasoup v3 APIs make this pretty easy. There are now "pipe transports" and "plain rtp transports" for moving the low-level rtp bits around between processes and between machines.[1]

        [0]-https://github.com/michaelfig/mediasoup-broadcast-example [1]-https://mediasoup.org/documentation/v3/differences-between-v...

  • kodablah 1801 days ago
    What makes this approach lower latency than, say, a centralized TURN server in one of those clouds?
    • adreamingsoul 1801 days ago
      From my understanding, the current challenge with ultra-low-latency is scale. Will it scale to 100, 1000, 100,000, concurrent users?

      At the moment, everyone in the industry is working to figure out how to provide a viewer experience that is similar, or better than the current expectations of live streaming.

      Live streaming is hard. Ultra-low-latency is just as hard.

      However, I'm still not sure why we need ultra-low-latency. Will it improve the live experience? Maybe. Aside from live-action sports or significant events, do we as content consumers really need ultra-low-latency?

      • Waterluvian 1801 days ago
        Ultra low latency absolutely improves the streaming experience for community streams like on Twitch. When everyone is +-20 seconds of each other the chat doesn't line up well.
        • AceJohnny2 1801 days ago
          Implying there's value to Twitch chat?
          • thrax 1801 days ago
            Yeah. Like millions if not billions of dollars.
            • physcab 1801 days ago
              As a former Twitch employee, I lol’d at this.
              • grwthckrmstr 1800 days ago
                By that, do you mean there's billions of $$ in Twitch chat, or is the concept too absurd (and hilarious as a result)?
              • gdsdfe 1801 days ago
                Why? I am genuinely curious
                • zaroth 1801 days ago
                  Might be hard for physcab to reply from their yacht, I guess we'll find out eventually though.
                • cavisne 1801 days ago
                  Probably because twitch chat is the absolute darkest place on the internet, at least of you are a mod.
                  • tinus_hn 1800 days ago
                    There’s always YouTube comments.
              • adreamingsoul 1800 days ago
                How so?
          • gsich 1801 days ago
            Feelsbadman

            Twitch chat is fun.

      • Jhsto 1801 days ago
        > However, I'm still not sure why we need ultra-low-latency.

        Simply for passive viewing probably not. For live streaming in which the viewer is also using actuators, e.g., gamepads, wheel, etc., for controlling the stream capture device, then yes.

        • alias_neo 1801 days ago
          You've just done a stellar job of making "gaming" sound like a tedious day job. I'm impressed.
          • Jhsto 1801 days ago
            Yeah I tried too fancy words. What I meant to say that there might be applications beyond gaming, like driving a remote car, vacuum cleaner, or who knows what.
            • alias_neo 1801 days ago
              Your choice of words were perfect, I just found it humorous, as a gamer, it was the first thing that came to mind!
      • harlanji 1801 days ago
        I did coding livestreams and spoke with the audience. The lower latency the better. Others have mentioned control of things, similar idea. HLS over Cloudflare surprisingly beat YouTube and matched Twitch in my side-by-side tests—I built an Open Source self-hosted live streaming system. Currently live sports have about 40-85 seconds of latency and BBC is shooting for 1.5 for the next olympics with ip multicast techniques [1].

        1. http://www.wired.co.uk/article/england-vs-croatia-live-strea...

      • pedrocr 1801 days ago
        How low is ultra-low-latency that it's relevant for sports? Does anyone really care if their broadcast is 30 seconds delayed from live?

        I subscribe to both Formula 1 and MotoGP online streaming services and couldn't care less if the broadcasts were even multiple minutes delayed. They may very well be already and I wouldn't know. As far as I can tell they both just use standard CDNs delivering DASH or similar (basically just short video files played in sequence).

        That seems more than enough for sports. Multiple minutes may result in getting real time information from elsewhere (e.g., Twitter, local radio) but anywhere from 10 to 30 seconds seems fine.

        • oaththrowaway 1801 days ago
          I work for a video streaming company (primarily sports). There is a huge problem with latency in sports.

          We call it the "Twitter effect". Essentially people either see things on social media or app notifications before it happens. Kinda ruins it for some people.

          We are working on sub-30 second latency in 4K w/ server side ad insertion, but it's hard work. Especially with encoding with DRM baked in.

          • isostatic 1800 days ago
            I had a normal latency stream (about 1.5 seconds) of the world cup on one England game. I saw the goal going in about 5-10 seconds before the cheers from the neighbours watching on DTT.

            For the FA Cup final I had two feeds, one the 4K BBC version, and one on SDI from Wembley. The goal went in, and I'd completely forgotten it by the time the 4K goal went in.

            The trouble is, things like twitter, whatsapp etc, will work to the lowest latency, and for a football game that's about 50-600 nanoseconds to the earliest viewer. Twitter can push the message within a few seconds (or the VAR result or whatever)

            Latency needs to be in the sub-10 second goal-glass range.

          • TickleSteve 1801 days ago
            Surely, in that case what would be important is relative-latency, not absolute.

            If you delay all your friends by 30s +-2s then that would probably be fine...

            That changes the problem somewhat.

            • isostatic 1800 days ago
              good luck having a conversation with them with that type of latency
          • pjc50 1800 days ago
            At the top of this thread I was wondering "how many miliseconds is low latency", I hadn't realised it was usually worse than 30 seconds!
          • pedrocr 1801 days ago
            Thanks! Do you know what the equivalent latency number is for cable TV? The Twitter effect only applies for the difference between the two numbers.
            • cstejerean 1801 days ago
              Broadcast TV latency is usually in the 4-10 second range, with typical latency being around 6s. Not sure how it breaks down between cable vs satellite vs over-the-air.
            • oaththrowaway 1801 days ago
              I'm actually not sure. I'm sure it's different for every station and mode of delivery. Our latency is measured directly from the broadcaster signal.
          • timeisapear 1800 days ago
            Fubo? Liked the service during trial but the 4K World Cup stream was flaky.
        • toast0 1801 days ago
          If your friends are watching the same events in other places; you basically need your video to be less delayed than your friends' viewing plus the latency of them texting you (friend observes something, types something, sends, then whatever network and push delay on the way to you).

          If you're not watching with someone else, it doesn't make that much difference; but 30 seconds is plenty for someone to cheer for a goal and the notification to distract you from seeing it.

          • pedrocr 1801 days ago
            That's already an issue with radio vs TV which is where I'm getting the ~10 to 30 second value. Normal cable TV already has a bunch of delay and someone listening on the radio is often 15+ seconds ahead of you. This can be disruptive. If that person is at a bar or other gathering they will disrupt the viewing because they will cheer (or not cheer) well ahead of the event so the suspense is gone. I've seen this happen.

            So given that what you are competing against (cable TV) already has a significant delay I doubt a normal web stream can't reach the same or better latency with just standard HTTP download of video files.

        • daniel-cussen 1801 days ago
          Yes. 30 seconds is too much.

          In Santiago, during national soccer matches everyone would watch, goals would reach us through the sound of neighbors rejoicing sooner than they would from actual television coverage because the satellite dish had a latency of say 3 seconds over cable.

          • pedrocr 1801 days ago
            You're giving an example that validates my point. Existing technologies already have widely variable delays (radio vs broadcast TV vs cable TV vs sattelite TV) and yet they're all in wide use. And you can't compare directly the 30s I suggested (end-to-end latency) with the 3 seconds of latency differential between technologies. It's probably less than that but it's possible your neighbors are in a 30s delay and you are in a 33s delay.

            And if the problem was actually that serious it would be easier to fix it by just adding additional delay to the faster technologies instead of having to reduce latency in internet streaming at huge cost.

            • asutekku 1801 days ago
              Then you would have the people at stadium tweeting about the results and ruining the experience for the rest. We can and we should strive to get the best possible result if it is feasible.
        • ajflores1604 1801 days ago
          Just experienced this delay being an issue for the 3rd period of this vegas-san jose playoff game. When goals pop off as fast as they do and it completely changes the game, that delay matters. Especially when my friends are either at a bar or streaming the game, it's hard to communicate and share the moment without spoiling. I basically have to keep a mental tab on who's stream is delayed and when I can get excited with who.
        • netwanderer3 1801 days ago
          Ultra-low-latency is absolutely vital in sports betting industry where the odds are constantly changing based on what actions are taking place in real time.

          Results in sports can switch from one side to the other end within split second, so whoever has the lowest latency will gain massive advantages. It's very similar to stock trading.

        • brightball 1801 days ago
          Yes, absolutely. Huge deal for live sports.

          It’s the entire reason that I won’t cut the cord.

          • pedrocr 1801 days ago
            The chord you're not cutting already has a large delay from live. If you're in the US even larger because besides the delay from the technology there's also intentional broadcast delay for censorship reasons.
      • harshitaneja 1801 days ago
        Google Stadia(and other online game streaming) and other systems where actuators would be used remotely like drone control systems and others would greatly benefit from ultra low latency.
      • donalhunt 1801 days ago
        industries serving the sports industry / creating products related to sports will be very interested... not your average consumer though...
        • stcredzero 1801 days ago
          I'm interested in low latency WebRTC datachannel. Does this apply to that as well?
      • fabioyy 1801 days ago
        hqtrivia
    • maxmcd 1801 days ago
      I think even with turn there's lots of chatter back and forth. Connected clients can NACK for frames and ask for a bandwidth reduction. I think the expected optimal implementation is that the group of WebRTC participants normalize towards an optimal bitrate.

      For 1->n streaming you just want very high quality broadcast->server and then presumably you want the downstream clients to not have the option to send packets back to the broadcaster. Transcoding is hard though, so consumers might be stuck with just the source video quality. I assume this lib just passes the frames on from the source (twitch did just that through almost all of its pre-acquisition growth).

      Some of my details might be a little off, but I did spend a few months trying to build something very similar to this with gstreamer. WebRTC was hard to grok.

    • mekya 1801 days ago
      Actually it is not lower latency than centralized TURN server. The difference from TURN is that Ant Media Server behaves like peer to each connected clients. Not relaying the video/audio as TURN. Additionally, it can create adaptive bitrates on the server side. It can record streams and control other things on server side.
  • rkagerer 1801 days ago
    500ms is low latency? What's usual?
    • BeefySwain 1801 days ago
      The best Vimeo/Livestream can do is about 30 seconds from the time the frame hits their server, as a reference.
    • isostatic 1801 days ago
      As far as I'm concerned, 500ms is normal latency (once setting aside network delay -- clearly a 200ms one way trip from Europe to NZ will add to that)

      I'd say ultra low latency would be sub-frame, so <40ms. Low latency on the 1 frame to half a second, normal upto say 1.5s, and high latency beyond that.

      But that's for point to point or multicast circuits, and ISPs don't like multicast as they can't charge for it.

    • rasz 1801 days ago
      Twitch random 1080@60 6K viewers stream

      >Latency To Broadcaster: 7.71 sec.

      >Latency Mode: Low Latency

    • naikrovek 1801 days ago
      Multiple seconds.
  • gugagore 1801 days ago
    There's also possible latency savings from having the video encoder be aware of network conditions on a per-frame basis: http://ex.camera but I don't think it's found its way to the "mainstream".
  • dboreham 1801 days ago
    And here I was blissfully unaware that I'm experiencing reality 30 seconds delayed. Had no idea, other than obviously there's some # frames delay needed to do motion compression, and some transit latency.
  • mgamache 1800 days ago
    I've used Wowza (video streaming server) for years and this is a direct competitor. The pricing is a little higher for Wowza, but Wowza is a mature product with tons of options for web streaming. The weakness of Wowza has been its support for WebRTC. I am using it, but it's not easy to stream from RTSP/RTMP to WebRTC. It also doesn't scale out for WebRTC
  • selim17 1801 days ago
  • rv11 1801 days ago
    It will help apps which require interaction from users, like google stadia's game streaming. for other use cases, I think one does not care even if the feed is delayed by few seconds.
    • IshKebab 1801 days ago
      Another interesting use case someone mentioned here before is the world cup - you don't want a penalty shoot-out ruined by your neighbours cheering a few seconds before you see the goal.

      A few seconds would also such for live streaming where the streamer interacts in chat.

  • MuffinFlavored 1801 days ago
    How does one-to-many streaming work? If 256 clients connect to one streamer, how does the streamer not need to send his/her stream 256 times?
    • pault 1801 days ago
      That's what the server is for.
  • fabioyy 1801 days ago
    The problem is that webrtc is a very intensive CPU ( each connection have its own encription ). How many users can this solution handle per server?
    • maxmcd 1801 days ago
      It looks like they support RTMP and a few other streaming methods. Maybe it's just WebRTC for ingress and then it converts to other formats. Given the advertised 500ms of latency I imagine it's not WebRTC->WebRTC.
    • ec109685 1801 days ago
      How is encryption expense different from normal https streaming from a cdn?
      • fabioyy 1801 days ago
        packet size of webrtc(rtp) is VERY VERY VERY small, and since they are UDP, each individual packet must be encrypted. ( i can't just encode a big chunk of data and send to the network like tls ).

        in fact, for scaling a solution that uses "near realtime broadcast", you probably don't want encryption at all. ( afaik you can't disable webrtc encryption. )

        in my tests i can get 3x more performance just using rtmp/rtsp. ( ~8000 video streaming connections on a aws c5.xlarge at 800kbit/s ).

      • est 1801 days ago
        most https connections are ephemeral and limited to connections per domain.

        WebRTC can spawn large number of p2p connections.

  • DiseasedBadger 1800 days ago
    I've heard enough of this anti-anonymity, anti-privacy protocol.