File-sharing networks also have been using udp for many years. eDonkey, Kad (Kademlia), bitTorrent, and several other p2p networks/protocols use udp.

What games have reliability or security layers? lol the most I've ever seen is a ping, or even just a timer expecting to be constantly receiving data

@@JohnRunyon literally every game, otherwise the players would desync horribly

IPv6 will take care of that with its global unicast addresses eliminating the need for a NAT at all

@@ashleigh. IPv6 isn't fully here just yet though.

@@unicodefox Yes it is, just some ISPs are holding back so they can take advantage of their IPv4 monopoly

@@ashleigh. I would still want a NAT though, because I'd rather not have random threat actors be able to directly send packets to my device.

QUIC has connection numbers and IP/port combinations, you can do those things you want if you really want to.

The middle boxes could try to understand QUIC and that is a good point. But beyond the initial handshake it is encrypted. If the middle boxes want to ship UDP anyway though they would have to just put up with arbitrary non QUIC data. It will be interesting to see how it develops.

You could though easily block a QUIC synflood (say).

I might be misremembering here (it's been a while since I studied QUIC in detail), but isn't a bunch of those snoop-worthy details hidden behind always-on encryption? As I remember it, QUIC is designed to hide away the details of what's going on from those middle boxes, exactly because its designers have foreseen that the middle boxes would otherwise indeed return to their dastardly meddling ways.

The beauty of the design and concept behind QUIC is that, for one, it realises that modern end devices are powerful enough to not need much offloading assistance, and two, that instead of locking in behind hardware accelerators which will hinder any attempts at change for years and years to come, by defining a lot of the handling of the stack in software instead, we can make changes as necessary much, much more easily.

Hadn't even thought of that but for satellite it is a real godsend.

For literal decades no less. Like since at least Quake 3 IIRC.

Video game developers were using UDP because that's what UDP was designed for. Data which doesn't require 100% packet reliability. Sound,video, relative coordinates... Sorry they were no magic future readers :)

"Reliably"- please the last thing I want to hear is reliably and UDP at the same time

Yes. When latency becomes an issue in real time coms you generally need a bespoke solution… which is one reason why udp exists.

@@jancizuletek670 dummy he said reliability BUILT ON TOP OF which is exactly what quic does

RTMFP

Oh you had a complete TCP user space 30 years ago. I did not know of that.

@@richardclegg8027 TCP is very old, it was developed when computers had kilobytes of memory for processes (think PDP 11). It is very small and simple, not much code. So, we were able to do what TCP did in user space, handle a bunch of connections at once and not worry about timeouts and filling up kernel capacity. Similar to what was described in this video. We had megabytes of memory at the time.

@@mike123abc I know the history of TCP, I teach it -- I was surprised by the "user space" reimplementation you mentioned. On a modern system you could just tweak the timeout from config.

@@richardclegg8027 30 years ago, the network stack for Windows was... well, there wasn't such a distinction back then between user space and kernel space :) but it was certainly a user-supplied implementation, at least. And there were plenty to pick from.

@@JohnRunyon also depends where you were. In 93 in the UK at least we used other protocols that since died like coloured book. A bit wild west.

To me, it feels like spaghetti programming, using UDP this way. Make a branch here, do also that, here we squeeze in some of that data, there we split it up, here we (try to) collect,... It's not simply unordered TCP.

@@coreC.. - I kind of agree. But adding new protocol types on top of IP is a huge undertaking, and I can easily understand using UDP to develop something before then going through the effort to make it an official protocol on top of IP.

If SCTP deployment was a success QUIC would probably never have existed.

Tons of systems add reliability and features to UDP: bittorrent, gaming, VPNs, etc. are all built on top of UDP.

QUIC has a trick up it's sleeve ! Because everything a middle box would maybe want to look at: is encrypted, because QUIC has encryption build in.

@@autohmae oh yeah, ofc forgot that part. Nice.

What will also happen is that all the home routers with crap UDP implementation will start creating new connection issues for end users. We know these routers exist, we just don't know how widespread the issues are.

@@Ownermode it's by design, they could have left some TCP-like parts unencrypted but the choose to encrypt all of it

But QUIC is "just" UDP. If the middlebox allows UDP it allows QUIC and any UDP based variant.

it's over 9000

Exactly. In company I work for we block QUIC. As far as I know there is no commercially available inline ssl inspection system for http/3.

Middleboxes just usually pass on UDP as "not our business" because it is a minority of traffic. Apart from QUIC I don't know anything that uses TLS (or SSL) and UDP.

@@richardclegg8027 D-TLS is a generic UDP version of TLS which be used by any UDP application to use TLS

@@richardclegg8027 WireGuard

@@Knirin @automahe hah - thought there probably would be some. :)

UDP is great yes. Simple and efficient. For bandwidth - well to fully utilise a link you need to put some form of "find bandwidth" algorithm on top of it and that is going to look like congestion control.

You forgot the main advantage of UDP over TCP: its application interface is actually packet based. TCP sacrifices message boundaries in order to do its magic and you need to either be stream oriented yourself or to implement it all manually (or to use some library for an extra application level protocol on top of TCP but below your own protocol). If you want a serial connection emulation over the network then TCP does exactly what it takes to get it but it makes simple low bandwidth custom TCP-based protocols much harder to design and implement than UDP-baded ones while the simple protocols actually tend to all be packet based. They generally look like "client-(datagram)->server, server-(ack)->client, client(sample_idx++)" so you really need your message boundaries and synchronisation. And that is on top of the fact that TCP itself is much hardet to implement than UDP and it becomes a real pain for low computing power custom embedded network devices which can not run a full on OS so all the pain sometimes just leads to more pain and it is all for the sole reason of not correcting your protocol specs while you still could and leaving the TCP curse there. Also did I mention that multicast and other high bandwith utilization efficiency network level tricks are only possible with UDP and not with TCP while for some apllications it is critical to have some of them? Sometimes TCP is not just wasteful but completely unsuitable for an application.

@@Kirillissimus message boundaries can be useful for sure and QUIC makes full use of that.

@@Kirillissimus but I would say that QUIC is more heavyweight to implement than TCP. You need to implement all TCP mechanisms plus some new ones.

@@richardclegg8027 and TLS is also mandatory

No mic can completely filter bad room reverb... Sadly.

My office is just a bit echoy I am afraid.

yeah but UDP exists anyway

Which is why UDP-based protocols like QUIC have some kind of authentication so they know they can drop the packets. Also things are improving when it comes to BGP routing with slowly more and more deployment of RPKI and ROA for BGP Origin Validation.

These improvements would be application specific, though, there is no need for compatibility between unrelated applications. Why would your app care how my app handles information exchange if neither clients ever connect to the other's host?

Absolutely get ready for improvements. Once this approach is taken you can tune a transport layer for other applications or roll your own QUIC improvements if the server and client both support it.

@@richardclegg8027 This is, pretty much, the HTML/Browser kerfluffle all over again, now on the Transport Protocol, instead of just the Application. Even decades after that whole thing died, we still have to be aware of multiple coexisting solutions for the same thing that need to be coded "just in case the user is on browser X": I still remember cellpadding/cellspacing on HTML previous to 4.0, and let's not forget the newer issues on different JS versions/implementations

@@yoshienverde Oh god. I remember it so well. Lots of browsers implementing different standards. Awful. I think this can avoid that problem. It's only real power users who need to concern themselves. Even if you want to switch it on on your webserver (say) you don't really need to know the protocol version and how it works. A lot of your internet browsing changed to use QUIC and, as a regular user, you didn't notice (or maybe you noticed that thngs ran a little faster). As a sysadmin you notice if and only if you want to turn on QUIC for your webserver. If it did branch to many many versions with different APIs (very different protocols can share an API) people writing web browsers or web servers would have a problem.

cuz it's supposed to follow the clean OSI / TCP/IP layered architecture. For example, your OS is responsible for congestion control in TCP and thusly you don't need to modify each application to change it.

Yeah exactly as VA Demon says. If it was just reliability I would not have a problem. People have done that for decades. QUIC reimplents all TCP mechanisms a layer up. That is what is hackish to me. In order to get a small tweak they implement everything TCP does, now running out of kernel and in userspace to get some tweaks. It is like you wanted to tweak your car engine to be a bit more efficient so you built a second engine in the backseat.

@@richardclegg8027 It's lovely to see you in comments, it happens so rarely with authors ❤️

Separation of layers violation. It means that every application that uses the protocol needs to have a full implementation of the protocol within it. Usually you'd get that from a library, but it's the resource footprint - you get all the memory overhead in every application. It also makes patching any bugs a lot harder, because so much of the code is in each application. In an ideal world, the OS would handle the transport protocol - as it does with TCP - and the application would access it with just a few API calls.

@@richardclegg8027 I wouldn't call it a hack, more like a *blasphemy* ;-)

CPU load is higher. Remember QUIC is doing everything TCP does -- it is maintaining session state -- it's a reimplementation of TCP at the application layer with some extra bits. However, it's implemented in user not kernel space so it's using a bit more resource.

This is why almost everything about QUIC is encrypted, so middleboxes can't interpret it.

@@rich1051414 The maximum size of UDP payload is 65507 Bytes, the MTU of ethernet is usually 1400-ish. If we are talking about minimum reassembly buffer size, its the same for all IPv4 (576 bytes) and doesn't depend on what protocol you are using. Given that TCP uses more techincal info, AND with constant buffer size, it actually leaves less space for payload in that case, so "safe payload" would be less than 508 for TCP. Also, the safe payload by the definition cannot be bigger than the MTU for a single packet, which is 1500-20 for the ethernet (if we don't fragment packets). Using "safe payload size" in this sense is almost meaningless, given that almost every channel these days have 1400+ MTU. Edit: grammar

That is why Google pioneered it. Chrome browser and ownership of prominent websites was the dream "we have the server and the client" scenario

This compression thing is nonsense

@@thewhitefalcon8539 Warframe netcode developers managed to get 82% compression with both payload and congestion control over UDP, making it to 55% of weigth of original UDP packets without compression

QUIC was designed to prevent it, because it already had encryption build it, they encrypt everything so that the middle boxes can't see anything anymore.

@@autohmae The first packet from a client is not encrypted. A smart router can use it in order to distinguish the QUIC packet from other UDP packets and record the IP addresses of the client and the server or just block the connection. Anything else would require a MITM attack with access to a legit CA certificate keys and it would take quite a powerful hardware SSL accelerated router to implement. No other metadata are openly transmitted.

@@Kirillissimus it's true it takes a (few) first packages to establish the encryption, just like previous versions of TLS/SSL. Every packet after that of a QUIC connection is recognizable as a QUIC packet and has a connection ID. One detail is important: that connection ID will be different when multipath/QUIC loadbalancer is involved, so it can't be correlated if it's from the same connection or not. QUIC works despite NAT and NAT can change IP and port numbers, so it's clear that those can be changed, as long as it's changed and restored for all packets in a connection in the same way (for example: change the IP-address of the sender for client to server packet and restore the client IP address of the recipient for the packet from server to client).

@@autohmae I even saw somewhere that QUIC supports connection resuming without a full reestablishment procedure with key exchange and everything. I don't know for sure if it is a part of the latest specification or not but if it is then it will make the job of monitoring such connections even harder since you need to share all the MITM related data with other routers and maybe even other ISPs and you need to store them for who knows how long. P.S. Thanks for the info about connection IDs, I completely forgot about them.

@@KirillissimusYes, QUIC supports 0RTT resume. If you reconnect within a few minutes (not sure of the precise time) you just "connect" with no handshake. You can see it in wireshark if you try. Take a sample to a website you know uses QUIC. Close the webpage and then connect again you will get a "no handshake" connection.

layer 9 and 10 are Budgetary and Political layers, office policies and religion, etc. depending on who you ask their definitions will differ.

Second City Transmission Protocol? 😎

Through this hack, QUIC can do what SCTP tried to do a decade or more before it and more. The biggest deployment of SCTP is probably Telecom and in the browsers as part of WebRTC. But I think SCTP in WebRTC will be replaced by QUIC as well, it might be part of Webtransport though.

@@autohmae AFAIK Netflix does also use it within its CDN backend 👀

@@liminos did not know and I can't find stuff to confirm it, only small hints

You can implement how you schedule UDP in the application if you so choose. Are you sure the limit at your client was not simply the network?

@@richardclegg8027 the limit was more with the device I was having the PC connect to. The problem I ran into was that I had to do a 1ms sleep between each UDP packet that was sent.

@@LaserFur oh dear. You're never going to get any performance with a huge 1ms sleep between packets for sure.

@@richardclegg8027 it's been awhile. maybe windows supports uS sleep intervals now. Windows also removed raw socket support.

While I also like the idea, I think it would just take too long, just look at the IPv6 adoption. Also, a new standard would need to support encryption to really save some unnecessary round-trips. Until the new standard is widely adopted the supported encryption methods may already be outdated again. I'm not sure if it would be possible to define the protocol in a generic way such that future TLS versions would definitely be supported.

No-one running those middle boxes would have a reason to support the new protocol, because there's no-one using it to demand that investment. That's why IPv6 adoption took /decades/ to make any progress at all. Why would any ISP waste money and training time on implementing a protocol when everything works fine without it?

I disagree, all the major players are invested in updating networking.

👍👍!

But isn't that a problem with HTTPS and E2E encryption in general?

Also the speed let us some uncensorship. I'm not saying everything should be uncensored but it's nice to have freedom and trusted links on things.

Hey at-least the new protocol on top of UDP can could fix the TCP issue where malicious packets can close any connection no matter if there is TLS traffic going over TCP.

Or we've sacrificed IDP for privacy 😉

You can just bring in a standard open source TLS1.3 if you are just implementing TCP for fun. Actually it is probably easier as now you are implementing in user space so you don't need to fiddle with the kernel to implement.

You can't, sorry. One of the fundamental aspects of QUIC is a hard requirement for always-on encryption - ie., TLS 1.3 is absolutely necessary. That being said, if you wanted to play around with it just for the fun of it, there's plenty of high-quality libraries that helps you implement the really nasty details of TLS - namely encryption primitives. Look into NaCl/libsodium, for example.

HTTP/3 looks suspiciously like what came before it (HTTP/2) though, nothing really changed, only how the data is transported (a different type of streams, which uses QUIC with TLS/1.3).

@@autohmae Upgraded security with lower latency are two things that are always worthwhile.

The idea behind separating the stack into distinct layers was to allow the layers above to change without any affect on the layers below. So if you are building at the physical layer then you don't care what the network layer looks like, they are just bits. At the network layer you should be able to ignore the transport layer, it's just payload. Shouldn't matter. So new transport layers should be able to be added without any change to the network layer since it is just a change in IP payload. The problem comes when the network layer equipment isn't content treating all payloads the same and they start reaching into the next layer on the stack to try and make optimizations or assumptions to make their job easier. Once they do that then the clear separation of layers is gone and changes to the above layer could cause things to break. For instance, if your router is running QOS it is snooping the transport layer to see what type of traffic it is to decide who to prioritize. If it were to receive a packet with a transport layer it doesn't recognize then it is just going to throw it away (probably assuming it is malformed). If every IP packet was treated identically by every hop in between you and the server, regardless the protocol, then you could throw any sort of new protocol on you wanted and know it would work. That is not what happened though so now we are basically stuck with what we have now or waiting 20+ years for full adoption of a new protocol.

@@zenmaster76 that's insightful thank you.

The bigger problem with this is that, if you're being probabilistic about what data is being piped to someone, and they can afford some parts of it to not be accurate...what do you do when that inaccurate data is executable code?

@@ZT1ST Then you don't use a probabilistic protocol? DVD equals music? Not computer code? What the heck are you talking about.

@@paulpinecone2464 DVDs aren't for computing code? *Stares at PS2 consoles, Xbox 360 consoles, and PC DVD autorun.exe solutions* Someone should tell them that.

IPv6 rise is about 5% per year, it's at 40% now, so 10 years to get to 90%

I don't think IPv6 will get SCTP to pass all middleboxes though. May be wrong.

@@richardclegg8027 It will unless it is manually blocked by a router's firewall somewhere near the endpoints of your connection. Basically connection will still not be guaranteed but the chance of success should get significantly higher.

@@richardclegg8027 You get rid of NAT... (And most firewalls can pass unknown / specific protocol numbers) (NAT needs to understand protocols - it needs to mess with port numbers)

@@GertvandenBerg yes. If you got rid of NAT, loadbalancers, traffic shapers, management boxes and so on the internet would be back closer to its original design spec and you could implement new transport protocols.

QUIC may know more things about the application. TCP will ensure that you get all data. So it asks for missing packets and will wait until it gets them. In a video call this may be a 1s delay between to frames. You will notice this. QUIC may assume that you will notice a missing frame and will not wait for the missing IP packet or ask for a new one. This may not sound like a huge thing for a single user. But if you connection is bad and ask for 5% new packets this means at least 5% more traffic for resent packets. 5% more traffic is not much on the client size. But in a data center with high load this is really expensive. You need more bandwith, more CPU and RAM for the boxes, more cooling, .... The networks cannot handle these 5% extra traffic any longer. They are at 98% of the actual maximum. They have either to reduce the traffic or to sit and wait for new hardware. Brotli compression and HTTP3 will help them a lot. And maybe this will reduce more traffic than new cloud apps and higher resolutions will add.

@@crumblethecookie6118 What a brilliant answer. Thanks!

There is an ack (if needed) it is just handled at the application layer (QUIC) instead of the transport layer

All the TCP header mechanisms are in the UDP packet data. SYN, ACK, FIN etc. It does what TCP does, using similar mechanisms with the header in the UDP data and the end points in the application layer not transport layer.

Thanks. I do like the cufflinks. Lecturer pay does not quite run to a new outfit every video. ;)

TCP is perfectly fine. But you can't really improve on that. So if you know you are in a realm that can grant optimizations while doing some assumptions, QUIC + UTP is the answer.

HTTP 2 added support for multiplexing. Meaning you send multiple datastreams over a single connection. But TCP simply isn't well-suited for that as it does not allow for reordering. Maybe some packets got lost for one data stream but not the other. You're totally out of luck. You have to wait for the earlier data for the other data stream to be re-transmitted.

I think it's because one TCP connection only supports a single stream so doing lots of concurrent operations (e.g. to retrieve the many bits of a website) either requires lots of parallel TCP connections or introduces a lot of latency (if lots of requests are performed sequentially). Lots of TCP connections requires a lot of server resources and doing operations sequentially makes websites take longer to load. QUIC supports multiplexing of streams so a single QUIC connection can support many operations at the same time.

I mention it quite briefly. Lots of small things. Getting the connection open quickly by combining TLS with handshake. Allowing receipt by application of out of order packets. None of these are in standard TCP.

@@richardclegg8027 so normally where we were taught that UDP has the disadvantages over TCP, it said 1. Packet delivery is not ensured, that's on you, the developer to ensure packet was delivered and what to do if it's dropped. 2. Packets may come out of order, that's on you to reorder and define timeout parameters for (1) Based on these then the solution I would make would be a version of sliding window protocol where the receiver has a window size > 1 Is that everything that quic is? A standardised way of doing just that? So you can; instead of writing all that into your apps; just let quic do the work for you? Or is there more things like traffic congestion etc... That it does on top of this? I didn't really understand this because if it does that kind of control, that would mean its aware of a connection or packet? Although in my networks course they didn't touch multicasting and such so I don't know what protocols are used there or how it's generally transmitted. Apparently from above comments it seems quic was targetted mostly for multicasting, some issues for TCP arise so they used UDP. Is it to standardise performance? Add a layer of abstraction to make it easier for Dev's to use UDP? All of the above?

unintentional destiny protocol ?

As usual, blame OpenSSL, if OpenSSL had implemented the changes needed earlier we would have had it in Linux distributions, etc. much earlier.

Of course. Workaround, hack... It is not what the original design intended is my point. You abuse the system and make it work.

@@richardclegg8027 Thank you for answering my comment :) Very good video as well. Its so common to have to do some workaround, very interesting to learn that html3 is basically that!