diff --git a/src/pages/blog/abusing-quic-datagrams.md b/src/pages/blog/abusing-quic-datagrams.md new file mode 100644 index 0000000..65069a0 --- /dev/null +++ b/src/pages/blog/abusing-quic-datagrams.md @@ -0,0 +1,224 @@ +# Abusing QUIC Datagrams +This is the first part of our QUIC hackathon. +Read [Abusing QUIC Streams](/blog/abusing-quic-streams) if you like ordered data like a normal human being. + +We're going to hack QUIC. +"Hack" like a ROM-hack, not "hack" like a prison sentence. +Unless Nintendo is involved. + +We're not trying to be malicious, but rather unlock new functionality while remaining compliant with the specification. +We can do this easily because unlike TCP, QUIC is implemented in *userspace*. +That means we can take a QUIC library, tweak a few lines of code, and unlock new functionality that the greybeards *attempted* to keep from us. +We ship our modified library as part of our application and nobody will suspect a thing. + +But before we continue, a disclaimer: + +## Dingus Territory +QUIC was designed with developers like *you* in mind. +Yes *you*, wearing your favorite "I 💕 node_modules" T-shirt. +I know you're busy, about to rewrite your website (again) using the Next-est framework released literally seconds ago, but hear me out: + +*You are a dingus*. + +The greybeards that designed QUIC, the QUIC libraries, and the corresponding web APIs do not respect you. +They think that given a shotgun, the first thing you're going to do is blow your own foot off. +And they're right of course. + +At first glace, UDP datagrams appear to be quantum: a superposition of delivered and lost. +We hear somebody say "5% packet loss" and our monkey brain visualizes a coin flip or dice roll. +But the reality is that congestion causes (most) packet loss at *our* level in the network stack. +Sending more packets does NOT let you reroll the dice. +Instead it compounds the packet loss, impacting other flows on the network and Google's ability to make money. + +If that was a mind-blowing revelation... that's why there is no UDP API on the web. +Fresh out of a coding bootcamp and you already managed to DDoS an ISP with your poorly written website, huh. +And before you "umh actually" me, WebRTC data channels (SCTP) are congestion controlled and quite flawed, hence why QUIC is a big deal. + +QUIC doesn't change this mentality either. +Short of wasting a zero-day exploit, we have to use the browser's built-in QUIC library via the WebTransport API. +Browser vendors like Google don't want *you*, the `node_modules` enjoyer, doing any of the stuff mentioned in this article on *their* web clients. +But that's not going to stop us from modifying the server or the clients we control (ex. native app). + +However, in doing so, you must constantly evaluate if you are the *dingus* in the exchange. +QUIC infamously does not let you disable encryption because otherwise *some dingus* would disable it because they think it make computer go slow (it doesn't) and oh no now North Korea has some more bitcoins. +Is this a case of the nanny state preventing me from using lead pipes? Yes. +Is AES-GCM slow and worth disabling? Absolutely not, profile your application and you'll find everything else, including sending UDP packets, takes significantly more CPU cycles. + +When the safe API is the best API 99% of the time, then it becomes the only API. +This article is the equivalent of using the `unsafe` keyword in Rust. +If you know what you're doing, then you can make the world a slightly better place (but mostly feel really smart). +But if you mess up, you wasted a ton of time for a worse product. + +So heed my warnings. +Friends don't let friends design UDP protocols. +(And we're friends?) +You should start simple and use the intended QUIC API before reaching for that shotgun. + + +## Proper QUIC Datagrams +I've been quite critical in the past about QUIC datagrams. +Bold statements like "they are bait", "never use datagrams", and "try using QUIC streams first ffs". +But some developers don't want to know the truth and just want their beloved UDP datagrams. + +The problem is that QUIC datagrams are *not* UDP datagrams. +QUIC datagrams: +1. Are congestion controlled. +2. Trigger acknowledgements. +3. Do not expose these acknowledgements. +4. May be batched. + +We're going to try to fix some of these short-comings in the standard by modifying a standard QUIC library. + + +### Congestion Control +The truth is that there's nothing stopping a QUIC library from sending an unlimited number of QUIC datagrams. +There's only a few pixels in the standard that say you SHOULD NOT do this. + +"Congestion Control" is what a library SHOULD do instead. +There's many congestion control algorithms out there, and put simply they are little more than an educated guess if the network can handle more traffic. +The simplest form of congestion control is to send less data when packet loss is high and send more data when packet loss is low. + +But this is an artifical limit that is begging to be broken. +It's often a hiderance as many networks could sustain a higher throughput without this pesky congestion control. +All we need to do is comment out one check and bam, we can send QUIC datagrams at an unlimited rate. + +But please do not do this unless you know what you are doing... or are convinced that you know what you are doing. + +[ percentile meme ] + +You *need* some form of congestion control if you're sending data over the internet. +Otherwise you'll suffer from congestion, bufferbloat, high loss, and other symptoms. +These are not symptoms of the latest disease kept under wraps by the Trump administration, but rather a reality of the internet being a shared resource. +Routers will queue and eventually drop excess packets, wrecking any algorithm that treats the internet like an unlimited pipe. + +But it's no fun starting a blog with back to back lectures. +We're here to abuse QUIC damnit, not the readers. + +But I did not say that you need to use the *default* congestion control. +The QUIC RFC outlines the dated TCP New Reno algorithm which performs poorly when latency is important or bufferbloat rampant. +But that's not set in stone, QUIC expects pluggable congestion control and is descriptive, not prescriptive. +Most libraries expose an interface so you choose the congestion controller or make your own. +You can't do this with TCP as it's buried inside the kernel, so +1 points to Quicendor (good job 'Arry). + +And note that custom congestion control is not specific to QUIC datagrams. +You can use a custom congestion controller for QUIC streams too! +They share the same connection so you can even prioritize/reserve any available bandwidth. + +If the default Reno congestion controller is giving you hives, get that checked out and then give BBR a try. +It works much better in bufferbloat scenarios and powers 99% of HTTP traffic at this point. +Or make your own by piping each ACK to ChatGPT and let the funding roll in. +Or be extra boring and write a master's thesis about curve fitting or something +Or completely disable congestion control altogether, I can't stop you. + +### Acknowledgements +QUIC will reply with an acknowledgement packet after receiving each datagram. +This might sound absolutely bonkers if you're used to UDP. +Why is my unreliable protocol telling me when its unreliable? +The sacrilege! + +Can you take a quess why? +Why go through the trouble of designing an API that looks like UDP only to twist the knife? +These acknowledgements are **not** used for retransmissions... they're only for congestion control. + +But what if we just disabled QUIC's congestion control? +Now we're going to get bombarded with useless acknowledgements! + +The good news is that QUIC acknowledgements are batched, potentially appended to your data packets, and are quite efficient. +It's only a few extra bytes/packets so step 1: get over it. +But I can already feel your angst; your uncontrollable urge to optimize this *wasted bandwidth*. + +The most cleverest of dinguses amongst us (amogus?) might try to leverage these ACKs. +What if we used these otherwise "useless" ACKs to tell our application if a packet was received? +That way we won't have to implement our own ACK/NACK mechanism for reliability. +Somebody call Nobel, that's a dynamite idea. + +You can absolutely hack a QUIC library to expose which datagrams were acknowledged by the remote. +More information is always better, right? + +...unfortunately there's an edge case [discovered by yours truely](https://github.com/quicwg/datagram/issues/15). +The QUIC may acknowledge a datagram but it gets dropped before being delivered it to the application. +This will happen if a QUIC library processes a packet, sends an ACK, but the application (ex. Javascript web page) is too slow and we run out of memory before processing it. +**Note:** QUIC streams don't suffer from this issue because they use flow control. + +This might not be a deal breaker depending on your application because it introduces false-positives. +They can't be treated as a definitive signal that a packet was processed. +If you need this reassurance then switch to QUIC streams or (*gasp*) implement your own ACKs/NACKs on top of QUIC datagrams. + +But not only is it more work to implement your own ACKs, the underlying QUIC ACKs will still occur. +This is gross because one datagram will trigger a QUIC ACK, your custom ACK, and a QUIC ACK for your custom ACK. +I bet the angst is overwhelming now. + +So let's get rid of these useless ACKs instead. +If you control the receiver, you can tweak the `max_ack_delay` parameter. +This is a parameter exchanged during the handshake that indicates how long the implementation can wait before sending an acknowledgement. +Crank it up to 1000ms (the default is 25ms) and the number of acknowledgement packets should slow to a trickle. + +Be warned that this will impact all QUIC frames, *especially* STREAM retransmissions. +It may also throw a wrench into the congestion controller too as they expect timely feedback. +So only consider this route if you've gone *full dingus* and completely disabled congestion control and streams. +The chaos you've sown will be legendary. + +### Batching +Most QUIC libraries will automatically fill a UDP packet with as much data as it can. +This is dope, but as we established, you can't settle for *dope*. +You don't get out of bed in the morning for *dope*, it needs to be at least *rad* or 🚀. + +But why disable batching? +Let's you're high on the thrill of sending unlimited packets after disabling congestion control. +However, sometimes a bunch of packets get lost and you need to figure out why. +Surely it can't be the consequences of your actions? + +"No! +It's the network's fault! +I'm going to send additional copies to ensure at least one arrives..." + +I cringed a bit writing that. +Not as much as you've cringed while reading this blog, but a close 🥈. +See, I've sat through too presentations by principal engineers claiming the same thing. +It turns out there's no secret cheat code to the internet: sending more packets will cause proportially *more* packet loss as devices get fully saturated. + +FEC is the solution to a problem, but a different problem. +I already wrote Never* Use Datagrams and you should read that. +Instead, we're going to focus on **atomicity**. + +Like I said earlier, packet loss instinctively feels like an independent event: a coin toss on a router somewhere. +But sending the same packet back-to-back does *not* mean you get a second flip of the coin. + +An IP packet is actually quite a high level abstraction. +Our payload of data has to somehow get serialized into a physical transmission and that's the job of a lower level protocol. +For example, 7 IP packets (1.2KB MTU*) can fit snug into a jumbo Ethernet frame. +These frames then get sliced into different dimensions, be it time or frequency or whatever, as they traverse an underlying medium. +A protocol like Wifi will automatically apply redundancy and even retransmissions based on the properties of the medium. +And let's not forget intermediate routers because they will batch packets too, it's just more efficient. + +So if your protocol depends on "independent" packets, then you will be distraught to learn that no such thing exists. +Packets can (and will) be dropped in batches despite your best efforts to avoid batching. + +That's why QUIC goes the other direction and batches everything, including datagrams. +An application may appear to send ten disjoint datagrams but under the hood, they may get secretly combined into one UDP datagram to avoid redundant headers. +If not QUIC, then another layer would perform (less efficient) batching. + +The ratio of lectures to hacks is approaching dangerous levels. +Fuck it, lets disable batching. + +If you control the QUIC library, one snip and you can short-circuit the batching. +Each QUIC datagram is now a UDP packet, hazzah! +The library should still perform *some* batching and append stuff like ACKs to packets. +Please have mercy and don't require separate UDP packets for our ill-fated ACK friends. + +But even if you don't control the QUIC library (ex. browser), you can abuse the fact that QUIC cannot split a datagram across multiple packets. +If your datagrams are large enough (>600 bytes*) then you can sleep easy knowing they won't get combined. +...unless the QUIC library supports MTU discovery, because while the minimum MTU is 1.2KB, the maximum is 64KB. + +I'm not sure why you would disable batching because it can only worsen performance, but I'm here (to pretend) not to judge. +Your brain used to be smooth but now it's wrinkly af 🧠🔥. + +## Conclusion +I know you just want your precious UDP datagrams but they're kept in a locked drawer lest you hurt yourself. +But I've given you the key and it's your turn to prove me right. + +If you want to "hack" QUIC for more constructive purposes, check out my next blog about QUIC streams. +There's actually some changes you could make without incurring self-harm. + +## Hack the Library +https://github.com/quinn-rs/quinn/blob/6bfd24861e65649a7b00a9a8345273fe1d853a90/quinn-proto/src/frame.rs#L211 diff --git a/src/pages/blog/abusing-quic-streams.md b/src/pages/blog/abusing-quic-streams.md new file mode 100644 index 0000000..22a7259 --- /dev/null +++ b/src/pages/blog/abusing-quic-streams.md @@ -0,0 +1,299 @@ +# Abusing QUIC Streams +This is the second part of our QUIC hackathon. +Read [Abusing QUIC Datagrams](/blog/abusing-quic-datagrams) if byte streams confuse you. + +Look, I may be one of the biggest QUIC fanboys on the planet. +I'm ashamed to admit that QUIC streams are meh for real-time latency. + +I know, I know, I just spent the last blog post chastising you, the `I <3 node_modules` developer, for daring to dream. +For daring to send individual IP packets without a higher level abstraction. +For daring to vibe code. + +Fret not because we can "fix" QUIC streams with some clever library tweaks. +It's more work and more wasted bytes but that's basically our job as programmers, right? +A small price to pay. + + +## QUIC 101 +QUIC streams trickle. + +Just like TCP, all of the data written to a QUIC steam will eventually arrive (unless cancelled). +If there's packet loss, retransmissions will *eventually* patch any holes. +If there's poor network conditions, congestion control will slow down the send rate until it *eventually* recovers. +If the receiver is CPU starved, flow control will pause transmissions until they *eventually* recover. + +Like the DMV, QUIC steams are little more than queues. +You write data to the end while the QUIC library transmits packet-sized chunks from the front. +In the business we call this a FIFO. + +But what happens when you have royal data that must arrive ASAP? +If we write to the end of a stream, then it might get queued behind peasant data that is blocked for whatever reason. +This called "head-of-line" blocking and it has plagued TCP since its inception. + +Enter Robespierre. +QUIC is a huge advancement over TCP because it offers off-with-the-head-of-line blocking. +Instead of appending to an existing stream, we can open a new stream and (optionally) reset the existing stream. +Our royal stream can be marked highest priority while the peasant stream can be sent to the guillotine. + +Yes I know the royals were actually guillotined first so the French Revolution wasn't the best analogy. +But look eventually every Frenchman with a history got the choppy choppy so you can treat it as a FILO queue. +But unlike the French Revolution, there's no cost for creating or cancelling a QUIC stream. + +So if you don't care about ordering, make a new QUIC stream for each message. +If newer messages are more important, then deprioritize or cancel older streams. +This sounds easy, so what's the problem? + +Come with me, little one. +We're going on an adventure. + + +## Detecting Loss +``` +| |i +|| |_ +``` + +QUIC streams are continuous byte streams that rely on retransmissions to *eventually* patch any holes caused by packet loss. +I keep putting *eventually* in *italics* and now it's finally time to explain why. + +QUIC was primarily designed for HTTP/3 and bulk data transfers. +The average transfer speed matters the most when you're downloading `pron.zip`. +In order to achieve that, QUIC and TCP won't waste bandwidth on retransmitting the same data again unless it's absolutely needed. + +**Pop quiz:** +*How does a QUIC library know when a packet is lost and needs to be retransmitted?* + +**Answer**: +Trick question, it doesn't. + +A pop quiz this early into a blog post? +AND it's a trick question? +That's not fair. + +There's no explicit signal from routers when a packet is lost. +L4S might change that on some networks but I wouldn't get your hopes up. +Instead, a QUIC library has to instead use FACTS and LOGIC to make an educated guess. +The RFC outlines a *recommended* algorithm that I'll attempt to simplify: + +- The sender increments a sequence number for each packet. +- Upon receiving a packet, the receiver will start a timer. Once this timer expires, it will ACK that sequence number and any others that arrive in the meantime. +- If the sender does not receive an ACK after waiting multiple RTTs, either the original packet or the ACK probably got lost. +- The sender will poke the receiver by sending another packet (potentially a 1-byte PING) to have them send another ACK. +- Eventually the poke works and the sender receives an ACK indicating which packets were received. +- **FINALLY** the sender *may* decide that a packet was lost if: + - 3 newer sequences were ACKed. + - or a multiple of the RTT has elapsed. +- As the congestion controller allows, retransmit any lost packets and repeat. + +Skipped that boring, "simplified" wall of text? +I don't blame you. +You're just here for the funny blog. + +I'll help. +What this means is that if a packet is lost, it takes anywhere from 1-3 RTTs to detect the loss and retransmit. +If you're sending a lot of data, then it's closer to 1RTT because new packets indirectly trigger ACKs for lost packets +But if you're sending a tiiiiny amount of data, and for the last packet in a burst, then it takes closer to 3RTT to recover. + +And just in case I lost you in the acronym soup, RTT is just another way of saying "your ping". +So if you're playing Counter Strike cross-continent with a ping of 150ms, you're already at a disadvantage. +Throw QUIC into the mix and some packets will take 300ms to 450ms. +*cyka bylat* + + +## Delta Encoding +We're not done with retransmissions yet, but let's put an `!Unpin` in it. + +If you're sending time series data over the Internet, there's a good chance that it can benefit from delta encoding. +The most common example is, *checks notes*, video encoding. +Wow that's + +Video encoding works by creating a base image called a "keyframe" (or I-frame) that is not too dissimilar from a PNG or JPEG. +Doing this for every frame requires a lot of data which is why animated "GIFs" used to look so ass. +Video encoding instead abuses the fact that most frames of a video are very similar and primarily encodes frames as deltas of previous (and sometimes future!) frames. + +Delta encoding significantly lowers the bitrate because it removes redundancies. +However it introduces dependencies, as packets now depend on previous packets otherwise the data is corrupt and causes trippy effects. +Some encodings are self-healing like Opus (audio), while other encodings have to start over with a new base (video). + +Streams (TCP, QUIC, Unix, etc) are great for delta encoding because they guarantee data arrives and in order. +The application can reference a previous byte range without worrying about pesky holes. +It's like a new pair of underwear, you can just put it on. + +But new underwear is not the most efficient. +If you're in a hurry, you grab whatever you can find and hope the holes aren't in unfortunate places. +There's no time to order new underwear; your rock it. +The user experience will suffer but Sonic gotta go fast. + +If you're rich, you can buy extra pairs to make your underwear redundant. +Maybe you get unlucky and grab a holey pair, but there's a fresh pair stapled to it just for such an unfortunate occasion. +Yeah it's more expensive because of tariffs but Scrooge gotta McDuck. + +QUIC streams take the slow approach. +If QUIC finds a hole in the metaphorical underwear, it will order a hole-sized patch and sow it on. +It gives you a good experience while wasting the least amount of pricy cotton. + +But we're smarter than *default behavior*. +We can modify a QUIC sender so there's some urgency. +The next shipment of underwear comes with a bag of patches for the previous shipment of underwear. + + +QUIC streams are absolutely built for delta encoding as they deliver data reliably and in order. +There's no holes in this pristine data stream. + +However, this causes a problem for real-time latency, as packets become dependent on each other. + +Let's suppose we want to stream real-time chat over QUIC. +But we're super latency sensitive, like it's a bad rash, and need the latest sentence as soon as possible. +We can't settle for random words; we need the full thing in order baby. +The itch is absolutely unbearable and we're okay being a little bit wasteful. + +If the broadcaster types "hello" followed shortly by "world", we have a few options. +Pop quiz, which approach is subjectively the best: + +Option A: Create a stream, write the word "hello", then later write "world". +Option B: Create a stream and write "0hello". Later, create another stream and write "5world". The number at the start is the offset. +Option C: Create a stream and write "hello". Later, create another stream and write "helloworld". +Option D: Abuse QUIC (no spoilers) + +If you answered D then you're correct. +Let's use a red pen and explain why the other students are failing the exam. +No cushy software engineering gig for you. + +### Option A +*Create a stream, write the word "hello", then later write "world".* + +This is classic head-of-line blocking. If the packet containing "hello" gets lost over the network, then we can't actually use the "world" message if it arrives first. +But that's okay in this scenario because of my arbitrary rules + +The real problem is that when the "hello" packet is lost, it won't arrive for *at least* an RTT after "world" because of the affirmationed retransmission logic. +That's no good. + +### Option B +*Create a stream and write "0hello". Later, create another stream and write "5world". The number at the start is the offset.* + +I didn't explain how multiple streams work because a bad teacher blames their students. +And I wanted to blame you. + +QUIC streams share a connection but are otherwise independent. +You can create as many streams *as the remote peer allows* with no\* overhead. +In fact, if you were considering retransmitting QUIC datagrams, you could totally use a QUIC stream per datagram instead. + +In this example, both "hello" and "world" will be (re)transmitted independently over separate streams and it's up to the receiver to reassemble them. +That's why we had to include the offset, otherwise the receiver would have no idea that "world" comes after "hello" (duh). +At some point we would also need to include a "sentence ID" if we wanted to support multiple sentences. +The receiver receives "helloworld" and voila, our itch is scratched. + +But this approach sucks. +Major suckage. + +But don't feel bad if this seemed like a good idea. +You literally just reimplemented QUIC streams and this is identical to **Option A**. +We did all of this extra work for nothing. + +Despite the fact that they're using separate streams, "hello" still won't be retransmitted until after "world" is acknowledged. +The fundamental problem is that QUIC retransmissions occur at the *packet level*. +We need to dive deeper, not just create an independent stream. + +### Option C +*Create a stream and write "hello". Later, create another stream and write "helloworld".* + +Now we're getting somewhere. +We're finally wasting bytes. + +This approach is better than Option A/B (for real-time latency) because we removed a dependency. +If "hello" is lost, well it doesn't matter because "helloworld" contains a redundant copy. + +But this approach is still not ideal. +What if "hello" is acknowledged *before* we write "world"? +We don't want to waste bytes unncessarily and shouldn't retransmit "hello". +Most QUIC libraries don't expose these details to the application. +And even if they did, we would have to implement **Option B** and send "5world". +And what if there's a gap, like "world" is acknowledged but not "hello" or a trailing "!"? + +We're delving back into *reimplementing QUIC streams* territory. +The wheel has been invented already. +If only there was a way to hack a QUIC library to do what we want... + +### Option D +*Abuse QUIC (no spoilers)* + +A QUIC stream is broken into STREAM frames that consist of an offset and payload. +The QUIC sender keeps track of which STREAM frames were stuffed into which UDP packets so it knows what to retransmit if a packet is lost. +The QUIC receiver reassembles these STREAM frames based on the offset then flushes it to the application. + +The magic trick depends on an important fact: +A QUIC receiver must be prepared to accept duplicate, overlapping, or otherwise redundant STREAM frames. + +See, there's no requirement that the *same* STREAM frame is retransmitted. +If STREAM 10-20 is lost, we could retransmit it as STREAM 10-15, STREAM 17-20, and STREAM 15-17 if we wanted to. +This is on purpose and super useful because we can cram a UDP packet full of miscallenous STREAM frames without worrying about overrunning the MTU. + +Grab your favorite sleeveless shirt because we are *abusive*. + +We're going to use a single stream like **Option A**. +Normally, "hello" is sent as STREAM 0-5 and "world" is sent as STREAM 5-10. +However, we can modify our QUIC library to actually transmit STREAM 0-10 instead, effectively sending "helloworld" in one packet. +More generally, we can retransmit any unacknowledged fragments of a stream. + +The easiest way to implement this is to have the QUIC library *assume* the in-flight fragments of a stream are lost and need to be retransmitted. +This won't impact congestion control because we don't consider the packets as lost... just some of their contents. +For you library maintainers out there, consider adding this as a `stream.retransmit()` method and feel free to forge my username into the git commit. + +## Revisiting Retransmissions +Remember the part where I said: + +> We're not done with packet loss yet, but let's put an `!Unpin` in it. + +We're back baby. +That's because as covered in the previous section, it's totally legal to retransmit a stream chunk without waiting for an acknowledgement. +The specification says don't do it, but there's nothing *actually* stopping us flooding the network with duplicate copies. +If QUIC receives a duplicate stream chunk, it will silently ignore it. + +So rather than wait 450ms for a (worst-case) acknowledgement, what if we just... don't? +We could just transmit the same stream chunk again, and again, and again, and again, and again every 50ms. + +But there's a pretty major issue with this approach: +**BUFFERBLOAT**. +Surprise! +It turns out that some routers may queue packets for an undisclosed amount of time when overloaded. + +Let's say you retransmit every 50ms and everything works great on your PC. +A user from Brazil or India downloads your application and it initially works great too. +But eventually their ISP gets overwhelmed and congestion causes the RTT to (temporarily) increase to 500ms. +...well now you're transmitting 10x the data, potentially aggravating any congestion and preventing recovery. +It's a vicious loop and you've basically built your own DDoS agent. + +For the distributed engineers amogus, this is the networking equivalent of an F5 refresh storm. +Blind retransmissions are the easiest way to join the UDP Wall of Shame. + + +### Congestion Control to the Rescue +Actually I just lied. + +This infinite loop of pain, suffering, and bloat is what would happen if you retransmitted using UDP datagrams. +But not with QUIC streams (and datagrams). + +Retransmissions are gated by congestion conntrol. +Even when a packet is considered lost, or my hypothetical `stream.retransmit()` is called, a QUIC library won't immediately retransmit. +Instead, stream retransmissions are queued up until the congestion controller allows more packets to be sent. + +The QUIC greybeards will stop you from doing bad thing. +The children yearn for the mines, but the adults yearn for child protection laws. + +But now we have a new problem. +Our precious data is getting queued locally and latency starts to climb. +If we do nothing, then nothing gets dropped and oh no, we just reimplemented TCP. + +At some point we have to take the L and wipe the buffer clean. +QUIC lets you do this by resetting a stream, notifying the receiver and cancelling any queued (re)transmissions. +We can then make a new stream (for free*) and start over with a new base. +For those more media inclined, this would mean resetting the current GoP and encoding a new I-frame on a new stream. + +To recap: +- Using UDP directly: our data gets queued and arbitrarily dropped by some router in the void. +- Using QUIC datagrams, our data get dropped locally (congestion control) and arbitrarily dropped by the void, although less often. +- Using QUIC streams, our data gets queued locally (congestion control) and explicitly dropped when we choose. + +I can't believe there aren't more QUIC stream fanboys. +It's a great abstraction because *you do not understand networking* nor should your application care how stuff gets split into IP packets. +Your application should deal with queues that get drained at an unpredictable rate. diff --git a/src/pages/blog/async-warts.md b/src/pages/blog/async-warts.md new file mode 100644 index 0000000..127236b --- /dev/null +++ b/src/pages/blog/async-warts.md @@ -0,0 +1,19 @@ +# Async Warts +An idea for a blog post about Rust async. + +## Cancel Safe +https://github.com/kixelated/moq-rs/blob/9707899bc13212e42b4bccfbe5d0522b2e18b57d/moq-transfork/src/model/group.rs#L136 + +## 'static and Arc> + +## Wtf is Pin + +## Send + Sync +And async traits + +## Cleanup is Cool + +## Runtime Agnostic is Cool + +## No locks across await is Cool +*but buggy diff --git a/src/pages/blog/using-quic-streams.md b/src/pages/blog/using-quic-streams.md new file mode 100644 index 0000000..bf56ebe --- /dev/null +++ b/src/pages/blog/using-quic-streams.md @@ -0,0 +1,165 @@ +# Using QUIC Streams +I was actually inspired to write this blog post because someone joined my (dope) Discord server. +They asked if they could do all of the above so they would have proper QUIC datagrams. + +The use-game is vidya games. +The most common approach for video games is to process game state at a constant "tick" rate. +VALORANT, for example, uses a tick rate of [128 Hz](https://playvalorant.com/en-us/news/dev/how-we-got-to-the-best-performing-valorant-servers-since-launch/) meaning each update covers a 7.8ms period. +It's really not too difficult from frame rate (my vidya background) but latency is more crucial otherwise nerds get mad. + +So the idea is to transmit each game tick as a QUIC datagram. +However, that would involve transmitting a lot of redundant information, as two game ticks may be very similar to each other. +So the idea is to implement custom acknowledgements and (re)transmit only the unacknowledged deltas. + +If you've had the pleasure of implementing QUIC before, this might sound very similar to how QUIC streams work internally. +In fact, this line of thinking is what lead me to ditch RTP over QUIC (datagrams) and embrace Media over QUIC (streams). +So why disassemble QUIC only to reassemble parts of it? +If we used QUIC streams instead, what more could you want? + +### What More Could We Want? +Look, I may be one of the biggest QUIC fanboys, but I've got to admit that QUIC is pretty poor for real-time latency. +It's not designed for small payloads that need to arrive ASAP, like voice calls. + +But don't take my wrinkle brain statements as fact. +Let's dive deeper. + +*How does a QUIC library know when a packet is lost?* + +It doesn't. +There's no explicit signal from routers (yet?) when a packet is lost. +A QUIC library has to instead use FACTS and LOGIC to make an educated guess. +The RFC outlines a *recommended* algorithm that I'll attempt to simplify: + +- The sender increments a sequence number for each packet. +- Upon receiving a packet, the receiver will start a timer to ACK that sequence number, batching with any others that arrive within `max_ack_delay`. +- If the sender does not receive an ACK after waiting multiple RTTs, it will send another packet (like a PING) to poke the receiver and hopefully start the ACK timer. +- After finally receiving an ACK, the sender *may* decide that a packet was lost if: + - 3 newer sequences were ACKed. + - or a multiple of the RTT has elapsed. +- As the congestion controller allows, retransmit any lost packets and repeat. + +Skipped that boring wall of text? +I don't blame you. +You're just here for the funny blog and *maaaaybe* learn something along the way. + +I'll help. +If a packet is lost, it takes anywhere from 1-3 RTTs to detect the loss and retransmit. +It's particularly bad for the last few packets in a burst because if they're lost, nothing starts the acknowledgement timer and the sender will have to poke. +"You still alive over there?" + +And just in case I lost you in the acronym soup, RTT is just another way of saying "your ping". +So if you're playing Counter Strike cross-continent with a ping of 150ms, you're already at a disadvantage. +Throw QUIC into the mix and some packets will take 300ms to 450ms of conservative retransmissions. +*cyka bylat* + +### We Can Do Better? +So how can we make QUIC better support real-time applications that can't wait multiple round trips? +Should we give up and admit that networking peaked with UDP? + +Of course not what a dumb rhetorical question. + +We can use QUIC streams! +A QUIC stream is nothing more than a byte slice. +The stream is arbitrarily split into STREAM frames that consists of an offset and a payload. +The QUIC reviever reassembles these frames in order before flushing to the application. +Some QUIC libraries even allow the application to read stream chunks out of order. + +How do we use QUIC streams for vidya games? +Let's suppose we start with a base game state of 1000 bytes and each tick there's an update of 100 bytes. +We make a new stream, serialize the base game state, and constantly append each update. +QUIC will ensure that the update and deltas arrive in the intended order so it's super easy to parse. + +But not so fast, there's a **huge** issue. +We just implemented head-of-line blocking and our protocol is suddenly no better than TCP! +I was promised that QUIC was supposed to fix this... + + + +We can abuse the fact that a QUIC receiver must be prepared to accept duplicate or redundant STREAM frames. +This can happen naturally if a packet is lost or arrives out of order. +You might see where this is going: nothing can stop us from sending a boatload of packets. + + + + +Our QUIC library does not need to wait for a (negative) acknowledgement before retransmitting a stream chunk. +We could just send it again, and again, and again every 50ms. +If it's a duplicate, then QUIC will silently ignore it. + +## A Problem + +But there's a pretty major issue with this approach: +**BUFFERBLOAT**. +Surprise! +It turns out that some routers may queue packets for an undisclosed amount of time when overloaded. + +Let's say you retransmit every 50ms and everything works great on your PC. +A user from Brazil or India downloads your application and it initially works great too. +But eventually their ISP gets overwhelmed and congestion causes the RTT to (temporarily) increase to 500ms. +...well now you're transmitting 10x the data, potentially aggravating any congestion and preventing recovery. +It's a vicious loop and you've basically built your own DDoS agent. + +For the distributed engineers amogus, this is the networking equivalent of an F5 refresh storm. + + +Either way, sending redundant copies of data is nothing new. +Let's go a step further and embrace QUIC streams. + +### How I Learned to Embrace the Stream + + +At the end of the day, a QUIC STREAM frame is a byte offset and payload. +Let's say we transmit our game state as STREAM 0-230 and 33ms later we transmit 20 bytes of deltas as STREAM 230-250. +If the original STREAM frame is lost, well even if we receive those 20 bytes, we can't actually decode them and suffer from HEAD-OF-LINE blocking. + +My game dev friend thinks this is unacceptable and made his own ACK-based algorithm on top of QUIC datagrams instead. +The sender ticks every 30ms and sends a delta from the last acknowledged state, even if that data might be in-flight already. +Pretty cool right? +Why doesn't QUIC do this? + +It does. + +(mind blown) + +QUIC will retransmit any unacknowledged fragments of a stream. +But like I said above, only when a packet is considered lost. +But with the power of h4cks, we could have the QUIC library *assume* the rest of the stream is lost and needs to be retransmitted. +For you library maintainers out there, consider adding this as a `stream.retransmit()` method and feel free to forge my username into the git commit. + +So to continue our example above, we can modify QUIC to send byte offsets 0-250 instead of just 230-250. +And now we can accomplish the exact* same behavior as the game dev but without custom acknowledgements, retransmissions, deltas, and reassembly buffers. + +Forking a library feels *so dirty* but it magically works. + + +### Some Caveats +Okay it's not the same as the game dev solution; it's actually better because of **congestion control**. +And once again, you do ~need~ want congestion control. + + + +Let's say you retransmit every 30ms and everything works great on your PC. +A user from Brazil or India downloads your application and it initially works great too. +But eventually their ISP gets overwhelmed and congestion causes the RTT to (temporarily) increase to 500ms. +...well now you're transmitting 15x the data and further aggravating any congestion. +It's a vicious loop and you've basically built your own DDoS agent. + +But QUIC can avoid this issue because retransmissions are gated by congestion control. +Even when a packet is considered lost, or my hypothetical `stream.retransmit()` is called, a QUIC library won't immediately retransmit. +Instead, retransmissions are queued up until the congestion controller deems it appropriate. +Note that a late acknowledgement or stream reset will cancel a queued retransmission (unless your QUIC library sucks). + +Why? +If the network is fully saturated, you need to send fewer packets to drain any network queues, not more. +Even ignoring bufferbloat, networks are finite resources and blind retransmissions are the easiest way to join the UDP Wall of Shame. +In this instance,.the QUIC greybeards will stop you from doing bad thing. +The children yearn for the mines, but the adults yearn for child protection laws. + +Under extreme congestion, or when temporarily offline, the backlog of queued data will keep growing and growing. +Once the size of queued delta updates grows larger than the size of a new snapshot, cut your losses and start over. +Reset the stream with deltas to prevent new transmissions and create a new stream with the snapshot. +Repeat as needed; it's that easy! + +I know this horse has already been beaten, battered, and deep fried, but this is yet another benefit of congestion control. +Packets are queued locally so they can be cancelled instantaneously. +Otherwise they would be queued on some intermediate router (ex. for 500ms).