diff --git a/docs/architecture/.nav.yml b/docs/architecture/.nav.yml index baa69923d..c30743fd5 100644 --- a/docs/architecture/.nav.yml +++ b/docs/architecture/.nav.yml @@ -2,3 +2,4 @@ flatten_single_child_sections: true nav: - "Introduction": index.md - "Sizing": sizing.md + - "Latency": latency.md diff --git a/docs/architecture/latency.md b/docs/architecture/latency.md new file mode 100644 index 000000000..96b5ac3fa --- /dev/null +++ b/docs/architecture/latency.md @@ -0,0 +1,293 @@ +# Latency + +The DSS keeps its data across all instances of a DSS pool using a single +distributed database cluster (e.g. CockroachDB, Yugabyte). + +Because the cluster is distributed and strongly consistent, the physical +distance between nodes directly affects how fast the DSS can answer. Latency is +therefore not a detail: it drives the responsiveness and the throughput of the +whole service. + +## Impact on performance + +### Time to answer a request + +When a call is made to the DSS, the core service queries the database. The +database is not a single node: it is a cluster spread across DSS participants. +To stay strongly consistent, a write must reach a quorum of nodes before it is +acknowledged, and a read may also need to contact other nodes depending on where +the data lives. + +Each of these internal hops adds to the round-trip latency between nodes. A single +API call can trigger more than one database operation, so the latencies stack +up. If the nodes are far apart, every consensus round pays that distance, and +the time to answer grows accordingly. + +![](../assets/generated/latency_tta.png) +/// caption +A generic request. Each +arrow adds cumulative latency +/// + +### Bandwidth + +Latency and bandwidth interact. With high latency, each operation (assuming no parallelism) takes longer +to complete, so fewer operations finish per second, which lowers effective +throughput even when raw bandwidth is available. Synchronization traffic between +nodes also competes for that bandwidth. + +This matters for high-volume cases such as many subscriptions or large query +results: the more data must be transferred and synchronized between distant +nodes, the more latency limits how much can realistically be moved in a given +time window. There is a practical ceiling on how much can be transferred before +responsiveness degrades. + +![](../assets/latency_impact.png) +/// caption +**Simulated** throughput vs +latency on a virtual link of a generic TCP connexion captured with [iperf3](https://software.es.net/iperf/), practical throughput is even lower. +/// + +### Example of latency vs performance + +The following graph shows, in a controlled environment, the impact of latency on +simple requests. + +Tests have been done with DSS version v0.22.0, CockroachDB, 3 USS with one node, +on a virtual machine with ample resources. + +Latency is injected with +`tc qdisc add dev eth0 root netem delay Xms 2ms distribution paretonormal` ([documentation](https://man7.org/linux/man-pages/man8/tc-netem.8.html)), with +X ranging from 0 to 50, in steps of 5ms. No loss is applied, nor latency between +a DSS and its datastore. + +Performance is measured by creating and deleting RID ISAs, without any +subscriptions, as it performs simple queries and doesn't create congestion +issues. + +Queries are done against all DSS, with 3 processes in parallel (for a total +of 9) and at the same time to remove as many variations as possible. Notice +there is still some expected variance, the goal here is to show global trends. + +![](../assets/latency_rid.png) +/// caption +Results of testing RID ISA calls with +various latencies. +/// + +This is just an example of one call, but it shows that even with a simple +operation, there is a incompressible limit on how fast queries can be processed. + +## Examples + +This part showcases the effect of latency on a sample deployment to +illustrate the effects of latency. + +!!! warning + Notice that values here are only a snapshot from when they were run. DSS code + has improved and will improve in the future, cloud providers' network resources + will evolve with time, and even the placement of your virtual machine could + randomly impact latency and performance. + + Results have to be interpreted relative to each other in various situations, + not as the performance to be expected from the DSS. + + The performance baseline will be described in another document. + +Locust instances are always located next to the DSS being tested, on the same +machine. The test ran is: + +```docker run -e AUTH_SPEC="DummyOAuth(http://172.17.0.1:8085/token,localhost)" -p 8089:8089 -v .:/app/ interuss/monitoring-dev uv run locust -f loadtest/locust_files/ISA.py -H http://172.17.0.1 -u 10 --uss-base-url http://dss.localututm```. + +The test has been chosen to be light and to be able to run it across a wide +number of configurations, with the possibility of adding one subscription to +force 'synchronization' between nodes. Others, more complex tests (like +FlightsInSubs which create one implicit subscription for each call) generate too +many contentions with version v0.22.0 to be meaningful. + +### Constraints: identifying the latency impact with a stable environment + +These examples measure the impact of **latency alone**. Every environment +parameter is held constant across the three scenarios; the only variable is +geographic distance, and therefore inter-node latency. Any difference in the +results is attributable to that latency as much as +practically possible. + +Held constant: + +* **DSS version**: v0.22.0 in every run. +* **Topology**: one node per location, three nodes across two providers. +* **Machine resources**: ample CPU and memory; compute is never the bottleneck. +* **CockroachDB encryption**: off. +* **Client placement**: Locust runs on the same machine as the DSS under test, + so client↔DSS latency is negligible. Only *inter-node* latency varies. +* **Load profile**: the same light test, fixed at 10 users (`-u 10`), with no + parallelization: each user sends one request, waits for the full response, + then sends the next. +* **Datastore connections**: left to default value (4). + +The serial load is what makes latency directly visible in throughput: each +user's rate is capped by the round-trip time, so once latency dominates, total +throughput falls as roughly `users / latency`. Read the q/s columns as a readout +of latency, not of DSS capacity. + +### US-West + +Cluster has been deployed on the west coast, in two providers. Regions between +providers are close, and the two machines in the same provider are in the same +region, but different availability zone. + +Latency measured by CockroachDB is as follows: + +![](../assets/generated/latency_west.png) + +Results after 2 minutes are: + +| Node | q/s | 50th latency | 95th latency | +| ---- | ----- | ------------ | ------------ | +| N1 | 19.35 | 2ms | 11ms | +| N2 | 19.01 | 3ms | 14ms | +| N3 | 18.2 | 17ms | 97ms | + +showing good performances in general. + +Notice primary CockroachDB node is probably located in 'Provider 1'. Other test +runs showed swapped latency, but that is to be expected. + +With one subscription added in the database, forcing a synchronized update: + +| Node | q/s | 50th latency | 95th latency | Failures/s | +| ---- | ----- | ------------ | ------------ | ---------- | +| N1 | 19.1 | 2ms | 17ms | 0 | +| N2 | 18.81 | 3ms | 14ms | 0 | +| N3 | 17.53 | 16ms | 120ms | 0 | + +While this shows a slight decrease in performance (approximately 2 percent for +q/s and 50th latency), the extra, synchronized step needed is almost invisible: +with nodes this close, the synchronization round-trip costs only a few +milliseconds. + +### Across US + +Cluster has been deployed across the US, in two providers: one node in the west, +one node in the east and, in a different provider, in the center. + +Latency measured by CockroachDB is as follows: + +![](../assets/generated/latency_coast_to_coast.png) + +Results after 2 minutes are: + +| Node | q/s | 50th latency | 95th latency | +| ---- | ----- | ------------ | ------------ | +| N1 | 17.42 | 2ms | 120ms | +| N2 | 17.08 | 64ms | 350ms | +| N3 | 17.95 | 35ms | 170ms | + +That's a 7% loss in performances, 700% increased latency for the 50th percentile +and more than 1000% of increase for the 95% percentile. Without a synchronized +step, latency is still low enough that the client cadence, not latency, sets the +rate - hence the small q/s loss. + +With one subscription added in the database, forcing a synchronized update: + +| Node | q/s | 50th latency | 95th latency | Failures/s | +| ---- | ---- | ------------ | ------------ | ---------- | +| N1 | 17.6 | 2ms | 92ms | 0 | +| N2 | 4.2 | 2200ms | 2500ms | 0 | +| N3 | 12.3 | 54ms | 100ms | 0 | + +That's a 40% loss in performances, 24000% increased latency for the 50th +percentile and more than 6000% of increase for the 95% percentile, compared to +the same step in previous scenario. + +The impact is way more visible there - and it is pure latency, not capacity: on +N2 the synchronized round-trip costs ~2200ms, and since each user must wait it +out before its next request, q/s falls to ~4 by simple arithmetic. The +CockroachDB leader (near N1) pays no such round-trip and keeps performing well. + +### Across Atlantic Ocean + +Cluster has been deployed across the US and in Europe, in two providers: one +node in the west, one in Europe and, in a different provider, in the center. + +Latency measured by CockroachDB is as follows: + +![](../assets/generated/latency_europe.png) + +Results after 2 minutes are: + +| Node | q/s | 50th latency | 95th latency | +| ---- | ---- | ------------ | ------------ | +| N1 | 18.4 | 2ms | 180ms | +| N2 | 14.2 | 150ms | 630ms | +| N3 | 18.4 | 34ms | 240ms | + +That's a 9% loss in performances, 1650% increased latency for the 50th +percentile and more than 2000% of increase for the 95% percentile. + +With one subscription added in the database, forcing a synchronized update: + +| Node | q/s | 50th latency | 95th latency | Failures/s | +| ---- | ---- | ------------ | ------------ | ---------- | +| N1 | 10.4 | 570ms | 4600ms | 0.15 | +| N2 | 1.85 | 10000ms | 10000ms | 0.5 | +| N3 | 5.45 | 1200ms | 10000ms | 0.18 | + +That's a 69% loss in performances, 122000% increased latency for the 50th +percentile and more than 35000% of increase for the 95% percentile, compared to +the same step in the first scenario. Compared to the non-synchronized case, we +lost about 63% of queries per second. + +Here the cross-Atlantic synchronization round-trip approaches the request +timeout (10000ms): each serial user spends nearly the whole timeout waiting on +quorum, so throughput collapses and requests start to fail - the only scenario +with failures. This is the latency of synchronizing across an ocean, not a +limitation of the DSS: even the leader region (N1) is slowed purely by the cost +of reaching quorum. At this distance, synchronized operations are latency-bound +to the point of timing out, especially away from the leader. + +### Summary + +![](../assets/latency_summary.png) + +The DSS runs on a 3-node CockroachDB cluster, tested across 3 geographic setups +with **all environment parameters held constant** - version, topology, machine +resources, encryption, client placement, and the serial 10-user load. The only +variable is geographic distance, hence inter-node latency (8.9ms -> 38.9ms -> +94ms): every difference in the results is attributable to it, and under the +serial load, q/s is a direct readout of latency, not of DSS capacity. + +The data shows: + +* Without a synchronized step, latency stays below the client cadence, so q/s + drops only ~7% (across US) to ~9% (Atlantic). +* With a synchronized step (one subscription), each request pays a consensus + round-trip that grows with distance: invisible in US-West, ~750ms and −40% + q/s coast-to-coast, and near the 10000ms timeout across the Atlantic - q/s + down to ~6 (−69%), with failures (up to 0.5/s) appearing only there. + +The conclusion is about latency, not the DSS. In no run is the DSS or database +saturated; they are waiting on quorum round-trips whose duration is set by +distance - ultimately by the speed of light - and no DSS or query optimization +can remove that. + +Finally, this test was done on endpoints performing relatively simple queries. +More complex ones, like SCD, involve more synchronized steps per operation and +are therefore even more sensitive to inter-node latency. + +## Conclusion + +To minimize the round-trip cost of consensus and synchronization, keep nodes as +close as possible, ideally within the same region. + +At the same time, you should probably keep distinct areas, datacenters and +providers for redundancy. A set of servers in a single rack will be very fast, +but have very low redundancy, for example if the rack loses power. On the +opposite side of the options, servers spread throughout the world will offer +maximum redundancy, for example against country-level issues, but will be very +slow. The goal is a balance: geographically close enough for low latency, +diverse enough for redundancy. + +Think and plan for the failures you want to handle, as a pool, and then spread +nodes across failure domains accordingly for resilience. diff --git a/docs/assets/generated/latency_coast_to_coast.png b/docs/assets/generated/latency_coast_to_coast.png new file mode 100644 index 000000000..651b460cc Binary files /dev/null and b/docs/assets/generated/latency_coast_to_coast.png differ diff --git a/docs/assets/generated/latency_europe.png b/docs/assets/generated/latency_europe.png new file mode 100644 index 000000000..fc46d61fb Binary files /dev/null and b/docs/assets/generated/latency_europe.png differ diff --git a/docs/assets/generated/latency_tta.png b/docs/assets/generated/latency_tta.png new file mode 100644 index 000000000..54a4336c0 Binary files /dev/null and b/docs/assets/generated/latency_tta.png differ diff --git a/docs/assets/generated/latency_west.png b/docs/assets/generated/latency_west.png new file mode 100644 index 000000000..405c0348a Binary files /dev/null and b/docs/assets/generated/latency_west.png differ diff --git a/docs/assets/latency_coast_to_coast.gv b/docs/assets/latency_coast_to_coast.gv new file mode 100644 index 000000000..bda7efa8e --- /dev/null +++ b/docs/assets/latency_coast_to_coast.gv @@ -0,0 +1,14 @@ +digraph time_to_answer { + rankdir=LR; + node [shape=circle, colorscheme=paired8, width=2.0, fixedsize=true]; + graph [dpi = 300]; + + N1 [label="N1\nProvider 1\nRegion W",color=1] + N2 [label="N2\nProvider 1\nRegion E",color=2] + N3 [label="N3\nProvider 2\nRegion C",color=3] + + N1 -> N2 [label="~58.8ms",dir=both] + N1 -> N3 [label="~31.1ms",dir=both] + N2 -> N3 [label="~26.7ms",dir=both] + +} diff --git a/docs/assets/latency_europe.gv b/docs/assets/latency_europe.gv new file mode 100644 index 000000000..6e50e8f1b --- /dev/null +++ b/docs/assets/latency_europe.gv @@ -0,0 +1,14 @@ +digraph time_to_answer { + rankdir=LR; + node [shape=circle, colorscheme=paired8, width=2.0, fixedsize=true]; + graph [dpi = 300]; + + N1 [label="N1\nProvider 1\nRegion W",color=1] + N2 [label="N2\nProvider 1\nRegion E",color=2] + N3 [label="N3\nProvider 2\nRegion C",color=3] + + N1 -> N2 [label="~141.9ms",dir=both] + N1 -> N3 [label="~31.2ms",dir=both] + N2 -> N3 [label="~109ms",dir=both] + +} diff --git a/docs/assets/latency_impact.png b/docs/assets/latency_impact.png new file mode 100644 index 000000000..90c7328f2 Binary files /dev/null and b/docs/assets/latency_impact.png differ diff --git a/docs/assets/latency_rid.png b/docs/assets/latency_rid.png new file mode 100644 index 000000000..82fcce2db Binary files /dev/null and b/docs/assets/latency_rid.png differ diff --git a/docs/assets/latency_summary.png b/docs/assets/latency_summary.png new file mode 100644 index 000000000..505fb7aac Binary files /dev/null and b/docs/assets/latency_summary.png differ diff --git a/docs/assets/latency_tta.gv b/docs/assets/latency_tta.gv new file mode 100644 index 000000000..00a0ed2e7 --- /dev/null +++ b/docs/assets/latency_tta.gv @@ -0,0 +1,28 @@ +digraph time_to_answer { + rankdir=LR; + node [shape=box, colorscheme=paired8]; + graph [dpi = 300]; + + client [label="Client"]; + core [label="DSS"]; + + subgraph cluster_db { + label="Distributed DB cluster"; + style=dashed; + leader [label="Node\n(leaseholder)"]; + peer1 [label="Peer node"]; + peer2 [label="Peer node"]; + } + + client -> core [label="API call"]; + core -> leader [label="query"]; + + leader -> peer1 [label="consensus\nround-trip"]; + leader -> peer2 [label="consensus\nround-trip"]; + peer1 -> leader [label="ack"]; + peer2 -> leader [label="ack (quorum)"]; + + leader -> core [label="result"]; + core -> client [label="response"]; + +} diff --git a/docs/assets/latency_west.gv b/docs/assets/latency_west.gv new file mode 100644 index 000000000..0aa919c6f --- /dev/null +++ b/docs/assets/latency_west.gv @@ -0,0 +1,14 @@ +digraph time_to_answer { + rankdir=LR; + node [shape=circle, colorscheme=paired8, width=2.0, fixedsize=true]; + graph [dpi = 300]; + + N1 [label="N1\nProvider 1\nRegion 1\nZone 1",color=1] + N2 [label="N2\nProvider 1\nRegion 1\nZone 2",color=2] + N3 [label="N3\nProvider 2\nRegion 1",color=3] + + N1 -> N2 [label="~0.78ms",dir=both] + N1 -> N3 [label="~13.4ms",dir=both] + N2 -> N3 [label="~12.6ms",dir=both] + +} diff --git a/docs/deployment_checklist.md b/docs/deployment_checklist.md index b781cd136..03f13714d 100644 --- a/docs/deployment_checklist.md +++ b/docs/deployment_checklist.md @@ -10,6 +10,7 @@ This checklist outlines the major decisions and steps required to deploy a non-l * This repository provides Terraform configurations for [Amazon Web Services (EKS)](infrastructure/aws.md) and [Google Cloud (GKE)](infrastructure/google.md) to deploy a Kubernetes cluster (the infrastructure into which the Services will be deployed). * This repository provides [Tanka](https://github.com/interuss/dss/blob/master/deploy/services/tanka/) files and [Helm Charts](https://github.com/interuss/dss/blob/master/deploy/services/helm-charts/dss) to be used to deploy Services into a Kubernetes cluster. Terraform will automatically generate these configurations if needed. * You may also choose to deploy manually or use custom configuration tools. + * Latency is an important factor to achieve good performances. Review the [latency documentation](architecture/latency.md) and plan with others participants of your DSS pool. * [ ] Prepare sufficient resources for the services. * In particular, review the [CockroachDB recommendations](https://www.cockroachlabs.com/docs/v24.1/recommended-production-settings#cloud-specific-recommendations) and [YugabyteDB recommendations](https://docs.yugabyte.com/stable/deploy/checklist/#public-clouds); the datastore will consume the majority of the resources. * Example sizing is also describled in [sizing](architecture/sizing.md). diff --git a/mkdocs.yml b/mkdocs.yml index a044de754..e45deba43 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -104,5 +104,8 @@ markdown_extensions: - attr_list # Enable creation of custom anchors + - pymdownx.blocks.caption + + extra_javascript: - assets/checkbox.js