-
Notifications
You must be signed in to change notification settings - Fork 78
Add async backing technical reference #1179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: staging/product-ia
Are you sure you want to change the base?
Changes from all commits
8a1b7e3
30e9e3d
f21afb1
5bc9f0f
e7e40dc
9ddd44f
353d84a
3f7c4ad
9c21f6b
a8561ce
214af0d
864b3bf
426bf7a
1ee9875
1aed941
86f6a2f
317b284
ce479bc
aca7036
bc7f4f1
85df8ce
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,271 @@ | ||
| TODO | ||
| --- | ||
| title: Asynchronous Backing | ||
| description: Understand how asynchronous backing pipelines parachain block production, the protocol changes it introduces on the Relay Chain, and how parachains participate safely and efficiently. | ||
| categories: Polkadot Protocol | ||
| --- | ||
|
|
||
| # Asynchronous Backing | ||
|
|
||
| ## Introduction | ||
|
|
||
| Asynchronous backing often shortened to _Async Backing_ is a parachain protocol feature that significantly improves performance, enabling parachains to produce blocks twice as fast (every 6 seconds instead of every 12) and to provide 4x more execution time per block (2 seconds instead of 0.5). | ||
|
|
||
| Technically, async backing is a parachain [configuration](https://paritytech.github.io/polkadot-sdk/master/cumulus_primitives_core/relay_chain/struct.AsyncBackingParams.html){target=\_blank} that allows collators and validators to build blocks ahead of the relay chain during the generation and backing stages of the [Inclusion Pipeline](/reference/parachains/consensus/inclusion-pipeline){target=\_blank} by using unincluded segments, which are chains of parachain blocks that have not yet been fully included in the relay chain. This decouples parachain block production from relay chain inclusion, improves coretime efficiency, and enables the parallel processing required for parachains to further scale throughput using [Elastic Scaling](/reference/parachains/consensus/elastic-scaling){target=\_blank}. | ||
|
|
||
| ## Configurations | ||
| The following configurations can be set by on-chain governance, dictating how many blocks ahead of the relay chain a given parachain's collators can run: | ||
|
|
||
| - [**`max_candidate_depth`**](https://paritytech.github.io/polkadot-sdk/master/cumulus_primitives_core/relay_chain/struct.AsyncBackingParams.html#structfield.max_candidate_depth){target=\_blank}: the number of parablocks a collator can produce that are not yet included in the relay chain. A value of `2` means that there can be a maximum of 3 unincluded parablocks at any given time. | ||
| - [**`allowed_ancestry_len`**](https://paritytech.github.io/polkadot-sdk/master/cumulus_primitives_core/relay_chain/struct.AsyncBackingParams.html#structfield.allowed_ancestry_len){target=\_blank}: the oldest relay parent a parablock can be built on top of. A value of `1` means collators can start building blocks 6 seconds in advance. | ||
|
|
||
| ## Synchronous VS. Asynchronous Processing | ||
|
|
||
| <!-- In the synchronous scenario, both the collators and validators draw context from the relay parent of the prior parablock, which lives on the relay chain. This makes the Backing and Generation steps tightly coupled to the prior parablock completing the inclusion pipeline. As a result, one parablock can be processed *every other* relay block, and only `0.5` seconds are assigned for execution. --> | ||
|
|
||
| The Polkadot-parachain protocol originally operated in synchronous mode, where both collators and validators drew context exclusively from the relay parent of the prior parablock, which lives on the relay chain. This made the Backing and Generation steps tightly coupled to the prior parablock completing the entire inclusion pipeline. As a result, one parablock could only be processed every other relay block, with just 0.5 seconds assigned for execution. | ||
|
|
||
| ```mermaid | ||
| --- | ||
| displayMode: compact | ||
| config: | ||
| themeCSS: " | ||
| #item1 { fill: #450693; stroke: #450693; } \n | ||
| #item2 { fill: #8C00FF; stroke: #8C00FF; } \n | ||
| #item3 { fill: #FFC400; stroke: #FFC400; } \n | ||
| #r { fill: #eb4172; stroke:none; font-size: 20px; } \n | ||
| svg text { font-size: 20px !important; } \n | ||
| svg .sectionTitle { font-size: 20px !important; } \n #p1padTop { display: none; } \n | ||
|
|
||
| /* Hide ALL task labels by default */ | ||
| text.taskText, | ||
| text.taskTextOutside, | ||
| [class*='taskText'] tspan { display: none !important; } \n | ||
|
|
||
| /* Show labels for the 'r' group (inside or outside, incl. tspans) */ | ||
| text.taskText[id^='r'], | ||
| text.taskTextOutside[id^='r'], | ||
| text[id^='r'] tspan { display: inline !important; font-size: 20px; color: #000 !important; } \n | ||
|
|
||
| /* Keep section titles styled */ | ||
| .sectionTitle { fill: #000 !important; font-weight: 700; font-size: 18px; } \n | ||
|
|
||
| /* Hide the first two section titles (F1, F2). Change indexes if needed. */ | ||
| .sectionTitle:nth-of-type(1), | ||
| .sectionTitle:nth-of-type(2) { display: none !important; } \n | ||
|
|
||
| /* Also hide SPACING row labels on the left */ | ||
| text.taskTextOutside[id^='p1padTop'] { display: none !important; } \n | ||
| " | ||
| themeVariables: | ||
| sectionBkgColor: '#fff' | ||
| gantt: | ||
| numberSectionStyles: 1 | ||
| barHeight: 70 | ||
| gridLineStartPadding: 100 | ||
| --- | ||
| %%{init: {"gantt": {"barHeight": 70 }}}%% | ||
| gantt | ||
| dateFormat YYYY | ||
| axisFormat %y | ||
| tickInterval '10year' | ||
|
|
||
| section F1 | ||
| R1 : r, 1905, 1907 | ||
| R2 : r, 1911, 1913 | ||
| R3 : r, 1917, 1919 | ||
| R4 : r, 1923, 1925 | ||
|
|
||
charlesHetterich marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| section F2 | ||
| SPACING : p1padTop, 1901, 1924 | ||
|
|
||
| section P1 | ||
| X : item1, 1900, 1901 | ||
| Backing : item2, 1901, 1906 | ||
| Inclusion : item3, 1906, 1912 | ||
|
|
||
| section P2 | ||
| X : item1, 1912, 1913 | ||
| Backing : item2, 1913, 1918 | ||
| Inclusion : item3, 1918, 1924 | ||
|
|
||
|
|
||
| ``` | ||
|
|
||
| <!-- In the asynchronous scenario, where both the collators and validators have access to [Unincluded Segments](/reference/parachains/consensus/inclusion-pipeline) as an additional context source, the Backing and Generation steps are no longer coupled to the prior block completing the full inclusion pipeline. Instead, the prior parablock only needs to complete the generation step and be added to the Unincluded Segments before the next parablock can begin the Backing and Generation steps. | ||
|
|
||
| This results in one parablock being processed *every* relay block, and allows for `2` seconds of execution. --> | ||
|
|
||
| The modern protocol now uses asynchronous backing, where both collators and validators have access to [unincluded segments](/reference/parachains/consensus/inclusion-pipeline){target=\_blank} as an additional context source. The Backing and Generation steps are no longer coupled to the prior block completing the full inclusion pipeline. Instead, the prior parablock only needs to complete the generation step and be added to the Unincluded Segments before the next parablock can begin the Backing and Generation steps. | ||
|
|
||
| This results in one parablock being processed every relay block (instead of every other relay block), and allows for more time to execute during the Generation step (0.5s β 2s). | ||
|
|
||
| ```mermaid | ||
| --- | ||
| displayMode: compact | ||
| config: | ||
| themeCSS: " | ||
| #item1 { fill: #450693; stroke: #450693; } \n | ||
| #item2 { fill: #8C00FF; stroke: #8C00FF; } \n | ||
| #item3 { fill: #FFC400; stroke: #FFC400; } \n | ||
| #r { fill: #eb4172; stroke:none; font-size: 20px; } \n | ||
| svg text { font-size: 20px !important; } \n | ||
| svg .sectionTitle { font-size: 20px !important; } \n #p1padTop { display: none; } \n | ||
|
|
||
| /* Hide ALL task labels by default */ | ||
| text.taskText, | ||
| text.taskTextOutside, | ||
| [class*='taskText'] tspan { display: none !important; } \n | ||
|
|
||
| /* Show labels for the 'r' group (inside or outside, incl. tspans) */ | ||
| text.taskText[id^='r'], | ||
| text.taskTextOutside[id^='r'], | ||
| text[id^='r'] tspan { display: inline !important; font-size: 20px; color: #000 !important; } \n | ||
|
|
||
| /* Keep section titles styled */ | ||
| .sectionTitle { fill: #000 !important; font-weight: 700; font-size: 18px; } \n | ||
|
|
||
| /* Hide the first two section titles (F1, F2). Change indexes if needed. */ | ||
| .sectionTitle:nth-of-type(1), | ||
| .sectionTitle:nth-of-type(2) { display: none !important; } \n | ||
|
|
||
| /* Also hide SPACING row labels on the left */ | ||
| text.taskTextOutside[id^='p1padTop'] { display: none !important; } \n | ||
| " | ||
| themeVariables: | ||
| sectionBkgColor: '#fff' | ||
| gantt: | ||
| numberSectionStyles: 1 | ||
| barHeight: 70 | ||
| gridLineStartPadding: 100 | ||
| --- | ||
| %%{init: {"gantt": {"barHeight": 70 }}}%% | ||
| gantt | ||
| dateFormat YYYY | ||
| axisFormat %y | ||
| tickInterval '10year' | ||
|
|
||
| section F1 | ||
| R1 : r, 1905, 1907 | ||
| R2 : r, 1911, 1913 | ||
| R3 : r, 1917, 1919 | ||
| R4 : r, 1923, 1925 | ||
| R5 : r, 1929, 1931 | ||
|
|
||
| section F2 | ||
| SPACING : p1padTop, 1901, 1930 | ||
|
|
||
| section P1 | ||
| X : item1, 1900, 1902 | ||
| Backing : item2, 1902, 1912 | ||
| Inclusion : item3, 1912, 1918 | ||
|
|
||
| section P2 | ||
| X : item1, 1906, 1908 | ||
| Backing : item2, 1908, 1918 | ||
| Inclusion : item3, 1918, 1924 | ||
|
|
||
| section P3 | ||
| X : item1, 1912, 1914 | ||
| Backing : item2, 1914, 1924f | ||
| Inclusion : item3, 1924, 1930 | ||
|
|
||
| section P4 | ||
| X : item1, 1918, 1920 | ||
| Backing : item2, 1920, 1930 | ||
| ``` | ||
|
|
||
| <!-- In the multi-core scenario, by using elastic scaling we can further increase the the number of parablocks processed for each relay block. By using two cores, here we double the number of parablocks being processed in a 30 second period: | ||
|
|
||
| ```mermaid | ||
| --- | ||
| displayMode: compact | ||
| config: | ||
| themeCSS: " | ||
| #item1 { fill: #450693; stroke: #450693; } \n | ||
| #item2 { fill: #8C00FF; stroke: #8C00FF; } \n | ||
| #item3 { fill: #FFC400; stroke: #FFC400; } \n | ||
| #r { fill: #eb4172; stroke: none; } \n | ||
| #p1padTop { display: none; } \n | ||
| text.taskText[id^=p1padTop] { fill: none !important; color: #000 !important; } \n | ||
|
|
||
|
|
||
| /* Default inside task text: white */ \n | ||
| .taskText { fill: #fff !important; color: #fff !important; font-weight: 700; font-size: 18px; } \n | ||
|
|
||
| /* Hide text inside bars */ | ||
| .taskText { display: none !important; } \n | ||
|
|
||
| /* Outside labels and section titles: black */ \n | ||
| .taskTextOutside, .sectionTitle { fill: #000 !important; color: #000 !important; font-weight: 700; font-size: 18px; } \n | ||
|
|
||
| /* Inside text for #r items: black */ \n | ||
| text.taskText[id^=r] { fill: #fff !important; } \n | ||
| " | ||
| themeVariables: | ||
| sectionBkgColor: '#fff' | ||
| gantt: | ||
| numberSectionStyles: 1 | ||
| barHeight: 70 | ||
| gridLineStartPadding: 100 | ||
| --- | ||
| %%{init: {"gantt": {"barHeight": 70 }}}%% | ||
| gantt | ||
| dateFormat YYYY | ||
| axisFormat %y | ||
| %% this next line doesn't recognise 'decade' or 'year', but will silently ignore | ||
| tickInterval '10year' | ||
|
|
||
| section F1 | ||
| R1 : r, 1905, 1907 | ||
| R2 : r, 1911, 1913 | ||
| R3 : r, 1917, 1919 | ||
| R4 : r, 1923, 1925 | ||
| R5 : r, 1929, 1931 | ||
|
|
||
| section F2 | ||
| SPACING : p1padTop, 1901, 1930 | ||
|
|
||
| section P1 | ||
| X : item1, 1900, 1902 | ||
| Backing : item2, 1902, 1912 | ||
| Inclusion : item3, 1912, 1918 | ||
|
|
||
| section P2 | ||
| X : item1, 1906, 1908 | ||
| Backing : item2, 1908, 1918 | ||
| Inclusion : item3, 1918, 1924 | ||
|
|
||
| section P3 | ||
| X : item1, 1912, 1914 | ||
| Backing : item2, 1914, 1924 | ||
| Inclusion : item3, 1924, 1930 | ||
|
|
||
| section F20 | ||
| SPACING : p1padTop, 1901, 1930 | ||
|
|
||
| section F21 | ||
| R1 : r, 1905, 1907 | ||
| R2 : r, 1911, 1913 | ||
| R3 : r, 1917, 1919 | ||
| R4 : r, 1923, 1925 | ||
| R5 : r, 1929, 1931 | ||
|
|
||
| section F22 | ||
| SPACING : p1padTop, 1901, 1930 | ||
|
|
||
|
|
||
| section P4 | ||
| X : item1, 1900, 1902 | ||
| Backing : item2, 1902, 1912 | ||
| Inclusion : item3, 1912, 1918 | ||
|
|
||
| section P5 | ||
| X : item1, 1906, 1908 | ||
| Backing : item2, 1908, 1918 | ||
| Inclusion : item3, 1918, 1924 | ||
|
|
||
| section P6 | ||
| X : item1, 1912, 1914 | ||
| Backing : item2, 1914, 1924 | ||
| Inclusion : item3, 1924, 1930 | ||
| ``` --> | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| --- | ||
| title: Inclusion Pipeline | ||
| description: Learn how Polkadot validates parachain blocks through the Inclusion Pipeline, detailing context, generation, backing, and inclusion stages. | ||
| categories: Polkadot Protocol | ||
| --- | ||
|
|
||
| # Inclusion Pipeline | ||
|
|
||
| ## Introduction | ||
|
|
||
| The inclusion pipeline is the multi-stage process through which every parachain block (parablock) is validated and secured before being finalized in the Polkadot relay chain. This pipeline ensures that all parachain blocks meet validity requirements through progressive verification by multiple sets of validators. | ||
|
|
||
| The pipeline exists to provide Polkadot's security guarantees, rather than relying on a single validator group. Each parablock passes through multiple validation stages with different validator sets, ensuring that invalid blocks cannot be finalized even if some validators are malicious or compromised. | ||
|
|
||
| Whether a parachain uses synchronous or [asynchronous backing](/reference/parachains/consensus/async-backing){target=\_blank}, all parablocks follow the same inclusion pipeline. The difference is in the timing: asynchronous backing allows multiple blocks to be at different stages of the pipeline simultaneously (pipelining), while synchronous backing processes one block through the entire pipeline before starting the next. | ||
|
|
||
| ## Pipeline Stages | ||
|
|
||
| The inclusion pipeline consists of three main stages: | ||
|
|
||
| ```mermaid | ||
| %%{init: {"flowchart": {"nodeSpacing": 40, "rankSpacing": 60}}}%% | ||
| flowchart LR | ||
| %% Keep the pipeline on one row (container is hidden) | ||
| subgraph Row[" "] | ||
| direction LR | ||
| G["Generation"] --> B["Backing"] --> I["Inclusion"] | ||
| end | ||
| style Row fill:none,stroke:none | ||
|
|
||
| %% Context: plain text (no box) pointing to both G and B | ||
| C["Context"]:::nobox | ||
| C -.-> G | ||
| C -.-> B | ||
|
|
||
| classDef nobox fill:none,stroke:none,color:inherit; | ||
| ``` | ||
| ### Context | ||
|
|
||
| To build a parablock during the generation and backing stages, collators and validators require access to the state context of the parachain. This context is derived from two sources: | ||
|
|
||
| - **Relay Parent**: The relay chain block to which the parablock is anchored. Note that the relay parent of a parablock is always different from the relay chain block that eventually includes it. This context source resides on the relay chain. | ||
|
|
||
| - **Unincluded Segments**: Chains of candidate parablocks that have not yet been included in the relay chain. These segments represent sequences of block ancestors and may contain candidates at any stage pre-inclusion. A key feature enabled by [async backing](/reference/parachains/consensus/async-backing){target=\_blank} is that collators can build new parablocks on top of these unincluded ancestors rather than being limited to ancestors already included in the relay chain state. This context source resides on the collators. | ||
|
|
||
| ### Generation | ||
|
|
||
| Collators execute their blockchain core functionality to generate a new block, producing a [proof-of-validity](https://paritytech.github.io/polkadot-sdk/book/types/availability.html?#proof-of-validity){target=\_blank} (PoV), which is passed to validators selected for backing. The PoV is composed of: | ||
|
|
||
| - A list of state transitions called the **block candidate** | ||
| - The values in the parachain's database that the block modifies | ||
| - The hashes of the unaffected points in the Merkle tree | ||
|
|
||
|
|
||
| ### Backing | ||
|
|
||
| A subset of active validators verify that the parablock follows the state transition rules of the parachain and sign a [validity statement](https://paritytech.github.io/polkadot-sdk/book/types/backing.html?#validity-attestation){target=\_blank} about the PoV which can have a positive or negative outcome. With enough positive statements (at least 2/3 of assigned validators), the candidate is considered backable. It is then noted in a fork on the relay chain, at which point it is considered backed, ready for the next stage of the pipeline. | ||
|
|
||
| ### Inclusion | ||
|
|
||
| Validators gossip [erasure code chunks](https://paritytech.github.io/polkadot-sdk/book/types/availability.html#erasure-chunk){target=\_blank} and put the parablock through the final [approval process](https://paritytech.github.io/polkadot-sdk/book/protocol-approval.html){target=\_blank} before it is considered *included* in the relay chain. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove all these commented lines.