Substrate Developer Hub

Substrate Developer Hub

  • Tutorials
  • Knowledge Base
  • Recipes
  • API Reference
  • Languages iconEnglish
    • 简体中文
    • Help Translate

›Advanced

Getting Started

  • Overview
  • Architecture
  • Installation
  • Getting Started on Windows
  • Glossary

Learn Substrate

  • Extrinsics
  • Transaction Pool
  • Account Abstractions
  • Session Keys
  • Transaction Weight
  • Off-Chain Features

Runtime

  • Runtime Overview
  • Runtime Primitives
  • FRAME
  • Pallets
  • Runtime Macros
  • Runtime Metadata
  • Runtime Storage
  • Runtime Origin
  • Runtime Execution
  • Runtime Events
  • Runtime Errors
  • Transaction Fees
  • Debugging
  • Runtime Tests
  • Runtime Benchmarking
  • On-Chain Randomness
  • Runtime Upgrades

Smart Contracts

  • Overview
  • ink! Smart Contracts
  • ink! Concepts
  • ink! Development
  • Contracts Pallet
  • EVM Pallet
  • ink! F.A.Q.

Integrate

  • Polkadot-JS
  • Client Libraries
  • Chain Specification
  • The Subkey Tool
  • Memory Profiling

Advanced

  • SCALE Codec
  • Consensus
  • The Block Import Pipeline
  • Runtime Executor
  • Cryptography
  • Storage
  • SS58 Address Format
  • Why are there no Hash collections in sp_std?

Contribute

  • Help Translate
Edit

The Block Import Pipeline

The Import Queue

The import queue is an abstract worker queue present in every Substrate node. It is not part of the runtime. The import queue is responsible for processing pieces of incoming information, verifying them, and if they are valid, importing them into the node's state. The most fundamental piece of information that the import queue processes is blocks themselves, but it is also responsible for importing consensus-related messages such as justifications and, in light clients, finality proofs.

The import queue collects incoming elements from the network and stores them in a pool. The elements are later checked for validity and discarded if they are not valid. Elements that are valid are then imported into the node's local state.

The import queue is codified abstractly in Substrate by means of the ImportQueue trait. The use of a trait allows each consensus engine to provide its own specialized implementation of the import queue, which may take advantage of optimization opportunities such as verifying multiple blocks in parallel as they come in across the network.

The import queue also provides some hooks via the Link trait that can be used to follow its progress.

The Basic Queue

Substrate provides a default in-memory implementation of the ImportQueue known as the BasicQueue. The BasicQueue does not do any kind of optimization, rather it performs the verification and import steps sequentially. It does, however, abstract the notion of verification through the use of the Verifier trait.

Any consensus engine that relies on the BasicQueue must implement the Verifier trait. The Verifier is typically responsible for tasks such as checking inherent data, and ensuring that the block is signed by the appropriate authority.

The Block Import Trait

When the import queue is ready to import a block, it passes the block in question to a method provided by the BlockImport trait. This BlockImport trait provides the behavior of importing a block into the node's local state database.

One implementor of the BlockImport trait that is used in every Substrate node is the Client, which contains the node's entire block database. When a block is imported into the client, it is added to the main database of blocks that the node knows about.

The Block Import Pipeline

In the simplest cases, blocks are imported directly into the client. But most consensus engines will need to perform additional verification on incoming blocks, update their own local auxiliary databases, or both. To allow consensus engines this opportunity, it is common to wrap the client in another struct that also implements BlockImport. This nesting leads to the term "block import pipeline".

An example of this wrapping is the PowBlockImport, which holds a reference to another type that also implements BlockImport. This allows the PoW consensus engine to do its own import-related bookkeeping and then pass the block to the nested BlockImport, probably the client. This pattern is also demonstrated in AuraBlockImport, BabeBlockImport, and GrandpaBlockImport.

BlockImport nesting need not be limited to one level. In fact, it is common for nodes that use both an authoring engine and a finality gadget to layer the nesting even more deeply. For example, Polkadot's block import pipeline consists of a BabeBlockImport, which wraps a GrandpaBlockImport, which wraps the Client.

Learn More

Several of the Recipes' nodes demonstrate the block import pipeline:

  • Basic PoW - the import pipeline includes PoW and the client
  • Hybrid Consensus - the import pipeline is PoW, then Grandpa, then the client
Last updated on 2/24/2021 by Dan Forbes
← ConsensusRuntime Executor →
  • The Import Queue
  • The Basic Queue
  • The Block Import Trait
  • The Block Import Pipeline
  • Learn More
Substrate Developer Hub
Developer Hub
TutorialsKnowledge BaseRecipesAPI Reference
Community
Community HomeNewsletterSubstrate Technical ChatSubstrate SeminarStack OverflowTwitterEvents
More
Substrate Builders ProgramBlogSubstrate GitHubDeveloper Hub GitHubPrivacy PolicyTerms of UseCookie Settings
Copyright © 2021 Parity Technologies