Substrate Recipes 🍴😋🍴

Substrate Recipes is a collection of simple code patterns that demonstrate best practices when building blockchains with Substrate. The repo used to build this book is open source. Check out the contributions guidelines for an overview of the structure and directions for getting involved.

The current scope is limited to pallet development and runtime configuration. To learn more about Substrate, see the official documentation.

What is Substrate?

Substrate is a framework for building blockchains. For a high level overview, read the following blog posts:

How to Use This Book

Start by cloning the repo on github:

git clone https://github.com/substrate-developer-hub/recipes

As you read through the book, practice compiling and testing recipes in recipes/kitchen. You can't learn to code by reading about it -- play with the code in the kitchen, extract patterns, and apply them to a problem that you want to solve!

It is useful to recognize that coding is all about abstraction. To accelerate your progress, I recommend skimming the patterns in this book, composing them into interesting projects, and building your own recipes.

Reach out for guidance on Stack Overflow or in the Substrate Technical Riot channel.

Learn Rust

To be productive with substrate requires some familiarity with Rust. Fortunately, the Rust community is known for comprehensive documentation and tutorials. The most common resource for initially learning Rust is The Rust Book. To see examples of popular crate usage patterns, Rust by Example is also convenient.

API Design

To become more familiar with commmon design patterns in Rust, the following links might be helpful:

Optimizations

To optimize runtime performance, Substrate developers should make use of iterators, traits, and Rust's other "zero cost abstractions":

It is not (immediately) necessary to become familiar with multithreading because the runtime operates in a single-threaded context. Even so, the runtime might take advantage of the offchain workers API to minimize the computation executed on-chain. Effectively using these features requires increased familiarity with advanced Rust.

For a high-level overview of concurrency in Rust, Stjepan Glavina provides the following descriptions in Lock-free Rust: Crossbeam in 2019:

  • Rayon splits your data into distinct pieces, gives each piece to a thread to do some kind of computation on it, and finally aggregates results. Its goal is to distribute CPU-intensive tasks onto a thread pool.
  • Tokio runs tasks which sometimes need to be paused in order to wait for asynchronous events. Handling tons of such tasks is no problem. Its goal is to distribute IO-intensive tasks onto a thread pool.
  • Crossbeam is all about low-level concurrency: atomics, concurrent data structures, synchronization primitives. Same idea as the std::sync module, but bigger. Its goal is to provide tools on top of which libraries like Rayon and Tokio can be built.

Asynchrony

Are we async yet?

Conceptual

Projects

Concurrency

Conceptual

Projects

Getting Started

If you do not have a Substrate development environment setup on your machine, please install it by following these directions.

For Linux / macOS

# Setup Rust and Substrate
curl https://getsubstrate.io -sSf | bash

For Windows

Refer to our Substrate Installation on Windows.

Kitchen Overview

The recipes/kitchen folder contains all the code necessary to run a Substrate node. Let us call it the Kitchen Node. There are three folders inside:

  • node - The Kitchen Node's client; the non-runtime parts of the node.
  • runtimes - Complete runtimes that can be used with the Kitchen Node.
  • pallets - Pallets that make up the runtimes. A pallet gives the runtime a particular piece of functionality. Currently, most of the recipe code is stored under this folder.

This section teaches users to interact with recipes/kitchen by

Run the Kitchen Node

To run the code in the recipes, git clone the source repository. We also want to kick-start the node compilation as it may take about 30 minutes to complete depending on your hardware.

git clone https://github.com/substrate-developer-hub/recipes.git
cd recipes/kitchen/node
./scripts/init.sh

# This step takes a while to complete
cargo build --release

Notes

Refer to the following sections to:

Once the compilation is completed, you can first purge any existing blockchain data (useful to start your node from a clean state in future) and then start the node.

# Inside `recipes/kitchen` folder

# Purge any existing blockchain data. Enter `y` upon prompt.
./target/release/kitchen-node purge-chain --dev

# Start the Kitchen Node
./target/release/kitchen-node --dev

Interact with the Kitchen Node

If you followed the instructions to get your node running, you should see blocks created on the console. You can now use our Polkadot-JS Apps to interact with your locally running node. You will use the Chain state tab to query the blockchain status and Extrinsics to send transactions to the blockchain.

To configure relevant type definitions, follow the directions in the polkadot-js docs.

Using the Kitchen

Let us take a deeper look at the Kitchen Node. Inside

kitchen/node/Cargo.toml

# -- snip --
runtime = { package = "super-runtime", path = "../runtimes/super-runtime" }
# -- snip --

You see node is bringing in the pallets in and using them to build up the node service in

kitchen/node/src/service.rs

// -- snip --
use runtime::{self, GenesisConfig, opaque::Block, RuntimeApi};
// -- snip --

macro_rules! new_full_start {
  // -- snip --
  let builder = substrate_service::ServiceBuilder::new_full::<
    runtime::opaque::Block, runtime::RuntimeApi, crate::service::Executor
  >($config)?
  // -- snip --
}

The runtime folder contains two folders for each runtime. Let's consider the super runtime as an example. super-genesis is for specifying how the first block on the blockchain (genesis block) is being produced, and super-runtime for specifying how the node runtime behaves. Now let us focus on one pallet simple-event in the runtime. In

kitchen/runtimes/super-runtime/Cargo.toml

# -- snip --

# `simple-event` pallet is specified as a relative path to the `pallets` folder
[dependencies]
simple-event = { path = "../../pallets/simple-event", default_features = false }

# -- snip --

This is where the node runtime includes the pallet simple-event written here in the recipes.

kitchen/runtimes/super-runtime/src/lib.rs

// -- snip --
use simple_event;

// -- snip --
impl simple_event::Trait for Runtime {
  type Event = Event;
}

// -- snip --
construct_runtime!(
  pub enum Runtime where
    Block = Block,
    NodeBlock = opaque::Block,
    UncheckedExtrinsic = UncheckedExtrinsic
  {
    // -- snip --
    SingleValue: single_value::{Module, Call, Storage, Event<T>},
  }
);

Finally, you can see how the simple-event pallet is specified in kitchen/pallets/simple-event/src/lib.rs.

This is the general pattern used throughout these recipes. We first talk about a new piece of pallet code stored in kitchen/pallets/<pallet-name>/src/lib.rs. The pallet is then included into a runtime by adding its name and relative path in kitchen/runtimes/<runtime-name>/Cargo.toml (if not yet added) and updating kitchen/runtimes/<runtime-name>/src/lib.rs.

Learn More

If you are interested to learn more about how to include your own pallets in a node runtime, we recommend you to go through the following two tutorials.

Pallet Fundamentals

kitchen/pallets/hello-substrate

Clone the substrate pallet template:

git clone https://github.com/substrate-developer-hub/substrate-pallet-template

At the top of the src/lib.rs file, import the following from frame-support:

use frame_support::{decl_module, decl_event, decl_storage, StorageValue, StorageMap};
use system::ensure_signed;

The blockchain's runtime storage is configured in decl_storage.

decl_storage! {
	trait Store for Module<T: Trait> as HelloWorld {
		pub LastValue get(fn last_value): u64;
		pub UserValue get(fn user_value): map T::AccountId => u64;
	}
}

Defined in decl_module, the runtime methods specify acceptable interaction with runtime storage.

decl_module! {
	pub struct Module<T: Trait> for enum Call where origin: T::Origin {
		fn deposit_event() = default;

		pub fn set_value(origin, value: u64) {
			let sender = ensure_signed(origin)?;
			LastValue::put(value);
			UserValue::<T>::insert(&sender, value);
			Self::deposit_event(RawEvent::ValueSet(sender, value));
		}
	}
}

Events are declared in decl_event. The emission of events is used to determine successful execution of the logic in the body of runtime methods.

decl_event!{
	pub enum Event<T> where
		AccountId = <T as system::Trait>::AccountId,
	{
		ValueSet(AccountId, u64),
	}
}

It is also possible to declare an error type for pallets with decl_error

Event

kitchen/pallets/simple-event, kitchen/pallets/generic-event

In Substrate, transaction finality does not guarantee the execution of functions dependent on the given transaction. To verify that functions have executed successfully, emit an event at the bottom of the function body.

Events notify the off-chain world of successful state transitions

To declare an event, use the decl_event macro.

Simple Event

The simplest example of an event uses the following syntax

decl_event!(
    pub enum Event {
        EmitInput(u32),
    }
);

The event is emitted at the bottom of the do_something function body:

Self::deposit_event(Event::EmitInput(new_number));

Events with Generic Types

Sometimes events might contin types from the pallet's Configuration Trait. In this case, it is necessary to specify additional syntax

decl_event!(
    pub enum Event<T> where AccountId = <T as system::Trait>::AccountId {
        EmitInput(AccountId, u32),
    }
);

The syntax for deposit_event now takes the RawEvent type because it is generic over the pallet's configuration trait

Self::deposit_event(RawEvent::EmitInput(user, new_number));

See the next example to use the simple event syntax in the context of verifying successful execution of an adding machine

Adding Machine

kitchen/pallets/adding-machine

A simple adding machine checks for overflow and emits an event with the result, without using storage. In the pallet's lib.rs file,

pub trait Trait: system::Trait {
    type Event: From<Event> + Into<<Self as system::Trait>::Event>;
}

decl_module! {
    pub struct Module<T: Trait> for enum Call where origin: T::Origin {
        fn deposit_event() = default;

        fn add(origin, val1: u32, val2: u32) -> Result {
            let _ = ensure_signed(origin)?;
            // checks for overflow
            let result = match val1.checked_add(val2) {
                Some(r) => r,
                None => return Err("Addition overflowed"),
            };
            Self::deposit_event(Event::Added(val1, val2, result));
            Ok(())
        }
    }
}

decl_event!(
    pub enum Event {
        Added(u32, u32, u32),
    }
);

NOTE: The event described above only wraps u32 values. If we want/need the Event type to contain multiple types from our runtime, then the decl_event would use the following syntax

decl_event!(
    pub enum Event<T> {
        ...
    }
)

In some cases, the where clause can be used to specify type aliasing for more readable code

decl_event!(
    pub enum Event<T>
    where
        Balance = BalanceOf<T>,
        <T as system::Trait>::AccountId,
        <T as system::Trait>::BlockNumber,
        <T as system::Trait>::Hash,
    {
        FakeEvent1(AccountId, Hash, BlockNumber),
        FakeEvent2(AccountId, Balance, BlockNumber),
    }
)

Single Value

kitchen/pallets/single-value

Within a specific pallet, a single value (u32 type) is stored in the runtime using the decl_storage macro

decl_storage! {
    trait Store for Module<T: Trait> as SingleValue {
        MyValue: u32;
    }
}

To interact with single storage values, it is necessary to import the frame_support::StorageValue type. Functions used to access a StorageValue are defined in frame-support:

/// Get the storage key.
fn hashed_key() -> [u8; 16];

/// true if the value exists in storage.
fn exists() -> bool;

/// Load the value from the provided storage instance.
fn get() -> Self::Query;

///Put the borrowed value at the key
fn put<Arg: Borrow<T>>(val: Arg);

/// Put an unsized and `Encode` value at the key
fn put_ref<Arg: ?Sized + Encode>(val: &Arg) where T: AsRef<Arg>;

/// Mutate the value at the key
fn mutate<R, F: FnOnce(&mut G::Query) -> R>(f: F) -> R;

/// Takes the value at the key
fn take() -> G::Query;

/// Clear the storage value
fn kill();

Therefore, the syntax to "put" Value:

<MyValue>::put(1738);

and to "get" Value:

let my_val = <MyValue>::get();

Note that we do not need the type T because the value is only of one type u32. If the T was polymorphic over more than one type, the syntax would include T in call

decl_storage! {
    trait Store for Module<T: Trait> as Example {
        MyValue: u32;
        MyAccount: T::AccountId;
    }
}

Now that we're using T::AccountId in the MyAccount storage value, it is necessary to specify that the call is generic over the trait Trait by writing

// in a runtime method in `decl_module` block
<MyAccount<T>>::get()

The requirements for setting the AccountId stored in MyAccount can be specified in the runtime and exposed via

<MyAccount<T>>::put(some_account_id);

The full example in kitchen/pallets/single-value emits events to also notify off-chain processes of when values were set and got.

Storage

The decl_storage documentation specifies how to define type-safe, persistent data that needs to be stored on-chain.

For cryptocurrencies, storage might consist of a mapping between account keys and corresponding balances.

More generally, blockchains provide an interface to store and interact with data in a verifiable and globally irreversible way. In this context, data is stored in a series of snapshots, each of which may be accessed at a later point in time, but, once created, snapshots are considered irreversible.

Arbitrary data may be stored, as long as its data type is serializable in Substrate i.e. implements Encode and Decode traits.

The previous single-value storage recipe showed how a single value can be stored in runtime storage. In this section, we cover

in-progress

Cache Multiple Calls

kitchen/pallets/storage-cache

Calls to runtime storage have an associated cost. With this in mind, multiple calls to storage values should be avoided when possible.

decl_storage! {
    trait Store for Module<T: Trait> as StorageCache {
        // copy type
        SomeCopyValue get(fn some_copy_value): u32;

        // clone type
        KingMember get(fn king_member): T::AccountId;
        GroupMembers get(fn group_members): Vec<T::AccountId>;
    }
}

For Copy types, it is easy to reuse previous storage calls by simply reusing the value (which is automatically cloned upon reuse). With this in mind, the second call in the following code is unnecessary:

fn swap_value_no_cache(origin, some_val: u32) -> Result {
    let _ = ensure_signed(origin)?;
    let original_call = <SomeCopyValue>::get();
    let some_calculation = original_call + some_val;
    // this next storage call is unnecessary and is wasteful
    let unnecessary_call = <SomeCopyValue>::get();
    // should've just used first_call here because u32 is copy
    let another_calculation = some_calculation + unnecessary_call;
    <SomeCopyValue>::put(another_calculation);
    let now = <system::Module<T>>::block_number();
    Self::deposit_event(RawEvent::InefficientValueChange(another_calculation, now));
    Ok(())
}

Instead, the initial call value should be reused. In this example, the SomeCopyValue value is Copy so we should prefer the following code without the unnecessary second call to storage:

fn swap_value_w_copy(origin, some_val: u32) -> Result {
    let _ = ensure_signed(origin)?;
    let original_call = <SomeCopyValue>::get();
    let some_calculation = original_call + some_val;
    // uses the original_call because u32 is copy
    let another_calculation = some_calculation + original_call;
    <SomeCopyValue>::put(another_calculation);
    let now = <system::Module<T>>::block_number();
    Self::deposit_event(RawEvent::InefficientValueChange(another_calculation, now));
    Ok(())
}

If the type was not Copy, but was Clone, then it is still preferred to clone the value in the method than to make another call to runtime storage.

decl_storage! {
    trait Store for Module<T: Trait> as StorageCache {
        // ...<copy type here>...
        // clone type
        KingMember get(fn king_member): T::AccountId;
        GroupMembers get(fn group_members): Vec<T::AccountId>;
    }
}

The runtime methods enable the calling account to swap the T::AccountId value in storage if

  1. the existing storage value is not in GroupMembers AND
  2. the calling account is in `Group Members

The first implementation makes a second unnecessary call to runtime storage instead of cloning the call for existing_key:

fn swap_king_no_cache(origin) -> Result {
    let new_king = ensure_signed(origin)?;
    let existing_king = <KingMember<T>>::get();

    // only places a new account if
    // (1) the existing account is not a member &&
    // (2) the new account is a member
    ensure!(!Self::is_member(existing_king), "is a member so maintains priority");
    ensure!(Self::is_member(new_king.clone()), "not a member so doesn't get priority");

    // BAD (unnecessary) storage call
    let old_king = <KingMember<T>>::get();
    // place new king
    <KingMember<T>>::put(new_king.clone());

    Self::deposit_event(RawEvent::InefficientKingSwap(old_king, new_king));
    Ok(())
}

If the existing_key is used without a clone in the event emission instead of old_king, then the compiler returns the following error

error[E0382]: use of moved value: `existing_king`
  --> src/lib.rs:93:63
   |
80 |             let existing_king = <KingMember<T>>::get();
   |                 ------------- move occurs because `existing_king` has type `<T as frame_system::Trait>::AccountId`, which does not implement the `Copy` trait
...
85 |             ensure!(!Self::is_member(existing_king), "is a member so maintains priority");
   |                                      ------------- value moved here
...
93 |             Self::deposit_event(RawEvent::InefficientKingSwap(existing_king, new_king));
   |                                                               ^^^^^^^^^^^^^ value used here after move

error: aborting due to previous error

For more information about this error, try `rustc --explain E0382`.
error: Could not compile `storage-cache`.

To learn more, run the command again with --verbose.

Fixing this only requires cloning the original call to storage before it is moved:

fn swap_king_with_cache(origin) -> Result {
    let new_king = ensure_signed(origin)?;
    let existing_king = <KingMember<T>>::get();
    // clone before existing_king is moved
    let old_king = existing_king.clone();

    // existing king is moved next
    ensure!(!Self::is_member(existing_king), "is a member so maintains priority");
    ensure!(Self::is_member(new_king.clone()), "not a member so doesn't get priority");

    // <no (unnecessary) storage call here>
    // place new king
    <KingMember<T>>::put(new_king.clone());

    // use cached old_king value here
    Self::deposit_event(RawEvent::BetterKingSwap(old_king, new_king));
    Ok(())
}

Not all types implement Copy or Clone, so it is important to discern other patterns that minimize and alleviate the cost of calls to storage.

Set Storage and Iteration

kitchen/pallets/vec-set

Storing a vector in the runtime can often be useful for managing groups and verifying membership. This recipe discusses common patterns encounted when storing vectors in runtime storage.

Verifying Group Membership

To maintain a set of AccountId to establish group ownership of decisions, it is straightforward to store a vector in the runtime of AccountId.

decl_storage! {
	trait Store for Module<T: Trait> as VecMap {
        Members get(fn members): Vec<T::AccountId>;
	}
}

It is easy to add the following helper method to verify membership elsewhere in the runtime.

impl<T: Trait> Module<T> {
    fn is_member(who: &T::AccountId) -> bool {
        <Members<T>>::get().contains(who)
    }
}

This helper method can be placed in other runtime methods to restrict certain changes to runtime storage to privileged groups. Depending on the incentive structure of the network/chain, the members in these groups may have earned membership and the subsequent access rights through loyal contributions to the system.

// use support::ensure
fn member_action(origin) -> Result {
    let member = ensure_signed(origin)?;
    ensure!(Self::is_member(&member), "not a member => cannot do action");
    // <action && || storage change>
    Ok(())
}

In this example, the helper method facilitates isolation of runtime storage access rights according to membership. In general, place ensure! checks at the top of each runtime function's logic to verify that all of the requisite checks pass before performing any storage changes.

NOTE: child trie storage provides a more efficient data structure for tracking group membership

Append vs. Mutate

decl_storage! {
	trait Store for Module<T: Trait> as VecMap {
	    CurrentValues get(fn current_values): Vec<u32>;
        NewValues get(fn new_values): Vec<u32>;
	}
}

Before 3071 was merged, it was necessary to call mutate to push new values to a vector stored in runtime storage.

fn mutate_to_append(origin) -> Result {
    let user = ensure_signed(origin)?;

    // this decodes the existing vec, appends the new values, and re-encodes the whole thing
    <CurrentValues>::mutate(|v| v.extend_from_slice(&Self::new_values()));
    Self::deposit_event(RawEvent::MutateToAppend(user));
    Ok(())
}

For vectors stored in the runtime, mutation can be relatively expensive. This follows from the fact that mutate entails decoding the vector, making changes, and re-encoding the whole vector. It seems wasteful to decode the entire vector, push a new item, and then re-encode the whole thing. This provides sufficient motivation for append:

fn append_new_entries(origin) -> Result {
    let user = ensure_signed(origin)?;

    // this encodes the new values and appends them to the already encoded existing evc
    let mut current_values = Self::current_values();
    current_values.append(&mut Self::new_values());
    Self::deposit_event(RawEvent::AppendVec(user));
    Ok(())
}

append encodes the new values, and pushes them to the already encoded vector without decoding the existing entries. This method removes the unnecessary steps for decoding and re-encoding the unchanged elements.

Iteration in the Runtime

In general, iteration in the runtime should be avoided. In the future, offchain-workers may provide a less expensive way to iterate over runtime storage items. Moreover, child tries enable cheap inclusion proofs without the same lookup costs associated with vectors.

Even so, there are a few tricks to alleviate the costs of iterating over runtime storage items like vectors. For example, it is cheaper to iterate over a slice than a vector. With this in mind, store items in the runtime as vectors and transform them into slices after making storage calls. 3041 introduced insert_ref and put_ref in order to allow equivalent reference-style types to be placed without copy (e.g. a storage item of Vec<AccountId> can now be written from a &[AccountId]). This enables greater flexibility when working with slices that are associated with vectors stored in the runtime.

Lists: Maps vs Linked Maps

kitchen/pallets/linked-map

Substrate does not natively support a list type since it may encourage dangerous habits. Unless explicitly guarded against, a list will add unbounded O(n) complexity to an operation that will only charge O(1) fees (Big O notation refresher). This opens an economic attack vector on your chain.

Emulate a list with a mapping and a counter like so:

use support::{StorageValue, StorageMap};

decl_storage! {
    trait Store for Module<T: Trait> as Example {
        TheList get(fn the_list): map u32 => T::AccountId;
        TheCounter get(fn the_counter): u32;
    }
}

This code allows us to store a list of participants in the runtime represented by AccountIds. Of course, this implementation leaves many unanswered questions such as

  • How to add and remove elements?
  • How to maintain order under mutating operations?
  • How to verify that an element exists before removing/mutating it?

This recipe answers those questions with snippets from relevant code samples:

Note: it is important to properly handle overflow/underflow and verify other relevant conditions for safety

Adding/Removing Elements in an Unbounded List

If the size of the list is not relevant, the implementation is straightforward. To add an AccountId, increment the the_counter and insert an AccountId at that index:

fn add_member(origin) -> Result {
    let who = ensure_signed(origin)?;

    let new_count = <TheCounter<T>>::get() + 1;
    // insert new member at next highest index
    <TheList<T>>::insert(new_count, who.clone());
    // increment counter
    <TheCounter<T>>::put(new_count);

    Self::deposit_event(RawEvent::MemberAdded(who));

    Ok(())
}

To remove an AccountId, call the remove method for the StorageMap type at the relevant index. In this case, it isn't necessary to update the indices of other proposals; order is not relevant.

fn remove_member_unbounded(origin, index: u32) -> Result {
    let who = ensure_signed(origin)?;

    // verify existence
    ensure!(<TheList<T>>::exists(index), "an element doesn't exist at this index");
    // for event emission
    let removed_member = <TheList<T>>::get(index);
    // remove member at provided index
    <TheList<T>>::remove(index);

    Self::deposit_event(RawEvent::MemberRemoved(removed_member));

    Ok(())
}

Because the code doesn't update the indices of other AccountIds in the map, it is necessary to verify an AccountId's existence before removing it, mutating it, or performing any other operation.

Swap and Pop for Ordered Lists

To preserve storage so that the list doesn't continue growing even after removing elements, invoke the swap and pop algorithm:

  1. swap the element to be removed with the element at the head of the list (the element with the highest index in the map)
  2. remove the element recently placed at the highest index
  3. decrement the TheCount value.

Use the swap and pop algorithm to remove elements from the list.

fn remove_member_bounded(origin, index: u32) -> Result {
    let _ = ensure_signed(origin)?;

    ensure!(<TheList<T>>::exists(index), "an element doesn't exist at this index");

    let largest_index = <TheCounter>::get();
    let member_to_remove = <TheList<T>>::take(index);
    // swap
    if index != largest_index {
        let temp = <TheList<T>>::take(largest_index);
        <TheList<T>>::insert(index, temp);
        <TheList<T>>::insert(largest_index, member_to_remove.clone());
    }
    // pop
    <TheList<T>>::remove(largest_index);
    <TheCounter>::mutate(|count| *count - 1);

    Self::deposit_event(RawEvent::MemberRemoved(member_to_remove.clone()));

    Ok(())
}

Linked Map

To trade performance for relatively simple code, use the linked_map data structure. By implementing StorageLinkedMap in addition to StorageMap, linked_map provides a method head which yields the head of the list, thereby making it unnecessary to also store the LargestIndex (the counters). The enumerate method also returns an Iterator ordered according to when (key, value) pairs were inserted into the map.

To use linked_map, import EnumerableStorageMap. Here is the new declaration in the decl_storage block:

use support::{StorageMap, EnumerableStorageMap}; // no StorageValue necessary

decl_storage! {
    trait Store for Module<T: Trait> as Example {
        LinkedList get(fn linked_list): linked_map u32 => T::AccountId;
        LinkedCounter get(fn linked_counter): u32;
    }
}

The method adding members is no different than the previously covered method, but the remove_member_linked method expresses swap and pop in a different way

fn remove_member_linked(origin, index: u32) -> Result {
    let _ = ensure_signed(origin)?;

    ensure!(<LinkedList<T>>::exists(index), "A member does not exist at this index");

    let head_index = <LinkedList<T>>::head().unwrap();
    // swap
    let member_to_remove = <LinkedList<T>>::take(index);
    let head_member = <LinkedList<T>>::take(head_index);
    <LinkedList<T>>::insert(index, head_member);
    <LinkedList<T>>::insert(head_index, member_to_remove);
    // pop
    <LinkedList<T>>::remove(head_index);

    Ok(())
}

This implementation incurs some performance costs (vs solely using StorageMap and StorageValue) because linked_map heap allocates the entire map as an iterator in order to implement the enumerate method.

Efficent Subgroup Removal by Subkey: Double Maps

kitchen/pallets/double-map

For some runtimes, it may be necessary to remove a subset of values in a key-value mapping. If the subset maintain an associated identifier type, this can be done in a clean way with the double_map via the remove_prefix api.

pub type GroupIndex = u32; // this is Encode (which is necessary for double_map)

decl_storage! {
	trait Store for Module<T: Trait> as Dmap {
        // member score (double map)
        MemberScore: double_map GroupIndex, twox_128(T::AccountId) => u32;
        // get group ID for member
        GroupMembership get(fn group_membership): map T::AccountId => GroupIndex;
        // for fast membership checks, see check-membership recipe for more details
        AllMembers get(fn all_members): Vec<T::AccountId>;
	}
}

For the purposes of this example, store the scores of each members in a map that associates this u32 value with two keys: (1) the hash of the member's AccountId and (2) a GroupIndex identifier. This allows for efficient removal of all values associated with a specific GroupIndex identifier.

fn remove_group_score(origin, group: GroupIndex) -> Result {
    let member = ensure_signed(origin)?;

    let group_id = <GroupMembership<T>>::get(member);
    // check that the member is in the group (could be improved by requiring n-of-m member support)
    ensure!(group_id == group, "member isn't in the group, can't remove it");

    // allows us to remove all group members from MemberScore at once
    <MemberScore<T>>::remove_prefix(&group_id);

    Self::deposit_event(RawEvent::RemoveGroup(group_id));
    Ok(())
}

Note: It is necessary for one of the two keys to be hashed; TODO

Child Tries

kitchen/pallets/child-trie, kitchen/pallets/smpl-crowdfund

A trie is an ordered tree structure for managing dynamic sets. For any given parent node, all descendants (children) share a common prefix associated with the parent.

This construction lends itself to efficient removal of subgroups of a dataset (similar to double_map). By associating a common prefix with related data, the dataset can be partitioned to effectively batch deletions.

Every change in the leaves percolates up to the root, thereby providing a complete, succinct history of all changes to the underlying data structure in the form of the trie root hash.

Runtime Child Storage

To interact with child tries, there are methods exposed in runtime child storage. Of the methods listed in the documentation, it is worth emphasizing the method associated with batch deletion.

/// Remove all `storage_key` key/values
pub fn kill_storage(storage_key: &[u8]) {
	runtime_io::kill_child_storage(storage_key)
}

/// Remove value associated with `key` in trie with `storage_key`
pub fn kill(storage_key: &[u8], key: &[u8]) {
	runtime_io::clear_child_storage(storage_key, key);
}

kill_storage deletes all (key, value) pairs associated with the storage_key. The basic API for interacting with a given child trie follows this format:

// pseudocode
child::do(trie_id, key, value);

To put an object in a child trie, the code would look something like

fn kv_put(index: ObjectCount, who: &T::AccountId, value_type: &ValueType) {
    let mut buf = Vec::new();
		buf.extend_from_slice(b"exchildtr");
		buf.extend_from_slice(&index.to_le_bytes()[..]);

	let id = CHILD_STORAGE_KEY_PREFIX.into_iter()
        .chain(b"default:")
        .chain(T::Hashing::hash(&buf[..]).as_ref().into_iter())
        .cloned()
        .collect();

	who.using_encoded(|b| child::put(id.as_ref(), b, value_type));
}

The code in kitchen/pallets/child-trie demonstrates a minimal way of organizing the basic child-trie api methods (as done in polkadot/runtime/crowdfund). It separates out the generation of the child trie id from the index with a runtime method id_from_index.

pub fn id_from_index(index: ObjectCount) -> Vec<u8> {
    let mut buf = Vec::new();
    buf.extend_from_slice(b"exchildtr");
    buf.extend_from_slice(&index.to_le_bytes()[..]);

    CHILD_STORAGE_KEY_PREFIX
        .into_iter()
        .chain(b"default:")
        .chain(Blake2Hasher::hash(&buf[..]).as_ref().into_iter())
        .cloned()
        .collect()
}

This results in less code for each method:

pub fn kv_put(index: ObjectCount, who: &T::AccountId, value_type: ValueType) {
    let id = Self::id_from_index(index);
    who.using_encoded(|b| child::put(id.as_ref(), b, &value_type));
}

smpl-crowdfund

kitchen/pallets/smpl-crowdfund

Child tries are useful for batch deletion of (key, value) pairs associated with a specific trie_id. This is relevant to the polkadot/crowdfund pallet, which tracks (AccountId, BalanceOf<T>) associated with a specific crowdfund. BalanceOf<T> represents the contributions of an AccountId. The identifier for each crowdfund is defined

type FundIndex = u32

With these three types, this storage item effectively manages (FundIndex, AccountId, BalanceOf<T>). By maintaining a separate child for every FundIndex, this api allows for efficient batch deletions when crowdfunds are ended and dissolved.

// polkadot/runtime/crowdfund
pub fn crowdfund_kill(index: FundIndex) {
    let id = Self::id_from_index(index);
    child::kill_storage(id.as_ref());
}

The child trie api is useful when data associated with an identifier needs to be isolated to facilitate efficient batch removal. In this case, all the information associated with a given crowdfund should be removed when the crowdfund is dissolved.

caveat coder

Each individual call to read/write to the child trie is more expensive than it would be for map or double_map. This cost is poorly amortized over a large number of calls, but can be significantly reduced by following a proper batch execution strategy.

Configurable Pallet Constants

kitchen/pallets/constant-config

To declare constant values within a runtime, it is necessary to import the Get trait from frame_support

use support::traits::Get;

Configurable constants are declared as associated types in the pallet's pub trait Trait block using the Get<T> syntax for any type T.

pub trait Trait: system::Trait {
	type Event: From<Event> + Into<<Self as system::Trait>::Event>;

    type Currency: Currency<Self::AccountId> + ReservableCurrency<Self::AccountId>;

    type MaxAddend: Get<u32>;

    // frequency with which the this value is deleted
    type ClearFrequency: Get<Self::BlockNumber>;
}

In order to make these constants accessible within the pallet, it is necessary to declare them with the const syntax in the decl_module block. Usually constants are declared at the top of this block, under fn deposit_event.

decl_module! {
    pub struct Module<T: Trait> for enum Call where origin: T::Origin {
		fn deposit_event() = default;

        const MaxAddend: u32 = T::MaxAddend::get();

        const ClearFrequency: T::BlockNumber = T::ClearFrequency::get();
    }
}

This example manipulates a single value in storage declared as SingleValue.

decl_storage! {
	trait Store for Module<T: Trait> as Example {
        SingleValue get(fn single_value): u32;
	}
}

SingleValue is set to 0 every ClearFrequency number of blocks in the on_finalize function that runs at the end of blocks execution.

fn on_finalize(n: T::BlockNumber) {
    if (n % T::ClearFrequency::get()).is_zero() {
        let c_val = <SingleValue>::get();
        <SingleValue>::put(0u32); // is this cheaper than killing?
        Self::deposit_event(Event::Cleared(c_val));
    }
}

Signed transactions may invoke the add_value runtime method to increase SingleValue as long as each call adds less than MaxAddend. There is no anti-sybil mechanism so a user could just split a larger request into multiple smaller requests to overcome the MaxAddend, but overflow is still handled appropriately.

fn add_value(origin, val_to_add: u32) -> Result {
    let _ = ensure_signed(origin)?;
    ensure!(val_to_add <= T::MaxAddend::get(), "value must be <= maximum add amount constant");

    // previous single value
    let c_val = <SingleValue>::get();

    // checks for overflow
    let result = match c_val.checked_add(val_to_add) {
        Some(r) => r,
        None => return Err("Addition overflowed"),
    };
    <SingleValue>::put(result);
    Self::deposit_event(Event::Added(c_val, val_to_add, result));
    Ok(())
}

In more complex patterns, the constant value may be used as a static, base value that is scaled by a multiplier to incorporate stateful context for calculating some dynamic fee (ie floating transaction fees).

To test the range of pallet configurations introduced by configurable constants, see custom configuration of externalities

Substrate Types and Traits

Pallets may access their own associated types as well as the associated types of other pallets in the runtime. This is seen in most pallets when they access frame_system's associated AccountId type. All pallets are tightly coupled to frame_system through this syntax, and thus have access to its types. You may optionally couple to additional pallets in this way.

Another option, loosely coupling to other pallets, is discussed further later.

pub trait Trait: system::Trait {}

This provides access to Hash, AccountId, and BlockNumber anywhere that specifies the generic <T: Trait> using T::<Type>. It also provides access to other useful types, declared in frame_system's configuration trait.

basically add a note here on why traits are important for runtime development => we are in the business of building libraries to support the configuration and modular and extensible digital infrastructure...

support::traits

Unlike in smart contract development, the way to inherit shared behavior is not to directly import other pallets. Instead, it is common to either implement the same logic in the new context or utilize a trait from frame_support to guide the new implementation. By abstracting common behavior from pallets into frame_support, Substrate makes it easy to extract and enforce best practices in the runtime.

Currency Types

kitchen/pallets/lockable-currency, kitchen/pallets/reservable-currency, kitchen/pallets/currency-imbalances

To use a balances type in the runtime, import the Currency trait from frame_support.

use support::traits::Currency;

The Currency trait provides an abstraction over a fungible assets system. To use such a fuingible asset from your pallet, include an associated type with the Currency trait bound in your pallet's configuration trait.

pub trait Trait: system::Trait {
    type Currency: Currency<Self::AccountId>;
}

Defining an associated type with this trait bound allows this pallet to access the provided methods of Currency. For example, it is straightforward to check the total issuance of the system:

// in decl_module block
T::Currency::total_issuance();

As promised, it is also possible to type alias a balances type for use in the runtime:

type BalanceOf<T> = <<T as Trait>::Currency as Currency<<T as system::Trait>::AccountId>>::Balance;

This new BalanceOf<T> type satisfies the type constraints of Self::Balance for the provided methods of Currency. This means that this type can be used for transfer, minting, and much more.

Reservable Currency

Substrate's Treasury pallet uses the Currency type for bonding spending proposals. To reserve and unreserve balances for bonding, treasury uses the ReservableCurrency trait. The import and associated type declaration follow convention

use frame_support::traits::{Currency, ReservableCurrency};

pub trait Trait: system::Trait {
    type Currency: Currency<Self::AccountId> + ReservableCurrency<Self::AccountId>;
}

To lock or unlock some quantity of funds, it is sufficient to invoke reserve and unreserve respectively

pub fn lock_funds(origin, amount: BalanceOf<T>) -> Result {
    let locker = ensure_signed(origin)?;

    T::Currency::reserve(&locker, amount)
            .map_err(|_| "locker can't afford to lock the amount requested")?;

    let now = <system::Module<T>>::block_number();

    Self::deposit_event(RawEvent::LockFunds(locker, amount, now));
    Ok(())
}

pub fn unlock_funds(origin, amount: BalanceOf<T>) -> Result {
    let unlocker = ensure_signed(origin)?;

    T::Currency::unreserve(&unlocker, amount);

    let now = <system::Module<T>>::block_number();

    Self::deposit_event(RawEvent::LockFunds(unlocker, amount, now));
    Ok(())
}

Lockable Currency

Substrate's Staking pallet similarly uses LockableCurrency trait for more nuanced handling of capital locking based on time increments. This type can be very useful in the context of economic systems that enforce accountability by collateralizing fungible resources. Import this trait in the usual way

use support::traits::{LockIdentifier, LockableCurrency}

pub trait Trait: system::Trait {
    /// The lockable currency type
    type Currency: LockableCurrency<Self::AccountId, Moment=Self::BlockNumber>;

    // Example length of a generic lock period
    type LockPeriod: Get<Self::BlockNumber>;
    ...
}

To use LockableCurrency, it is necessary to define a LockIdentifier.

const EXAMPLE_ID: LockIdentifier = *b"example ";

By using this EXAMPLE_ID, it is straightforward to define logic within the runtime to schedule locking, unlocking, and extending existing locks.

fn lock_capital(origin, amount: BalanceOf<T>) -> Result {
    let user = ensure_signed(origin)?;

    T::Currency::set_lock(
        EXAMPLE_ID,
        user.clone(),
        amount,
        T::LockPeriod::get(),
        WithdrawReasons::except(WithdrawReason::TransactionPayment),
    );

    Self::deposit_event(RawEvent::Locked(user, amount));
    Ok(())
}

Imbalances

Functions that alter balances return an object of the Imbalance type to express how much account balances have been altered in aggregate. This is useful in the context of state transitions that adjust the total supply of the Currency type in question.

To manage this supply adjustment, the OnUnbalanced handler is often used. An example might look something like

// runtime method (ie decl_module block)
pub fn reward_funds(origin, to_reward: T::AccountId, reward: BalanceOf<T>) {
    let _ = ensure_signed(origin)?;

    let mut total_imbalance = <PositiveImbalanceOf<T>>::zero();

    let r = T::Currency::deposit_into_existing(&to_reward, reward).ok();
    total_imbalance.maybe_subsume(r);
    T::Reward::on_unbalanced(total_imbalance);

    let now = <system::Module<T>>::block_number();
    Self::deposit_event(RawEvent::RewardFunds(to_reward, reward, now));
}

takeaway

The way we represent value in the runtime dictates both the security and flexibility of the underlying transactional system. Likewise, it is convenient to be able to take advantage of Rust's flexible trait system when building systems intended to rethink how we exchange information and value 🚀

Economic Security in Substrate

An algorithm is considered to be efficient if its running time is polynomial in the size of the input, and highly efficient if its running time is linear in the size of the input. It is important for all on-chain algorithms to be highly efficient, because they must scale linearly as the size of the Polkadot network grows. In contrast, off-chain algorithms are only required to be efficient. - Web3 Research

Any resources used by a transaction must explicitly be paid for, and it is a pallet author's job to ensure that appropriate fees are required. Maintaining the balance between resources used and price paid is an important design activity for runtime security.

Indeed, mispriced EVM operations have shown how operations that underestimate cost can open economic DOS attack vectors: Onwards; Underpriced EVM Operations, Under-Priced DOS Attacks on Ethereum

Substrate provides several ways to affect the fees charges for executing a transaction. Substrate developer hub contains full details about fees and weights.

  • Base fee - Applies a fixed fee to each and every transaction. A parameter in the transaction_payment pallet.

  • Length fee - Applies a fee proportional to the transaction's length in bytes. The constant is a parameter in the transaction_payment pallet.

  • Transaction weight - Each transaction can declare a weight, either fixed, or calculated from its parameters. This is exemplified briefly below and more thoroughly in the kitchen.

  • Weight to Fee - A function to convert weight to fee. It doesn't need to be linear, although it often is. The same conversion function is applied across all transactions from all pallets in the runtime. This is exemplified briefly below and more thoroughly in the kitchen.

Assigning Transaction Weights

For simple transactions a fixed weight will do.

decl_module! {
	pub struct Module<T: Trait> for enum Call {

		#[weight = SimpleDispatchInfo::FixedNormal(100)]
		fn store_value(_origin, entry: u32) -> Result {
			// --snip--
		}

For more complex transactions, custom weight calculations can be performed.

pub struct Conditional(u32);

impl WeighData<(&bool, &u32)> for Conditional {
	fn weigh_data(&self, (switch, val): (&bool, &u32)) -> Weight {

		if *switch {
			val.saturating_mul(self.0)
		}
		else {
			self.0
		}
	}
}

In addition to the WeightData Trait, shown above, types that are used to calculate transaction weights, must also implement ClassifyDispatch, and PaysFee. These examples and several others can be compiled in the kitchen.

While you can make reasonable estimates of resource consumption at design time, it is always best to actually measure the resources required of your functions through an empirical process. Failure to perform such rigorous measurement may result in an economically insecure chain.

Converting Weight To Fees

In many cases converting weight to fees 1:1 will suffice and be accomplished with ConvertInto. This approach is taken in the node template as well as the kitchen's own super runtime.

impl transaction_payment::Trait for Runtime {
	// --snip--
	type WeightToFee = ConvertInto;
}

This example uses a quadratic conversion and supports custom coefficients

pub struct QuadraticWeightToFee<C0, C1, C2>(C0, C1, C2);

impl<C0, C1, C2> Convert<Weight, Balance> for QuadraticWeightToFee<C0, C1, C2>
	where C0: Get<Balance>, C1: Get<Balance>, C2: Get<Balance> {

	fn convert(w: Weight) -> Balance {
		let c0 = C0::get();
		let c1 = C1::get();
		let c2 = C2::get();
		let w = Balance::from(w);

		// TODO use safe math
		c0 + c1 * w + c2 * w * w
	}
}

These examples, and several others can be compiled in the kitchen.

Instantiable Pallets

Instantiable pallets enable multiple instances of the same pallet logic within a single runtime. Each instance of the pallet has its own independent storage, and extrinsics must specify which instance of the pallet they are intended for. These patterns are illustrated in the kitchen in the last-caller and default-instance pallets.

Some use cases:

  • Token chain hosts two independent cryptocurrencies.
  • Marketplace track users' reputations as buyers separately from their reputations as sellers.
  • Governance has two (or more) houses which act similarly internally.

Substrate's own Balances and Collective pallets are good examples of real-world code using this technique. The default Substrate node has two instances of the Collectives pallet that make up its Council and Technical Committee. Each collective has its own storage, events, and configuration.

Council: collective::<Instance1>::{Module, Call, Storage, Origin<T>, Event<T>, Config<T>},
TechnicalCommittee: collective::<Instance2>::{Module, Call, Storage, Origin<T>, Event<T>, Config<T>}

Writing an Instantiable Pallet

Writing an instantiable pallet is almost entirely the same process as writing a plain non-instantiable pallet. There are just a few places where the syntax differs.

You must call decl_storage!

Instantiable pallets must call the decl_storage! macro so that the Instance type is created.

Configuration Trait

pub trait Trait<I: Instance>: system::Trait {
	/// The overarching event type.
	type Event: From<Event<Self, I>> + Into<<Self as system::Trait>::Event>;
}

Storage Declaration

decl_storage! {
	trait Store for Module<T: Trait<I>, I: Instance> as TemplatePallet {
		...
	}
}

Declaring the Module Struct

decl_module! {
	/// The module declaration.
	pub struct Module<T: Trait<I>, I: Instance> for enum Call where origin: T::Origin {
		...
	}
}

Accessing Storage

<Something<T, I>>::put(something);

If the storage item does not use any types specified in the configuration trait, the T is omitted, as always.

<Something<I>>::put(something);

Event initialization

fn deposit_event() = default;

Event Declaration

decl_event!(
	pub enum Event<T, I> where AccountId = <T as system::Trait>::AccountId {
		...
	}
}

Installing a Pallet Instance in a Runtime

The syntax for including an instance of an instantiable pallet in a runtime is slightly different than for a regular pallet. The only exception is for pallets that use the Default Instance feature described below.

Implementing Configuration Traits

Each instance needs to be configured separately. Configuration consists of implementing the specific instance's trait. The following snippet shows a configuration for Instance1.

impl template::Trait<template::Instance1> for Runtime {
	type Event = Event;
}

Using the construct_runtime! Macro

The final step of installing the pallet instance in your runtime is updating the construct_runtime! macro. You may give each instance a meaningful name. Here I've called Instance1 FirstTemplate.

FirstTemplate: template::<Instance1>::{Module, Call, Storage, Event<T>, Config},

Default Instance

One drawback of instantiable pallets, as we've presented them so far, is that they require the runtime designer to use the more elaborate syntax even if they only desire a single instance of the pallet. To alleviate this inconvenience, Substrate provides a feature known as DefaultInstance. This allows runtime developers to deploy an instantiable pallet exactly as they would if it were not instantiable provided they only use a single instance.

To make your instantiable pallet support DefaultInstance, you must specify it in four places.

pub trait Trait<I=DefaultInstance>: system::Trait {
decl_storage! {
  trait Store for Module<T: Trait<I>, I: Instance=DefaultInstance> as TemplateModule {
    ...
  }
}
decl_module! {
    pub struct Module<T: Trait<I>, I: Instance = DefaultInstance> for enum Call where origin: T::Origin {
        ...
    }
}
decl_event!(
	pub enum Event<T, I=DefaultInstance> where ... {
    ...
  }
}

Having made these changes, a developer who uses your pallet doesn't need to know or care that your pallet is instantable. They can deploy it just as they would any other pallet.

Genesis Configuration

Some pallets require a genesis configuration to be specified. Let's look to the default Substrate node's use of the Collective pallet as an example.

In its chain_spec.rs file we see

GenesisConfig {
	...
	collective_Instance1: Some(CouncilConfig {
		members: vec![],
		phantom: Default::default(),
	}),
	collective_Instance2: Some(TechnicalCommitteeConfig {
		members: vec![],
		phantom: Default::default(),
	}),
	...
}

Declarative Syntax

Unlike conventional software development kits that abstract away low-level decisions, Substrate grants developers fine-grain control over the underlying implementation. This approach fosters high-performance, modular applications. At the same time, it also demands increased attention from developers. With great power comes great responsibility.

Indeed, Substrate developers have to exercise incredible caution. The bare-metal control that they maintain over the runtime logic introduces new attack vectors. In the context of blockchains, the cost of bugs scale with the amount of capital secured by the application. Likewise, developers should generally abide by a few rules when building with Substrate. These rules may not hold in every situation; Substrate offers optimization in context.

Each of the recipes in this section are oriented around increasing

Pallet Development Criteria

  1. Pallets should be relatively independent pieces of code; if your pallet is tied to many other pallets, it should be a smart contract. See the substrate-contracts-workshop for more details with respect to smart contract programming on Substrate. Also, use traits for abstracting shared behavior.

  2. It should not be possible for your code to panic after storage changes. Poor error handling in Substrate can brick the blockchain, rendering it useless thereafter. With this in mind, it is very important to structure code according to declarative, condition-oriented design patterns.

Verify First, Write Last

Within each dispatchable function, it is important to perform requisite checks prior to any storage changes. Unlike existing smart contract platforms, Substrate requires greater attention to detail because mid-function panics will persist any prior changes made to storage.

Place ensure! checks at the top of each runtime function's logic to verify that all requisite conditions are met before performing any storage changes. This is similar to require() checks at the top of function bodies in Solidity contracts.

In the set storage and iteration, a vector was stored in the runtime to allow for simple membership checks for methods only available to members.

decl_storage! {
	trait Store for Module<T: Trait> as VecMap {
        Members get(fn members): Vec<T::AccountId>;
	}
}
...
impl<T: Trait> Module<T> {
    fn is_member(who: &T::AccountId) -> bool {
        <Members<T>>::get().contains(who)
    }
}

"By returning bool, we can easily use these methods in ensure! statements to verify relevant state conditions before making requests in the main runtime methods."

fn member_action(origin) -> Result {
    let member = ensure_signed(origin)?;
    ensure!(Self::is_member(&member), "not a member => cannot do action");
    // <action && || storage change>
    Ok(())
}

Indeed, this pattern of extracting runtime checks into separate functions and invoking the ensure macro in their place is useful. It produces readable code and encourages targeted testing to more easily identify the source of logic errors.

This github comment might help when visualizing declarative patterns in practice.

Bonus Reading

Safe Math

We can use the checked traits in substrate-primitives to protect against overflow/underflow when incrementing/decrementing objects in our runtime. To follow the Substrat collectable tutorial example, use checked_add() to safely handle the possibility of overflow when incremementing a global counter. Note that this check is similar to SafeMath in Solidity.

use runtime_primitives::traits::CheckedAdd;

let all_people_count = Self::num_of_people();

let new_all_people_count = all_people_count.checked_add(1).ok_or("Overflow adding a new person")?;

ok_or() transforms an Option from Some(value) to Ok(value) or None to Err(error). The ? operator facilitates error propagation. In this case, using ok_or() is the same as writing

let new_all_people_count = match all_people_count.checked_add(1) {
    Some (c) => c,
    None => return Err("Overflow adding a new person"),
};

todo

  • ? for error propagation
  • Permill, Perbill, Fixed64 types for large arithmetic
  • quantization benchmarks in the treasury tests to verify that large arithmetic stays in a comfortable error bound
  • ADD BACK IN NEW RECIPE: collide and the question of whether maps prevent key collisions? could discuss sort, sort_unstable, and the ordering traits here...

Permissioned Methods

kitchen/pallets/check-membership

It is often useful to designate some functions as permissioned and, therefore, accessible only to a defined group. In this case, we check that the transaction that invokes the runtime function is signed before verifying that the signature corresponds to a member of the permissioned set.

To manage the set of members allowed to access the methods in question, we may store a vector in runtime storage. Without access to the standard library, it is necessary to add the sr-std dependency to the Cargo.toml file and import its prelude:

[dependencies.rstd]
default_features = false
git = 'https://github.com/paritytech/substrate.git'
package = 'sr-std'
rev = '3e651110aa06aa835790df63410a29676243fc54'

The alias for sr-std used is rstd which follows substrate's conventions. To import a vector type that can be stored in the runtime:

use rstd::prelude::*;

In the runtime, the membership set can be stored as

decl_storage! {
    trait Store for Module<T: Trait> as PGeneric {
        Members get(fn members): Vec<&T::AccountId>;
    }
}

If the set was determined to be permissionless, we could express this in the runtime as

fn add_member(origin) -> Result {
	// unwrap signed extrinsic into AccountId
	let new_member = ensure_signed(origin)?;
	// check that the AccountId is contained in the `Members` vector
	ensure!(!Self::members().contains(&new_member), "already a member, don't add duplicates");
	// append the new member to the vec storage value
	<Members<T>>::append(&[new_member.clone()])?;
	Self::deposit_event(RawEvent::AddMember(new_member));
	Ok(())
}

To increase the readability of the code, the membership check can be extracted into its own auxiliary runtime method.

impl<T: Trait> Module<T> {
    pub fn is_member(who: &T::AccountId) -> bool {
        Self::members().contains(who)
    }
}

The add_member method now reads

fn add_member(origin) -> Result {
	let new_member = ensure_signed(origin)?;
	// the membership check is now shorter
	ensure!(!Self::is_member(&new_member), "already a member");

	<Members<T>>::append(&[new_member.clone()])?;
	Self::deposit_event(RawEvent::AddMember(new_member));
	Ok(())
}

sudo

custom origin

Testing

Although the Rust compiler ensures safe memory management, it cannot formally verify the correctness of a program's logic. Fortunately, Rust also comes with great libraries and documentation for writing unit and integration tests. When you initiate code with Cargo, test scaffolding is automatically generated to simplify the developer experience. Basic testing concepts and syntax are covered in depth in Chapter 11 of the Rust Book.

There's also more rigorous testing systems ranging from mocking and fuzzing to formal verification. See quickcheck for an example of a property-based testing framework ported from Haskell to Rust.

Kitchen Pallets with Unit Tests

The following modules in the kitchen have partial unit test coverage

Cooking in the Kitchen (Running Tests)

To run the tests, clone the repo

$ git clone https://github.com/substrate-developer-hub/recipes

Enter the path to the pallet to be tested

$ cd kitchen/pallets/<some-module>

For example, to test constant-config, used in Configurable Constants,

$ cd kitchen/pallets/constant-config/
$ cargo test

Writing unit tests is one of the best ways to understand the code. Although unit tests are not comprehensive, they provide a first check to verify that the programmer's basic invariants are not violated in the presence of obvious, expected state changes.

Sauce

Over the past few weeks, testing has driven a significant rewrite of the kitchen. This increased focus on testing and benchmarking will continue over the next few weeks in the sauce, starting with

Mock Runtime for Unit Testing

see root for list of kitchen pallets with unit test coverage

At the bottom of the pallet, place unit tests in a separate rust module with a special compilation flag

#[cfg(test)]
mod tests {
	...
}

To use the logic from the pallet under test, bring Module and Trait into scope.

use crate::{Module, Trait};

Now, declare the mock runtime as a unit structure

#[derive(Clone, PartialEq, Eq, Debug)]
pub struct TestRuntime;

The derive macro attribute provides implementations of the Clone + PartialEq + Eq + Debug traits for the TestRuntime struct.

The mock runtime also needs to implement the tested pallet's Trait. If it is unnecessary to test the pallet's Event type, the type can be set to (). See further below to test the pallet's Event enum.

impl Trait for TestRuntime {
	type Event = ();
}

Next, we create a new type that wraps the mock TestRuntime in the pallet's Module.

pub type HelloSubstrate = Module<TestRuntime>;

It may be helpful to read this as type aliasing our configured mock runtime to work with the pallet's Module, which is what is ultimately being tested.

impl system::Trait

In many cases, the pallet's Trait inherits system::Trait like

pub trait Trait: system::Trait {
	type Event: From<Event<Self>> + Into<<Self as system::Trait>::Event>;
}

The mock runtime must inherit and define the system::Trait associated types (remember). To do so, impl the system::Trait for TestRuntime with a few types imported from other crates,

use support::{impl_outer_event, impl_outer_origin, parameter_types};
use runtime_primitives::{Perbill, traits::{IdentityLookup, BlakeTwo256}, testing::Header};

#[derive(Clone, PartialEq, Eq, Debug)]
pub struct TestRuntime;
parameter_types! {
	pub const BlockHashCount: u64 = 250;
	pub const MaximumBlockWeight: u32 = 1024;
	pub const MaximumBlockLength: u32 = 2 * 1024;
	pub const AvailableBlockRatio: Perbill = Perbill::one();
}
impl system::Trait for TestRuntime {
	type Origin = Origin;
	type Index = u64;
	type Call = ();
	type BlockNumber = u64;
	type Hash = BlakeTwo256;
	type AccountId = u64;
	type Lookup = IdentityLookup<Self::AccountId>;
	type Header = Header;
	type Event = TestEvent;
	type BlockHashCount = BlockHashCount;
	type MaximumBlockWeight = MaximumBlockWeight;
	type MaximumBlockLength = MaximumBlockLength;
	type AvailableBlockRatio = AvailableBlockRatio;
	type Version = ();
}

pub type System = system::Module<TestRuntime>;

With this, it is possible to use this type in the unit tests. For example, the block number can be set with set_block_number

#[test]
fn add_emits_correct_event() {
	// ExtBuilder syntax is explained further below
	ExtBuilder::build().execute_with(|| {
		System::set_block_number(2);
		// some assert statements and HelloSubstrate calls
	}
}

Basic Test Environments

To build the test runtime environment, import runtime_io

use runtime_io;

In the Cargo.toml, this only needs to be imported under dev-dependencies since it is only used in the tests module,

[dev-dependencies.runtime-io]
default_features = false
git = 'https://github.com/paritytech/substrate.git'
package = 'sr-io'
rev = '6ae3b6c4ddc03d4cdb10bd1d417b95d20f4c1b6e'

There is more than one pattern for building a mock runtime environment for testing pallet logic. Two patterns are presented below. The latter is generally favored for reasons discussed in custom test environment

  • new_test_ext - consolidates all the logic for building the environment to a single public method, but isn't relatively configurable (i.e. uses one set of pallet constants)
  • ExtBuilder - define methods on the unit struct ExtBuilder to facilitate a flexible environment for tests (i.e. can reconfigure pallet constants in every test if necessary)

new_test_ext

kitchen/pallets/smpl-treasury

In smpl-treasury, use the balances::GenesisConfig and the pallet's Genesis::<TestRuntime> to set the balances of the test accounts and establish council membership in the returned test environment.

pub fn new_test_ext() -> runtime_io::TestExternalities {
	let mut t = system::GenesisConfig::default().build_storage::<TestRuntime>().unwrap();
	balances::GenesisConfig::<TestRuntime> {
		balances: vec![
			// members of council (can also be users)
			(1, 13),
			(2, 11),
			(3, 1),
			(4, 3),
			(5, 19),
			(6, 23),
			(7, 17),
			// users, not members of council
			(8, 1),
			(9, 22),
			(10, 46),
		],
		vesting: vec![],
	}.assimilate_storage(&mut t).unwrap();
	GenesisConfig::<TestRuntime>{
		council: vec![
			1,
			2,
			3,
			4,
			5,
			6,
			7,
		]
	}.assimilate_storage(&mut t).unwrap();
	t.into()
}

More specifically, this sets the AccountIds in the range of [1, 7] inclusive as the members of the council. This is expressed in the decl_module block with the addition of an add_extra_genesis block,

add_extra_genesis {
	build(|config| {
		// ..other stuff..
		<Council<T>>::put(&config.council);
	});
}

To use new_test_ext in a runtime test, we call the method and call execute_with on the returned runtime_io::TestExternalities

#[test]
fn fake_test() {
	new_test_ext().execute_with(|| {
		// test logic
	})
}

execute_with executes all logic expressed in the closure within the configured runtime test environment specified in new_test_ext

ExtBuilder

kitchen/pallets/struct-storage

Another approach for a more flexible runtime test environment instantiates a unit struct ExtBuilder,

pub struct ExtBuilder;

The behavior for constructing the test environment is contained the methods on the ExtBuilder unit structure. This fosters multiple levels of configuration depending on if the test requires a common default instance of the environment or a more specific edge case configuration. The latter is explored in more detail in Custom Test Environment.

Like new_test_ext, the build() method on the ExtBuilder object returns an instance of TestExternalities. Externalities are an abstraction that allows the runtime to access features of the outer node such as storage or offchain workers.

In this case, create a mock storage from the default genesis configuration.

impl ExtBuilder {
	pub fn build() -> runtime_io::TestExternalities {
		let mut storage = system::GenesisConfig::default().build_storage::<TestRuntime>().unwrap();
		runtime_io::TestExternalities::from(storage)
	}
}

which calls some methods to create a test environment,

#[test]
fn fake_test_example() {
	ExtBuilder::build().execute_with(|| {
		// ...test conditions...
	})
}

While testing in this environment, runtimes that require signed extrinsics (aka take origin as a parameter) will require transactions coming from an Origin. This requires importing the impl_outer_origin macro from support

use support::{impl_outer_origin};

impl_outer_origin!{
	pub enum Origin for TestRuntime {}
}

It is possible to placed signed transactions as parameters in runtime methods that require the origin input. See the full code in the kitchen, but this looks like

#[test]
fn last_value_updates() {
	ExtBuilder::build().execute_with(|| {
		HelloSubstrate::set_value(Origin::signed(1), 10u64);
		// some assert statements
	})
}

Run these tests with cargo test, an optional parameter is the test's name to only run that test and not all tests.

NOTE: the input to Origin::signed is the system::Trait's AccountId type which was set to u64 for the TestRuntime implementation. In theory, this could be set to some other type as long as it conforms to the trait bound,

pub trait Trait: 'static + Eq + Clone {
    //...
    type AccountId: Parameter + Member + MaybeSerializeDeserialize + Debug + MaybeDisplay + Ord + Default;
	//...
}

Common Tests

To verify that our pallet code behaves as expected, it is necessary to check a few conditions with unit tests. Intuitively, the order of the testing may resemble the structure of runtime method development.

  1. Within each runtime method, declarative checks are made prior to any state change. These checks ensure that any required conditions are met before all changes occur; need to ensure that panics panic.
  2. Next, verify that the expected storage changes occurred.
  3. Finally, check that the expected events were emitted with correct values.

Checks before Changes are Enforced (i.e. Panics Panic)

The Verify First, Write Last recipe encourages verifying certain conditions before changing storage values. In tests, it might be desirable to verify that invalid inputs return the expected error message.

In kitchen/pallets/adding-machine, the runtime method add checks for overflow

decl_module! {
    pub struct Module<T: Trait> for enum Call where origin: T::Origin {
        fn deposit_event() = default;

        fn add(origin, val1: u32, val2: u32) -> Result {
            let _ = ensure_signed(origin)?;
            // checks for overflow
            let result = match val1.checked_add(val2) {
                Some(r) => r,
                None => return Err("Addition overflowed"),
            };
            Self::deposit_event(Event::Added(val1, val2, result));
            Ok(())
        }
    }
}

The test below verifies that the expected error is thrown for a specific case of overflow.

#[test]
fn overflow_fails() {
	ExtBuilder::build().execute_with(|| {
		assert_err!(
			AddingMachine::add(Origin::signed(3), u32::max_value(), 1),
			"Addition overflowed"
		);
	})
}

This requires importing the assert_err macro from support. With all the previous imported objects,

#[cfg(test)]
mod tests {
	use support::{assert_err, impl_outer_event, impl_outer_origin, parameter_types};
	// more imports and tests
}

For more examples, see Substrate's own pallets -- mock.rs for mock runtime scaffolding and test.rs for unit tests.

Expected Changes to Storage are Triggered

Changes to storage can be checked by direct calls to the storage values. The syntax is the same as it would be in the pallet's runtime methods

#[test]
fn last_value_updates() {
	ExtBuilder::build().execute_with(|| {
		let expected = 10u64;
		HelloSubstrate::set_value(Origin::signed(1), expected);
		assert_eq!(HelloSubstrate::last_value(), expected);
		// .. more assert statements
	})
}

For context, the tested pallets's decl_storage block looks like

decl_storage! {
	trait Store for Module<T: Trait> as HelloSubstrate{
		pub LastValue get(fn last_value): u64;
		pub UserValue get(fn user_value): map T::AccountId => u64;
	}
}

Updates to UserValue are tested in last_value_updates in kitchen/pallets/hello-substrate.

Expected Events are Emitted

The common way of testing expected event emission behavior requires importing support's impl_outer_event! macro

use support::impl_outer_event;

The TestEvent enum imports and uses the pallet's Event enum. The new local pallet, hello_substrate, re-exports the contents of the root to give a name for the current crate to impl_outer_event!.

mod hello_substrate {
	pub use crate::Event;
}

impl_outer_event! {
	pub enum TestEvent for TestRuntime {
		hello_substrate<T>,
	}
}

impl Trait for TestRuntime {
	type Event = TestEvent;
}

Testing the correct emission of events compares constructions of expected events with the entries in the System::events vector of EventRecords. In kitchen/pallets/adding-machine,

#[test]
fn add_emits_correct_event() {
	ExtBuilder::build().execute_with(|| {
		AddingMachine::add(Origin::signed(1), 6, 9);

		assert_eq!(
			System::events(),
			vec![
				EventRecord {
					phase: Phase::ApplyExtrinsic(0),
					event: TestEvent::added(crate::Event::Added(6, 9, 15)),
					topics: vec![],
				},
			]
		);
	})
}

This check requires importing from system

use system::{EventRecord, Phase};

A more ergonomic way of testing whether a specific event was emitted might use the System::events().iter(). This pattern doesn't require the previous imports, but it does require importing RawEvent (or Event) from the pallet and ensure_signed from system to convert signed extrinsics to the underlying AccountId,

#[cfg(test)]
mod tests {
	// other imports
	use system::ensure_signed;
	use super::RawEvent; // if no RawEvent, then `use super::Event;`
	// tests
}

In kitchen/pallets/hello-substrate,

#[test]
fn last_value_updates() {
	ExtBuilder::build().execute_with(|| {
		HelloSubstrate::set_value(Origin::signed(1), 10u64);
		// some assert checks

		let id_1 = ensure_signed(Origin::signed(1)).unwrap();
		let expected_event1 = TestEvent::hello_substrate(
			RawEvent::ValueSet(id_1, 10),
		);
		assert!(System::events().iter().any(|a| a.event == expected_event1));
	})
}

This test constructs an expected_event1 based on the event that the developer expects will be emitted upon the successful execution of logic in HelloSubstrate::set_value. The assert!() statement checks if the expected_event1 matches the .event field for any EventRecord in the System::events() vector.

Custom Test Environment

execution-schedule's configuration trait has three configurable constants. For this mock runtime, the ExtBuilder defines setters to enable the TestExternalities instance for each unit test to configure the local test runtime environment with different value assignments. For context, the Trait for execution-schedule,

// other type aliases
pub type PriorityScore = u32;

pub trait Trait: system::Trait {
    /// Overarching event type
    type Event: From<Event<Self>> + Into<<Self as system::Trait>::Event>;

    /// Quota for members to signal task priority every ExecutionFrequency
    type SignalQuota: Get<PriorityScore>;

    /// The frequency of batch executions for tasks (in `on_finalize`)
    type ExecutionFrequency: Get<Self::BlockNumber>;

    /// The maximum number of tasks that can be approved in an `ExecutionFrequency` period
    type TaskLimit: Get<PriorityScore>;
}

The mock runtime environment extends the previously discussed ExtBuilder pattern with fields for each configurable constant and a default implementation.

This completes the builder pattern by defining a default configuraton to be used in a plurality of test cases while also providing setter methods to overwrite the values for each field.

pub struct ExtBuilder {
    signal_quota: u32,
    execution_frequency: u64,
    task_limit: u32,
}
impl Default for ExtBuilder {
    fn default() -> Self {
        Self {
            signal_quota: 100u32,
            execution_frequency: 5u64,
            task_limit: 10u32,
        }
    }
}

The setter methods for each configurable constant are defined in the ExtBuilder methods. This allows each instance of ExtBuilder to set the constant parameters for the unit test in question.

impl ExtBuilder {
    pub fn signal_quota(mut self, signal_quota: u32) -> Self {
        self.signal_quota = signal_quota;
        self
    }
    pub fn execution_frequency(mut self, execution_frequency: u64) -> Self {
        self.execution_frequency = execution_frequency;
        self
    }
    pub fn task_limit(mut self, task_limit: u32) -> Self {
        self.task_limit = task_limit;
        self
    }
    // more methods e.g. build()
}

To allow for separate copies of the constant objects to be used in each thread, the variables assigned as constants are declared as thread_local!,

thread_local! {
    static SIGNAL_QUOTA: RefCell<u32> = RefCell::new(0);
    static EXECUTION_FREQUENCY: RefCell<u64> = RefCell::new(0);
    static TASK_LIMIT: RefCell<u32> = RefCell::new(0);
}

Each configurable constant type also maintains unit structs with implementation of Get<T> from the type T assigned to the pallet constant in the mock runtime implementation.

pub struct SignalQuota;
impl Get<u32> for SignalQuota {
    fn get() -> u32 {
        SIGNAL_QUOTA.with(|v| *v.borrow())
    }
}

pub struct ExecutionFrequency;
impl Get<u64> for ExecutionFrequency {
    fn get() -> u64 {
        EXECUTION_FREQUENCY.with(|v| *v.borrow())
    }
}

pub struct TaskLimit;
impl Get<u32> for TaskLimit {
    fn get() -> u32 {
        TASK_LIMIT.with(|v| *v.borrow())
    }
}

The build method on ExtBuilder sets the associated constants before building the default storage configuration.

impl ExtBuilder {
    // setters
    pub fn set_associated_consts(&self) {
        SIGNAL_QUOTA.with(|v| *v.borrow_mut() = self.signal_quota);
        EXECUTION_FREQUENCY.with(|v| *v.borrow_mut() = self.execution_frequency);
        TASK_LIMIT.with(|v| *v.borrow_mut() = self.task_limit);
    }
    // build()
}

To build the default test environment, the syntax looks like

#[test]
fn fake_test() {
    ExtBuilder::default()
        .build()
        .execute_with(|| {
            // testing logic and checks
        })
}

To configure a test environment in which the execution_frequency is set to 2, the eras_change_correctly test invokes the execution_frequency setter declared in as a method on ExtBuilder,

#[test]
fn fake_test2() {
    ExtBuilder::default()
        .execution_frequency(2)
        .build()
        .execute_with(|| {
            // testing logic and checks
        })
}

The test environment mocked above is actually used for the cursory and incomplete test eras_change_correctly. This test guided the structure of the if condition in on_initialize to periodically reset the SignalBank and increment the Era.

For more examples of the mock runtime scaffolding pattern used in execution-schedule, see balances/mock.rs and contract/tests.rs.

Dessert 🍫

Check out awesome-substrate for projects, events, and all the latest Substrate news!

Featured Tutorials

More Resources

You can learn more knowledge about Substrate by following these resources: