Boot into the live environment of your installation medium. Open a terminal and change the password of the root user to something easy using:

1

$ sudo passwd root

and then change the user to root using

1

$ su -p

This helps us avoid using sudo at the beginning of each command. The rest of the guide assumes you’re the root user.

Change the default btrfs partition configuration

First, we need to modify the configuration file for partman that the installer uses for partitioning the disks.

Open /usr/lib/partman/mount.d/70btrfs with a text editor and add the options noatime,compress=zstd,space_cache=v2 alongside subvol as specified below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

case $type in
    btrfs)
    options="${options%,subvol=*}"
    options="${options%subvol=*}"
    mount -t btrfs ${options:+-o "$options"} $fs /target$mp || exit 1
    case $mp in
        /)
        btrfs subvolume create /target$mp/@
        chmod 755 /target$mp/@
        umount /target$mp
        options="${options:+$options,}subvol=@,noatime,compress=zstd,space_cache=v2" # <-- Change here
        mount -t btrfs -o $options $fs /target$mp
        ;;
        /home)
        btrfs subvolume create /target$mp/@home
        chmod 755 /target$mp/@home
        umount /target$mp
        options="${options:+$options,}subvol=@home,noatime,compress=zstd,space_cache=v2" # <-- Change here
        mount -t btrfs -o $options $fs /target$mp
        ;;
    esac
    echo "umount /target$mp"
    exit 0
    ;;
esac

Start the installer and proceed with the installation as you normally would.

Disk partition

• Create an EFI system partition with around 512 MB of space
• Create another partition that spans the rest of the disk, format it with btrfs and set the mount point at /

Choose the partition that you created for EFI (EFI System Partition) to install the bootloader to and proceed with the installation to the end. DO NOT boot into the installed system just yet and stay in the live environment (do not restart).

Creating the necessary subvolumes

We will be using the nested subvolume layout as described in the btrfs documentation.

Mount the btrfs partition that you installed your system on using the same options you used in the previous section to /mnt. For example:

root@kubuntu:~# mount /dev/vda2 -o noatime,compress=zstd,space_cache=v2 /mnt

You should see the following two directories under /mnt:

root@kubuntu:~# ls /mnt
@  @home

Go to /mnt/@ and remove the following empty directories:

root@kubuntu:/mnt/@# rm -rf opt/ tmp/ srv/ home/

Rename the following directories and add .bak to the end of their names:

root@kubuntu:/mnt/@# mv root/ root.bak
root@kubuntu:/mnt/@# mv var/ var.bak
root@kubuntu:/mnt/@# mv boot/grub/x86_64-efi boot/grub/x86_64-efi.bak

Create an empty directory called var

root@kubuntu:/mnt/@# mkdir -p var/lib
root@kubuntu:/mnt/@# mkdir -p root

Create the following subvolumes under @:

btrfs subvolume create /mnt/@/boot/grub/x86_64-efi
btrfs subvolume create /mnt/@/home
btrfs subvolume create /mnt/@/root
btrfs subvolume create /mnt/@/opt
btrfs subvolume create /mnt/@/srv
btrfs subvolume create /mnt/@/tmp
btrfs subvolume create /mnt/@/var/log
btrfs subvolume create /mnt/@/var/crash
btrfs subvolume create /mnt/@/var/cache
btrfs subvolume create /mnt/@/var/tmp
btrfs subvolume create /mnt/@/var/opt
btrfs subvolume create /mnt/@/var/spool
btrfs subvolume create /mnt/@/var/lib/flatpak

Go to var and change disable Copy-on-Write for the created directories:

root@kubuntu:/mnt/@/var# chattr +C cache/ crash/ tmp/ opt/ log/ spool/ lib/flatpak/

You can use lsattr to check and see if CoW has correctly been disabled.

Now move the contents of @home, root.bak, and var.bak to the new locations: (ignore any errors for root.bak, since it might be empty of non-hidden files)

mv /mnt/@home/{.,}* /mnt/@/home
mv /mnt/@/root.bak/{.,}* /mnt/@/root
mv /mnt/@/boot/grub/x86_64-efi.bak/* /mnt/@/boot/grub/x86_64-efi

cp -a --reflink=never /mnt/@/var.bak/crash/. /mnt/@/var/crash/
cp -a --reflink=never /mnt/@/var.bak/cache/. /mnt/@/var/cache/
cp -a --reflink=never /mnt/@/var.bak/log/.   /mnt/@/var/log/
cp -a --reflink=never /mnt/@/var.bak/spool/. /mnt/@/var/spool/
cp -a --reflink=never /mnt/@/var.bak/tmp/.   /mnt/@/var/tmp/
cp -a --reflink=never /mnt/@/var.bak/opt/.   /mnt/@/var/opt/

Go to var.bak and remove the copied directories:

root@kubuntu:/mnt/@/var.bak# rm -rf crash/ cache/ tmp/ opt/ log/ spool/

Copy the rest of var.bak to var

cp -a /mnt/@/var.bak/. /mnt/@/var

remove the following directories:

rm -rf /mnt/@/root.bak /mnt/@/var.bak /mnt/@/boot/grub/x86_64-efi.bak/

and remove the @home subvolume:

btrfs subvolume delete /mnt/@home/

The final subvolume layout should look like this:

root@kubuntu:~# sudo btrfs subvolume list -apt /mnt
ID	gen	parent	top level	path	
--	---	------	---------	----	
256	81	5	5		@
258	65	256	256		<FS_TREE>/@/home
259	69	256	256		<FS_TREE>/@/root
260	54	256	256		<FS_TREE>/@/opt
261	55	256	256		<FS_TREE>/@/srv
262	56	256	256		<FS_TREE>/@/tmp
269	64	256	256		<FS_TREE>/@/var/lib/flatpak
270	78	256	256		<FS_TREE>/@/var/log
271	77	256	256		<FS_TREE>/@/var/crash
272	78	256	256		<FS_TREE>/@/var/cache
273	77	256	256		<FS_TREE>/@/var/tmp
274	77	256	256		<FS_TREE>/@/var/opt
275	77	256	256		<FS_TREE>/@/var/spool
276	84	256	256		<FS_TREE>/@/boot/grub/x86_64-efi

Edit the fstab file at /mnt/@/etc/fstab and add the following entries (change the UUID accordingly)

UUID=1A5D-5465                             /boot/efi              vfat     umask=0077                                                                   0       1
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /                      btrfs    defaults,noatime,compress=zstd,space_cache=v2                                0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /home                  btrfs    defaults,subvol=@/home,noatime,compress=zstd,space_cache=v2                  0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /root                  btrfs    defaults,subvol=@/root,noatime,compress=zstd,space_cache=v2                  0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /opt                   btrfs    defaults,subvol=@/opt,noatime,compress=zstd,space_cache=v2                   0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /srv                   btrfs    defaults,subvol=@/srv,noatime,compress=zstd,space_cache=v2                   0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /tmp                   btrfs    defaults,subvol=@/tmp,noatime,compress=zstd,space_cache=v2                   0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /var/lib/flatpak       btrfs    defaults,subvol=@/var/lib/flatpak,noatime,compress=zstd,space_cache=v2       0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /var/log               btrfs    defaults,subvol=@/var/log,noatime,compress=zstd,space_cache=v2               0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /var/crash             btrfs    defaults,subvol=@/var/crash,noatime,compress=zstd,space_cache=v2             0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /var/cache             btrfs    defaults,subvol=@/var/cache,noatime,compress=zstd,space_cache=v2             0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /var/tmp               btrfs    defaults,subvol=@/var/tmp,noatime,compress=zstd,space_cache=v2               0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /var/opt               btrfs    defaults,subvol=@/var/opt,noatime,compress=zstd,space_cache=v2               0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /var/spool             btrfs    defaults,subvol=@/var/spool,noatime,compress=zstd,space_cache=v2             0       0
UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /boot/grub/x86_64-efi  btrfs    defaults,subvol=@/boot/grub/x86_64-efi,noatime,compress=zstd,space_cache=v2  0       0

Set the default subvolume for your btrfs file system to @

btrfs subvolume set-default /mnt/@

Disable updatedb indexing for snapshots

To disable the indexing of snapshots, simply edit the file /mnt/@/etc/updatedb.conf and add .snapshots to the list of PRUNENAMES (you may need to uncomment it first)

PRUNENAMES=".snapshots"

Restart the system and boot into your new installation.

Install the necessary packages

First install snapper and snapper-gui using apt.

sudo apt install snapper snapper-gui

Then install grub-btrfs from source; the instructions are available in the README of its repository. Make sure you install the required dependencies such as inotify-tools beforehand. (at the time of writing it is not yet available in the repos)

Create snapper config

Run the following command to create a configuration named root for the all the files under our root directory (only stopping at subvolume boundaries)

sudo snapper -c root create-config /

This should create a new subvolume named .snapshots nested under subvolume @ along with its corresponding directory at /.snapshots. Run the following command to see it:

root@kubuntu:~$ btrfs subvolume list -apt /
ID      gen     parent  top level       path
--      ---     ------  ---------       ----
256     231     5       5               <FS_TREE>/@
258     231     256     256             @/home
259     128     256     256             @/root
260     54      256     256             @/opt
261     55      256     256             @/srv
262     230     256     256             @/tmp
269     64      256     256             @/var/lib/flatpak
270     231     256     256             @/var/log
271     213     256     256             @/var/crash
272     231     256     256             @/var/cache
273     214     256     256             @/var/tmp
274     77      256     256             @/var/opt
275     224     256     256             @/var/spool
276     84      256     256             @/boot/grub/x86_64-efi
277     231     256     256             @/.snapshots

A new configuration file at /etc/snapper/configs/root is also created. Please refer to the snapper documentation and change the options to your liking. Make sure you add your username to ALLOW_USERS. I personally reduce the number of timeline snapshots to once a week or turn them off entirely. I also reduce the NUMBER_LIMIT to 10.

Edit /etc/fstab and add a mount point for .snapshots just like the other subvolumes.

UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f  /.snapshots  btrfs  defaults,subvol=@/.snapshots,noatime,compress=zstd,space_cache=v2  0  0

Enable systemd timers and services

Enable this timer to allow snapper to take time-based snapshots based on the configuration you provided.

sudo systemctl enable --now snapper-timeline.timer

Enable this timer to allow snapper to take clean up its snapshots.

sudo systemctl enable --now snapper-cleanup.timer

Enable the grub-btrfs service.

sudo systemctl enable --now grub-btrfsd.service

By default, snapper creates a new snapshot on every boot. You can disable this behaviour using the following command.

 sudo systemctl disable --now snapper-boot.timer

Reboot the system.

10_linux grub configuration file

Open /etc/grub.d/10_linux with a text editor and navigate to the following segment:

1
2
3
4
5
6
7

case x"$GRUB_FS" in
    xbtrfs)
        rootsubvol="`make_system_path_relative_to_its_root /`"
        rootsubvol="${rootsubvol#/}"
        if [ "x${rootsubvol}" != x ]; then
            GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
        fi;;

remove rootflags=subvol=${rootsubvol} from the GRUB_CMDLINE_LINUX string:

1

GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX}"

Do the same thing for /etc/grub.d/20_linux_xen and update grub using:

sudo update-grub

This allows grub to look for kernel and root directory in the correct location after each rollback.

Test out booting from a snapshot

Let’s install a simple package like hello.

sudo apt install hello

Now let’s see the snapshots that snapper has created for this action:

kubuntu@kubuntu:~$ snapper list
 # | Type   | Pre # | Date                              | User | Cleanup | Description | Userdata
---+--------+-------+-----------------------------------+------+---------+-------------+---------
0  | single |       |                                   | root |         | current     |
1  | pre    |       |                                   | root | number  | apt         |
2  | post   |     1 |                                   | root | number  | apt         |

There are two snapshots. One is taken before the action is performed (pre) and one is taken after it (post).

Now reboot. In the boot menu you should see Ubuntu Snapshots as an option. Choose it and select the pre apt snapshot and select a kernel after that. After booting into this read-only system you can see that you don’t have the hello binary anymore and that apt has no record of installing it.

arash@kubuntu:~$ hello
Command 'hello' not found, but can be installed with:
sudo apt install hello              # version 2.10-3, or
sudo apt install hello-traditional  # version 2.10-6

and

arash@kubuntu:~$ sudo apt list --installed | grep hello

returns no result. Issuing snapper list tells you which snapshot you are currently booted into using a - next to the snapshot number.

Please note that this is a read-only snapshot, and you cannot change the contents of the snapshot (for example trying to install anything with apt would fail). You can however change the contents of other subvolumes such as home. This is why we chose to put the contents of some subdirectories of /var such as /var/log under separate subvolumes so that the system would be able to boot.

Reboot into the default entry of grub (your original system).

Rollbacks

In case the system gets broken, we want to be able to boot into a working read-only snapshot, issue a rollback command, and reboot into a normal writable snapshot by default. OpenSUSE achieves this by patching the grub2 source code and adding extra functionalities that allow grub to boot into rollback snapshots directly and by default. (search for btrfs in source files and you will see the patches)

BTRFS default subvolume

From the btrfs docs:

A freshly created filesystem is also a subvolume, called top-level, internally has an id 5. This subvolume cannot be removed or replaced by another subvolume. This is also the subvolume that will be mounted by default, unless the default subvolume has been changed (see btrfs subvolume set-default).

This means that when we changed the default subvolume to @ in the previous sections and did not specify subvol= in the mount options of / in our fstab file, the default subvolume of the btrfs system would be mounted at / (in our case @). Snapper uses this functionality to roll back to a snapshot. It changes the default subvolume of the btrfs filesystem to the snapshot we want to roll back into and the OS using the fstab file that we provided mounts that default subvolume unto /.

Early grub configuration file

Normally, when your system is turned on, the EFI firmware calls the grub bootloader, grub reads the initial grub.cfg configuration file that is stored at /boot/efi/EFI/ubuntu/grub.cfg and that configuration file calls into the main grub.cfg stored in /boot/grub/grub.cfg that is generated by update-grub and in turn by grub-mkconfig.

Let’s compare the file early grub configuration file that Ubuntu uses to that of OpenSUSE with their patched grub2 package. Ubuntu’s configuration arash@kubuntu:~# cat /boot/efi/EFI/ubuntu/grub.cfg:

1
2
3

search.fs_uuid 84cc93cb-d382-45f8-a0f0-79a0316ba88f root
set prefix=($root)'/@/boot/grub'
configfile $prefix/grub.cfg

OpenSUSE’s configuration arash@opensuse ~# cat /boot/efi/EFI/opensuse/grub.cfg:

1
2
3
4

set btrfs_relative_path="yes"
search --fs-uuid --set=root 512bb31b-c5a9-468b-a549-22ad3b4e99fb
set prefix=(${root})/boot/grub2
source "${prefix}/grub.cfg"

as you can see the grub2 package of OpenSUSE supports the btrfs_relative_path option. This allows grub to choose the /boot/grub/grub.cfg file that resides in the default subvolume of the btrfs filesystem (in our case the snapshot that we have rolled back into). Ubuntu on the other hand, has a static path for /boot/grub/grub.cfg under the @ subvolume.

To fix this issue, we install a plugin for snapper that changes line 2 in /boot/efi/EFI/ubuntu/grub.cfg to the snapshot that we want to roll back into, for example:

1

set prefix=($root)'/@/.snapshots/5/snapshot/boot/grub'

Snapper EFI plugin

To achieve the behaviour described in the previous section, create a file named 99_efi in /usr/lib/snapper/plugins and add the following script into it and make it executable using chmod +x 99_efi.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/sh

set -e

distro=$(awk -F "=" '/^NAME=/ {gsub(/"/, "", $2); print tolower($2)}' /etc/os-release)

case $1 in
        rollback*)
                sed -i "s/^set prefix=(\$root).*/set prefix=(\$root)\'\/@\/\.snapshots\/$5\/snapshot\/boot\/grub\'/" /boot/efi/EFI/$distro/grub.cfg
esac

Every time snapper performs an action it calls all the plugins it has with the name of the action as its argument. This script changes the entry in /boot/efi/EFI/ubuntu/grub.cfg so that it points to the location of the new default snapshot that snapper rollback has set.

Perform the first rollback

Using snapper list and snapper remove <list of snapshot numbers> delete all the previous snapshots. Install cowsay using apt, reboot the system and boot into the pre snapshot (where cowsay is not installed).

Perform the very first rollback using:

arash@kubuntu:~$ sudo snapper --ambit classic rollback
Ambit is classic.
Creating read-only snapshot of default subvolume. (Snapshot 3).
Creating read-write snapshot of current subvolume. (Snapshot 4).
Setting default subvolume to snapshot 4.

The contents of /boot/efi/EFI/ubuntu/grub.cfg should also indicate that grub will use the configuration file in snapshot 4.

1
2
3

...
set prefix=($root)'/@/.snapshots/4/snapshot/boot/grub'
...

Now reboot the system. The default grub entry should boot you into snapshot 4. You can confirm this using the following commands:

• First make sure that cowsay is indeed not available anymore in this system.
• Run sudo btrfs snapshot show / and confirm that the subvolume mounted at / is the snapshot we rolled back into.

Please note that --ambit classic is only needed on the first ever rollback. After that you can roll back using:

sudo snapper rollback

update-grub after each rollback

In your new rolled back system, check the contents of /boot/grub/grub.cfg. You can see that grub still invokes the kernel from the previous default subvolume of the btrfs filesystem

...
linux   /@/boot/vmlinuz-6.2.0-27-generic root=UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f ro  quiet splash $vt_handoff
initrd  /@/boot/initrd.img-6.2.0-27-generic
...

To fix this, after you boot into your rolled back system for the first time, you must update grub using

sudo update-grub

This makes sure that the default entry in grub points to the kernel inside the rolled back snapshot instead of the kernel in previous system. Now /boot/grub/grub.cfg points to the correct kernel:

linux   /@/.snapshots/4/snapshot/boot/vmlinuz-6.2.0-27-generic root=UUID=84cc93cb-d382-45f8-a0f0-79a0316ba88f ro  quiet splash $vt_handoff
initrd  /@/.snapshots/4/snapshot/boot/initrd.img-6.2.0-27-generic

Delete the contents of the original root subvolume @

Since from now on you will always boot into a snapshot subvolume, you won’t be needing the contents of @. First mount it at /mnt:

sudo mount /dev/vda2 /mnt -o subvol=@

and then delete all the files (make sure you only delete the files inside @ and not the files of other subvolumes nested inside @).

1
2
3

rm -rf /mnt/{usr,etc,lib,lib32,lib64,libx32,bin,sbin,cdrom,media,mnt}
ls -d /mnt/var/lib/* | grep -v "flatpak" | xargs rm -r
ls -d /mnt/boot/* | grep -v 'boot/\(efi\|grub\)' | xargs rm -r

Conclusion

You can now enjoy your rock solid system. If you happen to mess up system configuration or an update bricks your system, you can easily boot into a working snapshot and issue a rollback command from there. I find this method a suitable alternative for immutable distros for people who may not want to fully commit to them.

Snapper also includes a lot of other features such as undoing changes and seeing the difference between snapshots. You can check out the documentation of snapper for more info.

Resources

The following resource helped with the creation of this guide:

• An Arch Linux Installation on a Btrfs Filesystem with Snapper for System Snapshots and Rollbacks
• How to Install Fedora 38 with Snapshot and Rollback Support

I start with a quick history of AI and how after decades from its inception, the initial goal of building a system with human capabilities has slowly transitioned towards applied AI platforms that the market demands. An overview of the differences between biological neurons and those used in today’s machine learning algorithms is then presented. After that, I go over what consciousness is and how it is related to feelings, homeostasis, self-organizing systems, the Markov blanket, and the free energy principle. I then go over the possibility of building a conscious AI using these physical models. In the last part, I talk about how organismic agents differ from AI agents. I explain what affordances are and how organisms can achieve their goals through jury-rigging. I end the talk with a discussion of whether true AGI is even possible in the current algorithmic frame of development and explain what this may entail regarding simulating evolution and various deep learning models.

Link to the Video In Persian: YouTube

Slides available here

Thumbnail background from: Patheon (TV series)

Asynchronous programming allows the development of services that can handle millions of requests without saturating memory and CPU utilization. Support for asynchrony is usually baked into the programming language; we take a look at async support in Rust, a type-safe and memory-safe systems programming language that guarantees safety at compile time using rules that eliminate many issues prevalent in traditional languages. We take a peek at the inner workings of Tokio, an asynchronous runtime for Rust that provides scheduling, networking, and many other primitive operations for managing asynchronous tasks. Runtime libraries rely on the operating system for registering and dispatching asynchronous events. We compare the different asynchronous APIs that Linux provides and conclude why the new io_uring interface is the best approach. This article was originally written in 2022-09.

Introduction

The need for asynchronous programming grew when traditional multithreaded programs failed to meet the demands of handling a huge number of requests without upgrading the hardware. Traditional threads provide their own call stack and are periodically placed on a core as part of a context switch by the operating system, to advance their execution. Because of these memory requirements, spawning a thread for handling every request becomes infeasible in terms of memory usage when the number of requests grows. Supporting asynchronous programming on a language level needs constructs built into the language as well as a runtime library for managing asynchronous green threads (tasks) and scheduling them on CPU cores. This runtime is also responsible for interfacing with the operating system and dispatching resources associated with each event that the OS emits.

Rust is a programming language that first had its 1.0 release back in 2014. This systems programming language promises safety and efficiency for programs that serve all kinds of functionality e.g. web servers, operating systems, GUI and CLI programs, machine learning models, game engines and etc. Rust enforces some rules at compile time that allow the programmer to write memory and type-safe code without the need for excessive testing. Even with testing, traditional programming languages like C/C++ still fail to create memory-safe programs, even if these programs were written by the best programmers in the world. Some examples of memory issues that lead to dangerous vulnerabilities in software include buffer and stack overflows, use-after-free, out-of-bounds indexing, double freeing and etc. Rust tries to solve all these problems at compile time while providing a modern and standard syntax for programmers.

Asynchronous support is built into the Rust language. Rust uses the concept of Futures (a value that might be available in the future) and state machines to create resumable green threads that use a cooperative strategy to run themselves to completion. There can be other futures inside a future leading to the creation of a tree-like structure where the futures at the root, called a task, are directly managed by the runtime, and as the runtime polls the task for advancement, the task then polls its inner futures to completion. There leaf futures of this tree are futures that communicate with the I/O driver and the operating system. They register resources with the OS and wait for the OS to emit events, stating the status of those resources.

Tokio is an asynchronous runtime for the Rust programming language. It provides a scheduler, an I/O and time driver, and other functionalities like networking, inter-task communications, and synchronization primitives. Tokio provides a multithreaded scheduler that creates multiple workers on different threads. These workers each have a run queue that tasks reside in; they then pop tasks from their run queue and try to advance them to completion. The I/O driver uses an abstraction over different operating systems for registering events. It then waits for events to come from the OS and iterates through all of them, dispatching tasks that were blocked on the resource associated with each event.

Linux provides different system calls for working with asynchronous operations, these system calls tell the called about the readiness of file descriptors it was given. The problem with these system calls is that they don’t work as expected for file operations and always return a read status for file descriptors backed by regular files. For this reason, io_uring was introduced back in 2019 in the Linux kernel. It provides a uniform asynchronous interface at programs can use to manage all their asynchronous operations.

The Rust Programming Language

Rust is a new programming language made to create safe and efficient systems by modern standards. The thing that makes rust safe is a set of rules that rust enforces on the programmer during the compilation time. These rules are designed to catch memory bugs and race conditions in multi-threaded programs. Rust does all these while still being as efficient as the C programming language. A huge portion of software vulnerabilities is because of memory safety violations, things such as buffer overflow, use after free, and index out-of-bounds. There have been many attempts at trying to reduce these memory violations by statically checking C/C++ code but were mostly in vain. Today more than 70% of software vulnerabilities are because of memory safety issues [2]. For this reason, companies like Google and Mozilla started to come up with languages such as Go and Rust respectively. These new languages aim to be as efficient as C/C++ but without all their memory issues. Rust is different from Go in terms of how they manage their memory; Go is garbage collected whereas Rust lacks one and most values are simply dropped at the end of the scope.

Rust has become the most loved language every year since 2016 in Stackoverflow surveys [16] and has seen a significant adaptation among companies and individual developers. For example, in the embedded area companies like STMicrocontrollers and Espressif Systems introduced their community projects for enabling the use of the Rust programming language on their SoCs and modules.

The Rust programming language was first conceived back in 2009 in Mozilla and had its 1.0 release in 2014. Rust achieves its safety using two new concepts: ownership and lifetimes which we will be talking about shortly. The main drawback of using Rust is its steep learning curve. Like languages such as Haskell or Prolog, Rust introduces new concepts and new semantics that have a striking difference from what most system programmers are accustomed to. When starting to learn Rust, most of the time will be spent clashing with the compiler and trying to get the code to run.

Overview

Rust inherits from both object-oriented and functional paradigms. It supports generics, enums, pattern matching, and modules and has support for polymorphism using traits; traits are like interfaces in Java and allow types that implement them to be used as an instance of that trait or perform constraints on other types. Variables are by default immutable and can only be mutated if explicitly marked with the mut keyword.

One of the most loved features of Rust is its easy-to-use build system and package manager called Cargo. Cargo can download packages, resolve dependencies, build projects for different targets and so much more. Rust has a packages registry with more than 50,000 packages (or crates as we call them in Rust).

Ownership

For Rust to not use a garbage collector it needs to know when it is ok to free values; it does this using the concept of ownership. Ownership has three rules which are enforced by the compiler:

• Values in Rust only have one owner.
• There can only be one owner at a time for a value.
• When the owner goes out of scope the value is dropped. In listing 1 we can see an example. At line \( 11 \) a new scope is started using a curly bracket; this scope ends at line \( 15 \) with a closing bracket. At line \( 12 \) a new variable called \(x\) is created and assigned a value of \( 22 \). After its scope is ended at line \( 15 \) the \( x \) is dropped. If we try to use the value of \( x \) at line \( 17 \) we hit a compile error.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

fn total(numbers Vec<i32>) -> i32 {
    let mut sum = 0; // create a mutable variable

    for num in numbers {
        sum = sum + num
    }

    sum
}
fn main() {
    {
        let x = 22;

        println!("x: {}", x);
    }

    // println!("x: {}", x); //! compile error

    let a = vec![1, 2, 3]; // a heap allocated array
    let b = a;             // move ownership of array from 'a' to 'b'

    println!("b: {:?}", b); // This is correct
    // println!("a: {:?}", a); //! compile error

    let t = total(b); // b is moved

    // println!("numbers {}", b); //! compile error
}

Listing 1: Ownership

At line \( 19 \) a heap-allocated array (a vector) is created and is assigned to an immutable variable called \( a \). The ownership of the vector is then passed from \( a \) to another variable called \( b \). After this assignment, we cannot use \( a \) anymore and the print statement at line \( 23 \) would fail at compile time.

\( b \) is then passed to a function that takes a vector as its single argument. In this case, the ownership of our vector is passed into the function and is given to the variable \( numbers \). At the end of the function, the vector is freed because its owner which is \( number \) goes out of scope. Thus, using \( b \) at line \( 27 \) is a compile error. [17](Chapter~4.1)

Borrowing

If you wanted to use a variable without taking its ownership, Rust provides a concept called Borrowing. Borrows are in a sense references or statically checked pointers that allow the use of a non-owned variable as long as only one of the following conditions hold:

• There is only one mutable reference to a variable
• There are any number of immutable references to a variable In listing 2 a vector is created and is assigned to a variable called \( nums \). At lines \( 4 \) and \( 5 \) two immutable references are created. At line \( 6 \) pushing a new value into \( nums \) fails because using an associated function such as push takes a mutable reference to that variable and thus this violates the borrowing conditions: There cannot be a mutable reference to a variable along with other immutable references. The statement in line \( 7 \) also fails for the same reason. At line \( 11 \) after the references are out of scope, we are allowed to take mutable references to variables.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

fn main() {
    let mut nums = vec![1, 2, 3];
    {
        let nums1 = &nums; // first reference
        let nums2 = &nums; // second reference
        // nums.push(4); // ! compile error
        // let mut nums3 = &mut nums; // ! compile error
        // nums3.push(4); // ! compile error
        println!("ref1: {:?}, ref2: {:?}", nums1, nums2);
    }
    nums.push(5);

    let mut nums4 = &mut nums;
    nums4.push(6);
    // nums.push(7); ! compile error

    println!("{:?}", nums4);
}

Listing 2: Borrowing

At line \( 13 \) a new mutable reference is created. Since there can only be one mutable reference to a variable at a time (same as above push takes a mutable reference from the associated variable). An important observation is that if we removed the statement at line \( 17 \) then the push operation at line \( 15 \) wouldn’t fail; this is because the rust compiler is smart enough to notice that the mutable reference \( num4 \) is not used anymore after the push operation and is immediately freed after the push.

These rules are enforced by the Rust compiler and prevent memory-safety issues described before such as freeing more than once, use after free, and data races. These rules, however, hinder programmers from easily creating data structures such as linked lists or interface with hardware and are forced to use unsafe blocks. In these blocks, Rust ownership and borrowing rules don’t apply.[17](Chapter~4.2)

Lifetimes

Rust avoids situations where a resource is used after it has been freed in other places using lifetimes. Lifetimes and lifetime annotations allow the compiler to understand the scope in which a variable is valid. The Rust compiler can infer the lifetime of variables in most cases and manual annotation is not needed. For example, the lifetime of \( nums2 \) and \( nums1 \) in listing 2 ends at line \( 10 \).

An example where manual lifetime annotations are needed is described in listing 3. In this example, the \( smaller \) function takes two references and returns a reference. Without lifetimes annotations we can’t make the compiler ensure that when the program runs it avoids cases where the reference that the function returns has been freed beforehand and thus is an invalid reference. Lifetime annotations are like types; sometimes you have to manually specify the type of a variable because the compiler cannot infer it. In our case, the lifetime annotations define the relationship between the inputs and the output that the programmer desires, and then the compiler can analyze it [17](Chapter~10.3).

The lifetime annotation \( `a \) tells the compiler that the return value should live as long as the smaller lifetime of the two input references.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

fn smaller<'a>(str1: &'a str, str2: &'a str) -> &'a str {
    if str1.bytes().len() < str2.bytes().len() {
        str1
    } else {
        str2
    }
}

fn main() {
    let name1 = String::from("Arash");
    let name2 = String::from("Kiarash");

    println!("{}", smaller(&name1, &name2));
}

Listing 3: Lifetimes

Concurrency

Rust does not provide a runtime library to manage and schedule threads but instead uses a one-to-one mapping model where all its threads are mapped to OS threads. Rust supports synchronization primitives such as mutex, conditional variable, atomic operations, barriers and etc. Rust uses lifetimes to decide whether a resource should be unlocked; if a resource that is behind a lock goes out of scope the lock is unlocked. One can send ownership of variables between threads through message passing mechanisms such as channels. It should be noted that Rust cannot stop the programmer from creating deadlocks using primitive synchronizations for example [17](Chapter 16.1).

Conclusion

In this section, we introduced Rust, a relatively young systems programming language that provides comparable efficiency and safety guarantees missing from other languages such as C/C++. We outlined the reasons that lead to its creation at Mozilla and discussed the mechanisms and rules by which the Rust compiler can eliminate memory-safety and type-safety issues plaguing the traditional programming languages.

Asynchronous Programming

In this chapter, I am going to discuss multitasking with which operating systems are able to run multiple tasks at the same time. We explore preemptive and cooperative multitasking and take a peek at how Rust achieves asynchrony.

Multitasking

Outside of the embedded space and in mainstream servers or personal computers, being able to execute multiple tasks at the same time is a must. For example on a physical server or VM, multiple processes could be running at the same time: one for a mail server, the other for serving a webpage, another routing traffic through a virtual private network, etc.

In most cases, there are more tasks than there are CPU cores and thus for the operating system to be able to run all these tasks at the same time, it needs to schedule more than one task on a single core. In order to create a feeling that all these tasks are running at the same time, the OS periodically switches out running tasks for those that are waiting to have their time slice. This is concurrency, an illusion that tasks are running at the same time. Parallelism is when tasks are actually executing at the same time on different cores.

There are two main ways to achieve multitasking:

• The OS forcefully pauses the execution of a task and switches it with another waiting one.
• The tasks themselves give up control of the CPU (yield) so that other tasks can execute.

Preemptive Scheduling and Multithreading

In preemptive scheduling, the operating system can decide when to switch out a task based on a hardware interrupt. This interrupt could be the result of a hardware timer timeout or other sources of interrupts. In figure 1 we can see that while the CPU is executing task \( A \) a hardware interrupt occurs and the operating system takes control of the CPU to execute the respecting interrupt handler and switch out task \( A \) with task \( B \) [11].

These tasks in most operating systems are called threads of execution and each have a separate call stack reserved by the operating system. For threads to be resumed later in time, the CPU needs to save the thread’s state when it was preempted. This requires saving all the CPU registers at the point of preemption and restoring them when the thread resumes. These threads are multiplexed by the operating system and threads ready to be executed are given CPU cycles. If a thread blocks for some operation or its time slice has ended, the OS changes that thread for another ready thread.

Using threads to read from sockets or write to files are fine for a lot of applications. They have the advantage that their execution is controlled by the operating system and CPU time is shared fairly between them. Also in environments where tasks could be spawned from untrusted applications, it is important for the OS to be able to halt their execution. Their problem is that each thread needs its own call stack and thus, the more threads are spawned the higher the memory usage gets. When switching between threads, there is an overhead in saving the registers and restoring them again for the thread to continue. This back and forth between the thread on the operating system level is costly when applications need high performance.

If you use blocking interfaces for doing I/O, e.g. wait until \( x \) amount of bytes is read from this socket, you would need a separate thread for each call; one thread to read from a socket, another to fetch some records from a database, and so on.

In this method, the application uses the OS APIs to create user threads that map to kernel threads. This way the application does not need to have the facilities for managing threads and the binary sizes are smaller.

Cooperative Scheduling and Asynchrony

In Cooperative scheduling, tasks themselves yield execution to allow other tasks to run on the CPU. This is different from preemptive scheduling where the OS had to pause the execution of a task by force [11]. Cooperation is usually done with the help of the language and compiler and a runtime library. This runtime could either be embedded into the language itself or in the case of Rust, be included as a dependency.

Cooperative scheduling is useful for situations where there is a lot of I/O happening and if threads were used for each blocking operation, most of the time of that thread would simply be spent waiting for that operation (waiting for bytes to arrive from a socket or reading a file from disk). In asynchronous operations, however, the calling thread does not wait for the operation to be available and simply goes to do other things. When the resource associated with the asynchronous operation is ready, it notifies the thread that it can start executing the codes after that asynchronous operation.

In this type of scheduling, tasks are more fine-grained than threads and multiple tasks can be assigned to run on a single thread. For example, we can create a task for listening on a socket \( TaskA \) and another one for saving some information into a database \( TaskB \). Both of these tasks can be run on the same thread. When \( TaskA \) tries to read from the socket and no data is available, it saves its state and yields. After that, since our thread is now not doing anything it can try to execute \( TaskB \); \( TaskB \) executes and tries to write as many bytes as possible and then saves its state yields to the runtime. Now our thread goes to sleep and waits for either of the tasks to be available for execution. If there are new bytes available at the socket, for example, the operating system notifies the runtime and the runtime in turn wakes up \( TaskA \) and executes it on our thread. In figure 2 we can see an example of this scenario.

Saving the state is usually done by the language. In Rust, all the state needed for a task, for example, local variables or some parts of the call stack, is stored in a struct. The code for creating this struct and inserting the values into them is automatically generated by the compiler. In this way, the tasks don’t need a separate call stack and use a shared one (that of the thread that they are running on). Now we can generate millions of tasks without having to worry about memory limitations.

This method uses a kind of many-to-one mapping where multiple user tasks are mapped onto a user thread which themselves are mapped to a kernel thread. Its disadvantage is that a rouge task that does not yield, can starve all the other tasks waiting to be executed. Because of this, cooperative scheduling is usually used in environments where tasks can be trusted e.g. inside a project where tasks are created by the developers themselves.

Comparison

Let’s check out an example that outlines the difference between multithreading and asynchrony. In listing listing 4 we can see a simple TCP server that listens on a port for new connections. Each time a connection arrives from a client, it creates a new socket for the client and passes that connection to a new thread. The handle_connection function is used to manage a single connection on a separate thread.

Depending on the memory of the machine that this server is run on, this approach is probably fine for thousands of connections; but when the number of connections gets to millions the memory requirements of the server process become infeasible.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

fn main() {
    let listener = TcpListener::bind("127.0.0.1:8090").unwrap();

    for socket_wrapped in listener.incoming() {
        let socket = socket_wrapped.unwrap();

        thread::spawn(|| {
            handle_connection(socket);
        });
    }
}

Listing 4: Threaded TCP

We can use asynchronous tasks in Rust to handle connections concurrently. In this way, a single thread can handle multiple tasks and the maximum number of connections can go up to millions (of course if we ignore other limitations). The code in listing 5 shows an example of using asynchronous tasks in Rust using the Tokio runtime. As you can see, asynchronous code looks a lot like normal synchronous code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("127.0.0.1:8090").await.unwrap();

    loop {
        let (mut socket, _) = listener.accept().await.unwrap();
        tokio::spawn(async move {
            handle_connection(socket);
        });
    }
}

Listing 5: Async TCP

Futures and State Machines

Rust has support for cooperative tasks using futures and async/await. A future is a value that will be available at some time later. Instead of blocking for the value to be available, the caller can advance to do other things and only come back when the future is resolved and the value is ready. The cooperative tasks in Rust are a kind of future and futures are structs that implement a trait (interface) called \( Future \) (listing 6). Types that implement the \( Future \) trait must implement a method called \( poll \) and specify the value that the future will resolve to. For example, you can call a future that returns the number of open file descriptors of the current process; when this future resolves it returns an integer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

pub trait Future {
    type Output;
    fn poll(self: Pin<&mut Self>, cx: &mut Context)
                             -> Poll<Self::Output>;
}

pub enum Poll<T> {
    Ready(T),
    Pending,
}

Listing 6: Future Trait

The \( poll \) method’s output type is an enum called \( Poll \) that has two items. Calling the \( poll \) method tries to run the future to completion; if the future completes it returns Ready(T) with the value inside it and if the value is still not ready it returns Pending.

Asynchronous functions in Rust return a structure that implements the \( Future \) trait and its \( poll \) method. If this future resolves, the type of the value in Ready(T) is the same as the return type of the \( async \) function. This is done by the compiler and allows the programmer to write asynchronous code that looks similar to normal synchronous code. We can use the \( await \) keyword inside an \( async \) function to wait for the value of another future.

The Rust compiler converts \( async \) functions into a state machine (a struct) that implements the \( Future \) trait. This state machine is the future that is returned when an \( async \) function is called. Listing 7 shows an example of such function. At line \( 4 \) we await reading all the contents of a file into a byte array; when reaching this statement while executing the task, since reading from disk is much slower than the speed of our processor, the task returns immediately return Poll::Pending to allow other tasks to proceed. read_all_file is another \( async \) function and calling it returns a future; \( await \)ing on that future stops the outer future (our \( async \) function) from going further and causes the outer function to return [13] [15] [14].

After the file has been read into the file_bytes buffer, the pipe_to_socket future is polled again it resumes from where it left off, after \( 6 \) and goes to \( await \) another future which tries to create a UDP socket. This goes on until the whole function is executed; each time the function hits an \( await \) point, it \( yields \) to the runtime so that other tasks can advance. The state of the function is saved across \( await \) points. For example, when waiting to receive bytes from the socket, the file_bytes and the socket variables are saved inside the compiler-generated struct and are used after the future (our function) is polled again to resume.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

async fn pipe_to_socket() -> i32 {

    // Read all the contents of a file
    let file_bytes = read_all_file("myFile.txt").await;

    // Create a UDP socket
    let socket = UdpSocket::bind("0.0.0.0:8080").await;

    // Receive some data from a client
    let (len, addr) = socket.recv_from(&mut [0; 1024]).await;

    // Send all the contents of the file
    write_all_socket(socket, addr, file_bytes).await;

    return len;
}

Listing 7: Async Function Example

This yielding and resuming later creates a state machine. The state machine for the \( async \) function in listing 7 is shown in figure 3.

As you can see, each time an \( await \) point is hit while executing a future, the control is yielded back to the runtime to execute other tasks. After an event occurs that the future is waiting on, the runtime finds the future waiting on that event and tries to poll it again so the future can progress. Each time a future returns to waiting on a resource, the runtime registers that resource to for example an operating system. The runtime then listens for events. These events come from the operating system signaling the runtime that some resource that has been registered with it, is now ready [14].

An example of what the compiler-generated state machine could look like is outlined in listing 8 (note that the code might not be semantically correct but nevertheless, it serves as a good example of how futures behave internally). The type of our state machine could be a simple enum with 6 different enumerations. Each enumeration is a state that contains all the necessary data needed to pause and resume the state machine. We implement the \( Future \) trait for this type and add a poll method for it. Each time the poll method is called, it tries to advance the state machine. It first determines which state it is in using pattern matching and then runs the code generated for that state.

For example, if the state machine was on State1 and it was polled by the runtime, first it would extract the variables stored in the current state, in the case of State1 the only variable is a future. This future is the result of calling the \( async \) function read_all_file when transitioning from State0 to State1. It would then try to poll the read_all_file future, if the future was not ready and returned \( Pending \) then the result of our outer future would be \( Pending \). However, if polling read_all_file reloved to a Poll:Ready(file_bytes) then we can transition to the next state State2 by first creating an instance of State2 and passing in the required variables, naming our byte array and the result of calling UdpSocket::bind which is a future. These variables are part of the state that our stat machine carries.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

enum PipeToSocketStateMachine {
    Start(),
    State1 { read_all_file: impl Future<Output = Vec<u8>> },
    State2 { file_bytes: Vec<u8>,
        bind: impl Future<Output = UdpSocket> },
    State3 { file_bytes: Vec<u8>,
        socket: UdpSocket,
        recv_from: impl Future<Output = (usize, SocketAddr)> },
    State4 { file_bytes: Vec<u8>,
        socket: UdpSocket,
        len: usize,
        addr: SocketAddr,
        write_all_socket: impl Future<Output = ()> },
    Return(),
}

impl Future for PipeToSocketStateMachine {
    type Output = i32;

    fn poll(self: Pin<&mut Self>,
            cx: &mut Context) -> Poll<Self::Output> {
        loop {
            match self {
                PipeToSocketStateMachine::
                                Start() => {…}
                PipeToSocketStateMachine::
                                State1{read_all_file} => {
                    match read_all_file.poll(cx) {
                        Poll::Ready(file_bytes) => {
                            // Proceed to the next state
                            *self = PipeToSocketStateMachine::State2 {
                                file_bytes,
                                bind: UdpSocket::bind("0.0.0.0:8080")
                            }
                        },
                        Poll::Pending => {
                            return Poll::Pending
                        }
                    }
                }
                PipeToSocketStateMachine::
                                State2{file_bytes, bind} => {…}
                PipeToSocketStateMachine::
                                State3{file_bytes, socket, recv_from}
                                                               => {…}
                PipeToSocketStateMachine::
                                State4{file_bytes, socket, len,
                                       addr, write_all_socket} => {…}
                PipeToSocketStateMachine::
                                Return(state) => {…}
            }
        }
    }
}

Listing 7: Generated State Machine

This state machine generation is done by generators. A generator is a “resumable function” that syntactically resembles a closure but compiles to much different semantics in the compiler itself [18]. The Rust compiler inserts \( yield \) operation in place of \( await \)s so that when the result of an \( await \) is Pending then the future pauses and yields; so each yield is in fact a state of our state machine. The Pin type in listing 6 makes sure that the wrapping type does not move around in memory; this is important in the case of self-referential structs where if the object is moved, its internal pointers to itself will be invalidated. The state machine that the compiler generates has these self-referential pointers and thus these futures must be pinned [14].

Executors

When a future is polled and it returns Pending something has to poll them again when they are able to make progress, otherwise our state machine won’t advance. In the previous sections, we said that the “runtime” does the polling and that’s true; the part of the runtime that polls futures to completion is called the executor. Listing 9 shows a very basic and primitive executor. It has a queue where futures are pushed into as tasks and it repeatedly tries to pop a task from the start of the queue and run it. Tasks are the futures that the executor works with; an example of a task is outlined in listing 10. Tasks are structs that usually hold an instance of a future in them, when we say “run a task” we mean to poll the future inside that task [15].

1
2
3

while let Some(task) = self.queue.pop() {
    task.run();
}

Listing 9: Executor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

pub struct Task {
    ...
    future: Pin<Box<dyn Future<Output = ()>>>,
    ...
}

impl Task {
    fn run(&mut self) {
        ...
        match self.future.poll() {
            Poll::Pending => {...},
            Poll::Ready(val) => {...}
        }
        ...
    }
}

Listing 10: Task

In listing 9 when a task is popped and its future is polled, two things can happen, either the task’s future returns Poll::Ready(value) in which case there is nothing else to do, since the task has finished to completion, or it returns Poll:Pending indicating that this task has to be polled on a later time (for example it might be waiting on a socket or a read from the disk). For a task to be able to be run again, it needs to be added into the run queue of the executor; the question that remains is who does that?

Wakers

Wakers are the mechanism that Rust uses to signal the executor that a given task should be run (its inner future be polled). Wakers have a wake() method that when called, does this signaling. In the context of futures, wakers are usually wrapped in a Context type and are passed to the poll method of that future; listing 11 shows an example of a Context struct. For our basic executor calling wake() on the waker of the context of a task, would simply add that task back to the executor run queue so that it can be polled once again. If the executor was asleep, it would also wake up the executor [15].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

pub struct Waker {
    ...
}

pub struct Context<'a> {
    waker: &'a Waker,
}

impl Waker {
    pub fn wake(self) {
        ...
    } 
}

Listing 11: Context

Future Tree

The futures that we have seen so far all call other futures. In line 4 of listing 7 for example, the async pipe_to_socket function waits for the output of the future that calling read_all_file results; an example of what this function could look like is shown in listing 12. The same thing happens throughout the function. The futures that we are awaiting on themselves can await on other futures. This structure forms a tree [3] [12].

1
2
3
4
5
6
7
8

async fn read_all_file(path: &str) -> Vec<u8> {
    let file =  File::open(path).await;

    let mut buffer = Vec::new();
    file.read_to_end(&mut buffer).await;

    return buffer;
}

Listing 12: Nested Async Function Example

Figure 4 shows this tree for the example in listing 12 and 7.

As we can see tasks are the root of our future tree and are the futures that our executor is in contact with. Executors provide APIs for spawning new tasks by giving them a future. In listing 13 we call our async pipe_to_socket function which returns a future, and pass this future to the spawn method of our executor (in this case Tokio’s executor). The spawn method creates a task out of this future and adds it to the run queue of our executor.

1
2
3
4
5
6
7

#[tokio::main]
async fn main() {

    ...
    tokio::spawn(pipe_to_socket());
    ...
}

Listing 13: Root Future

As we saw in line 28 of our state machine in listing 8 when we call poll on our main task’s future, it in turn, tries to poll its inner futures. But we can’t keep polling other futures, somewhere in this tree there are futures that when they are polled, other than returning Poll:Pending they also make sure that the task that polled them will be run again when the resource its waiting on is available. These are the futures at the leaf nodes of our future tree. In figure 4 when we poll our task for the first time and give it a waker, which when its wake() method is called adds this task back to the run queue, it tries to poll the read_all_file future. The read_all_file future in turn tries to poll the open(path) future. The open(path) future is a leaf future so when it is polled, instead of polling another future it registers its waker and the resource it’s waiting on with the driver (also called the reactor). In this case, the future asks the driver to open a file with the given path and the driver in turn asks the operating system to open the given file. The waker registered with the driver is passed down from the main task and the driver keeps a record of which wakers are related to which waiting resources. The leaf future then returns Poll:Pending and this return value is propagated back to the task and back to the executor. The executor then tries polling other tasks.

The driver periodically waits for the operating system to return a set of events. It then iterates through the events and dispatches all the wakers that have been registered with the given event’s resource. In our case, when the file has been opened by the operating system, it returns an event. After seeing this event, the driver calls the wake() method on the waker registered with this event, and the task related to this waker is pushed back into the run queue to be polled again. When the task is polled again, since all of our futures are state machines and the future tree is somewhat a state machine tree, it starts polling from the root future all the way back down to the leaf future. The futures saved their state from the last time they were polled and now they resume from where they left off. This time when the leaf future is polled, it receives the resource it was waiting on (in our example, the file descriptor of the opened file) and returns Poll::Ready(fd) to its parent future. The parent future then goes on to poll the next future and so on.

Conclusion

In this section, we discussed different ways that multitasking could be done. We outlined the advantages and disadvantages of preemptive and cooperative scheduling and talked about how asynchrony helps us utilize our memory and CPU cycles. We saw how rust implements asynchronous programming through the use of aync/await keywords and futures. Finally, we saw an example of how the Rust compiler helps us to create state machines from these futures and how executors and runtime libraries run these futures.

Tokio

Tokio is an asynchronous runtime for Rust, it provides the executor we talked about in the previous section that allows running futures. It is by far the most widely used runtime in the Rust ecosystem and many other projects are built on top of Tokio. Tokio provides a whole host of functionalities:

• Task scheduler
• An I/O driver (reactor) for interfacing with the operating system
• Asynchronous network interfaces such as sockets
• Communication and synchronization primitives for working with asynchronous tasks such as different kinds of channels, mutex, barrier, condvar, and …

Tokio is broken into modules each providing different functionalities. Here we will be taking a closer look at the runtime module. This module does not expose a lot of APIs to the user of the library; most of them are for creating a runtime or spawning tasks unto the runtime. The runtime module is where the magic that allows writing high-performance asynchronous code in Rust happens and it itself consists of multiple parts [19].

• A scheduler for assigning tasks to be run on the available cores
• The I/O driver that wakes up waiting tasks when new events come form the operating system,
• A time driver with a resolution of one millisecond used for time-based scheduling

In the remainder of this chapter, we will be taking a closer look at the inner workings of the scheduler and the I/O driver of the Tokio runtime.

Scheduler

The scheduler has the job of assigning tasks to the CPU cores until they yield back to the scheduler. The future inside the task is polled and it runs on the CPU until it hits an inner future that needs an external resource to be resolved. Tokio provides two kinds of schedulers: Multithreaded and Single-threaded. Here we will be taking a look at the multithreaded scheduler (the inner workings of the single-threaded one is similar).

Multithreaded scheduler

When using the multithreaded scheduler (the default) Tokio creates a number of threads to scheduler tasks on. It then assigns a local run queue to each thread; threads then enter a loop where they constantly pop tasks from the start of their queue and run the task until it yields and then pops another one. When tasks are scheduled unto the runtime they usually come in batches by the I/O driver (when new events from the OS are received and corresponding tasks are woken to be added to a run queue) all these tasks get scheduled unto the local queue of one of the threads. Without any mechanism to distribute this load across all the available threads, we may experience resource underutilization where some threads are idle and others are doing much of the work.

To solve this issue and distribute the work uniformly, the Tokio scheduler uses work-stealing. When a thread has nothing else to do (its local run queue is empty), before going to sleep it checks the run queue of other threads for work and tries to steal some of their tasks. This is outlined in figure 5. Thread 2 has run out of tasks and before going to sleep it checks other threads for work; it iterates through all the threads in descending order and finds right away that thread 4 has work to steal so it steals half of the tasks of thread 4 from the beginning of the queue.

The good thing about this approach is that synchronization and cross-thread communication is mostly avoided as the load on the system increases and threads never reach an empty queue to try to steal from others. What happens when a thread tries to steal from other threads, doesn’t find any work to steal, goes to sleep, and then a batch of tasks is pushed unto the run queue of other threads? A mechanism must be implemented so that threads with work can notify other sleeping threads so that they can wake up and steal work [5].

Queue and Work Stealing Implementation

The run queue is backed by a circular buffer outlined in listing 14. The \( Arc \) type provides shared ownership of a value and is a thread-safe reference-counting pointer. Meaning that it’s a reference to an object in the heap and can be cloned as many times; once the last reference has gone out of scope the object itself is deleted from the heap. The circular buffer has a tail that is only updated by a single thread and read by multiple threads and a head that can be updated concurrently by multiple threads. It is backed by a constant-sized array of Tasks; the elements of this array are uninitialized at first. The Box type indicates that the value will be allocated in the heap.

Two handles are created from the local queue of each thread, one is a \( Steal \) handle and the other is a \( Local \). The local handle is used by a single thread to push and pop tasks from the queue and the steal handle is available for all other threads so that they can peek into this thread’s local queue and steal tasks if necessary. This is why the head and tail values are atomic; because they may be written to or read by multiple threads.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

/// Producer handle. May only be used from a single thread.
pub(crate) struct Local {
    inner: Arc<Queue>,
}

/// Consumer handle. May be used from many threads.
pub(crate) struct Steal{
    inner: Arc<Queue>
};

pub(crate) struct Queue {
    /// Concurrently updated by many threads.
    head: AtomicU32,
    /// Only updated by producer thread but read by many threads.
    tail: AtomicU16,
    buffer: Box<[MaybeUninit<task::Task>; LOCAL_QUEUE_CAPACITY]>,
}

Listing 14: Circular Buffer

The 32-bit unsigned head of the queue is actually made up of two 16-bit unsigned integers. The left-most 16 bits are called the steal head and the right-most 16 bits are called the real head. These two heads are used for stealing operations and act as a signal to other threads that specify whether someone is currently stealing from the given queue or not. The real head specifies the position of the next task to be popped and the steal head specifies the starting position of the tasks that are going to be stolen from a run queue.

Pushing tasks into the queue is done by first adding the task to the backing array at the position indicated by the tail and then incrementing the tail. Before pushing a task, the value of the real head is subtracted from the tail and is compared with the maximum length of the backing array; if there were enough space available, the pushing continues.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

impl Steal {
    pub(crate) fn steal_into(&self, dst: &mut Local<T>) {
        let dst_tail = dst.inner.tail.unsync_load();

        let mut prev_packed = self.inner.head.load();
        let mut next_packed;

        let (src_head_steal, src_head_real) = unpack(prev_packed);
        let src_tail = self.inner.tail.load();

        // If these two do not match, another thread is concurrently
        // stealing from the queue.
        if src_head_steal != src_head_real { return 0; }

        // Number of available tasks to steal
        let n = (src_tail.wrapping_sub(src_head_real)) / 2;

        // No tasks available to steal
        if n == 0 { return 0; }

        // Update the real head index to acquire the tasks.
        let steal_to = src_head_real.wrapping_add(n);
        next_packed = pack(src_head_steal, steal_to);

        // Claim all those tasks.
        self.inner.head.compare_exchange(prev_packed, next_packed);

        let (first, _) = unpack(next_packed);
        for i in 0..n {
            // Compute the positions
            let src_pos = first.wrapping_add(i);
            let dst_pos = dst_tail.wrapping_add(i);

            // Read the task
            let task = self.inner.buffer[src_pos].read();

            // Write the task to the new slot
            dst.inner.buffer[dst_pos].write(task);
        }
        let mut prev_packed = next_packed;

        // Update `src_head_steal` to match `src_head_real`
        let head = unpack(prev_packed).1;
        next_packed = pack(head, head);
        self.inner.head.compare_exchange(prev_packed, next_packed);

        // No tasks were stolen
        if n == 0 { return None; }

        // Make the stolen items available to consumers
        dst.inner.tail.store(dst_tail.wrapping_add(n));
    }
}

Listing 15: Stealing

Popping is also straightforward, first, it is checked that the value of the real head and tail are not equal, if they are then the buffer is empty and no tasks can be popped. Otherwise, the tasks at position real_head are taken out from the buffer and the value of real head is incremented by one.

A simplified version of the stealing algorithm is outlined in listing 15. The steal_into function is called by the stealing thread on the steal handle of the run queue of the thread that it wants to steal from. This function takes a local run queue handle as the destination queue into which tasks will be stolen. Stealing threads pass their own run queue to this function.

Lines \( 3 \) through \( 9 \) load the tail and head values for the source and destination queues. The head value is unpacked into steal and real head. Unpacking is simply splitting the 32-bit value into two 16-bit values. At line \( 13 \), if the steal head and the real head of the source queue do not match then this means that another thread is concurrently stealing from this queue and the function returns immediately. At line \( 16 \) the number of tasks to be stolen \( n \) is assigned half of the number of available tasks on the source queue. To signal other threads that a stealing operation is underway, the real head of the source queue is advanced by \( n \) at line \( 22 \) and after repacking the head values into a 32-bit number, it is placed back into the atomic head field of the source queue. We then go through all the claimed tasks and transfer them one by one from the source queue to the destination queue. We then update the steal head of the source queue to match its real head; this signals that the stealing operation has ended. Finally, the tail of the destination queue is updated so that the stolen tasks can be popped and executed by the consumer thread. This procedure is depicted in figure 6.

For the sake of simplicity, all the code concerning memory orderings has been removed from the examples. There are also some subtleties that must be noted while pushing a task unto the run queue of a thread:

• If the queue is not full, then the task is simply pushed back to the queue.
• If the queue is full and there is a thread currently stealing from the queue (the steal head and the real head are not equal) then this means that the run queue will be empty soon so the given task is instead pushed to the global queue. The global queue (also called the inject queue) is a shared linked list among all the threads where the insertion or deletion of tasks in the queue is guarded by a mutex.
• If the queue is full and no other thread is currently stealing from the local run queue, then push_overflow is called. This function takes half of the tasks of the local run queue and adds them in a batch into the global queue.

Workers

There are a couple of important data structures that are outlined in listing 16. When the multithreaded scheduler is created, a number of workers are also created. Each worker is assigned to a newly spawned thread (one thread per worker). An instance of the \( Core \) structure is given to each thread; this struct contains data such as the local run queue, the number of ticks, etc. The \( Shared \) struct contains data that is shared across all workers (threads) such as the global queue (inject queue), a list of \( Steal \) handles corresponding to the queues of all the threads (for example \( remotes[2] \) corresponds to the \( Steal \) handle of the third worker’s run queue) and some other structures.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

/// Core data
struct Core {
    tick: u32,
    lifo_slot: Option<Task>,
    /// The worker-local run queue.
    run_queue: queue::Local,
    is_searching: bool,
    ...
}

/// State shared across all workers
pub(super) struct Shared {
    /// Per-worker remote state. All other workers have access
    /// to this and is how they communicate between each other.
    remotes: Box<[Remote]>,
    /// Global task queue.
    inject: Inject<Arc<Handle>>,
    ...
}

/// Used to communicate with a worker from other threads.
struct Remote {
    /// Steals tasks from this worker.
    steal: queue::Steal<Arc<Handle>>,
    /// Unparks the associated worker thread
    unpark: Unparker,
}

Listing 16: Structs

The LIFO slot in the core structure is an optimization for message passing scenarios. When two tasks are communicating over a channel, the following scenario happens frequently:

• \( Task1 \) and \( Task2 \) are communicating over a channel.
• \( Task2 \) is currently blocked on side of the channel waiting for a message from \( Task1 \)
• \( Task1 \) sends a message through the channel
• \( Task2 \) is wakened up and is placed at the end of the run queue The problem here is that since \( Task2 \) is placed at the end of the run queue, there might be a lot of latency between the sending of the message by \( Task1 \) and its receive by \( Task2 \). For this reason, the Tokio scheduler has a LIFO (last in first out) slot where new tasks are placed inside it and when the scheduler wants to pop tasks from the queue, it first checks the LIFO slot for a task and only if it was empty goes for the other task at the head of the queue.

Let’s now take a look at how workers are created and what they do when they start. Listing 17 shows this procedure. First of all, each worker is run on a separate new thread. The workers then start executing statements inside a loop until the runtime is shut down. The \( tick \) field of the \( Core \) structure holds the number of iterations that the worker has had inside this loop. At line \( 15 \) the tick is incremented. If the value of tick is multiple of event_interval (part of the shared scheduler configuration) then the worker proceeds to try and run the driver at line \( 19 \). We’ll talk about the driver in the next section.

In line \( 23 \) the worker tries to acquire a task. The next_task function first checks the LIFO slot for a task and if it was empty it pops a task from the local run queue of the worker. Running a task as we discussed in the previous chapter, means pulling its associated future.

The steal_work function at line \( 30 \) would first try to steal from sibling workers into its run queue and return one of the stolen tasks (the stealing mechanism was explained previously) if no tasks could be stolen from the siblings, it then checks the global run queue for a task. If no tasks were available, the worker is parked (goes to sleep) until an influx of tasks prompts other threads to wake this worker up. Before going to sleep, some threads may also try to take and run the driver/

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

... {
    ...
    // as a part of the staring sequence
    // of the runtime.
    for worker in self.all_workers() {
        runtime::spawn_thread(move || run(worker));
    }
    ...
}

fn run(worker: Arc<Worker>) {
    let core = worker.core.take();
    while !core.is_shutdown {
        // Increment the tick
        core.tick();

        if core.tick % self.worker.shared.handle
                       .config.event_interval == 0 {
            // Run I/O Driver, Timer, ...
            core = self.run_driver(core);
        }

        // First, check work available to the current worker.
        if let Some(task) = core.next_task(&self.worker) {
            core = self.run_task(task, core)?;
            continue;
        }

        // There is no more **local** work to process, try to
        // steal work from other workers.
        if let Some(task) = core.steal_work(&self.worker) {
            core = self.run_task(task, core)?;
        } else {
            // Wait for work
            core = self.park(core);
        }
    }
    ...
}

Listing 17: Worker

I/O Driver

The runtime driver is protected by a mutex and only a single worker at a time can have control of the driver. Threads try to take control of the driver on a regular basis (every 64 ticks for example). The runtime driver is composed of different drivers that serve different purposes: I/O driver, time driver, process driver, etc. The I/O driver is responsible for listening to for events from the operating system and dispatching resources associated with those events.

In listing 17 at line 20, when a worker tries to run the runtime driver, it first has to lock a mutex, if the lock was successful (no other worker is in control of the driver) the worker can proceed. Here we will be focusing on the I/O driver of the runtime driver. (when the run_driver function is called it runs its inner drivers including the I/O driver).

The I/O driver contains a data structure that can save the records that specify which resources are associated with which events. Right now Tokio uses Mio a low-level I/O library for Rust; Mio is an abstraction over different operating systems’ asynchronous APIs [10]. The leaf futures register resources they are interested in with Mio through the data structure that the I/O driver provides. Mio then interfaces with the operating system and passes down these events using whatever API the underlying OS provides. The I/O driver then blocks on a call to poll function from Mio and waits for events to arrive. After receiving the events from the operating system, Mio passes up these events to the caller of poll, and its the caller’s job to handle which resources should be notified.

For example in the case of reading from a socket, when a task blocks for a socket to be readable, it is the I/O driver’s job to get the events from Mio, see that there is an event associated with the socket saying that the socket is now readable, and call the waiting task’s waker wake() method.

Listing 18 shows the function that gets called when running the I/O driver. At line \( 5 \) prepare an array of events (this array is reused across calls to the I/O driver). Then at line \( 9 \) the thread that is running the driver blocks on a call to poll passing the array of events to be filled and waiting for events to arrive from Mio (and hence the operating system). Once the events arrive, it iterates through them one by one and dispatches the sources associated with them (for example rescheduling tasks back on the run queue).

One important thing to notice is that the tasks are all scheduled on the run queue of the worker taking control of the driver, this is why earlier we said that tasks come in batches. Now we let the work-stealing algorithm do the rest and distribute the tasks across workers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

fn turn(&mut self, max_wait: Option<Duration>) {

    self.tick = self.tick.wrapping_add(1);

    let mut events = self.events.take();

    // Block waiting for an event to happen,
    // peeling out how many events happened.
    self.poll.poll(&mut events, max_wait);

    // Process all the events that came in, dispatching appropriately
    let mut ready_count = 0;
    for event in events.iter() {
        let token = event.token();
        self.dispatch(token, Ready::from_mio(event));
        ready_count += 1;
    }

    self.events = Some(events);
}

Listing 18: Driver

Conclusion

In this section, we introduced Tokio: an asynchronous runtime for Rust. Tokio plays the role of the executor that we talked about in chapter Asynchronous Programming. We took a look at how the multithreaded scheduler works and what the work-stealing algorithm does and why it is important. We then dug deeper into the implementation of the queue used by the scheduler and how work-stealing functions with them. The inner workings of the workers were discussed, how they are created and run on their own thread, what data types they use and how they function. Finally, we took a more in-depth look at the I/O driver that has been discussed since chapter Asynchronous Programming.

io_uring

Linux provides a set of system calls for doing asynchronous I/O. epoll is an example of such a system call that is used in Mio as its Linux backend (see chapter Tokio section I/O Driver, These system calls can monitor a set of file descriptors such as network sockets and check their status for doing I/O, e.g. see whether the socket is readable.

epoll takes a set of interests containing the file descriptors that a process wants to monitor. It then returns a ready list (a subset of interest list) which is a set of file descriptors that are ready for I/O; the process can block on the epoll_wait system call until a set of ready file descriptors is returned by the kernel.

epoll for Files

Being able to use asynchronous I/O for file system operations is consequential for performance. Large-scale Webservers, SFTP servers, databases, and a lot of other applications all need the asynchrony that epoll provides; because when you reach a point where there are thousands of requests and concurrent connections, blocking for read and write operations on the file system can have a huge impact on performance.

The problem with epoll is that it always returns a ready status for files. In fact, using epoll is only sensible for file descriptors that normally block when being used for I/O such as network sockets and pipes. Because of this, Tokio and other libraries in different languages use a thread pool for file operations meaning when a future wants to do I/O on files, the request for I/O is sent to another thread and is executed synchronously there. This is where io_uring comes in. [9]

The I/O Uring

io_uring was introduced in Linux Kernel version 5.1 in 2019. The goal of io_uring is to replace old interfaces for async I/O and bring improved performance by reducing overhead for applications. io_uring solves the problem with regular files and brings a uniform API under which programs can enjoy the benefits of asynchrony.

io_uring uses two ring buffers. One for submitting requests to the kernel and the other for receiving the completion of those requests.

The io_uring_setup() system call is used by the program to create these two ring buffers. These two buffers are created using mmap with the goal of sharing them between user space and kernel space. This reduces the overhead of copying data structures between the two. [6]

Submissions are added to the submission buffer in the form of a submission queue entry (SQE) type. SQE is a struct with fields that can support a lot of the operations that a program would need, e.g. reading from a socket, creating a file, writing to a file, etc. After adding SQEs to the submission buffer, the program can call the io_uring_enter() system call to signal the kernel that there are SQEs available and that it should start reading them. A simplified version of the SQE struct is available in listing 19.

Different fields of this struct serve different purposes depending on the type of the operation. For example, if you wanted to do a rename operation as the opcode you would use IOURING_OP_RENAMEAT (one of the cases in a huge enum called io_uring_op), set addr as the old path and addr2 as the new path; other fields may also need to be filled. This is effectively equivalent to the renameat2 system call. [7]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

struct io_uring_sqe {
	__u8	opcode;		/* type of operation for this sqe */
	__u8	flags;		/* IOSQE_ flags */
	__s32	fd;		/* file descriptor to do IO on */
	union {
		__u64	addr;	/* pointer to buffer or iovecs */
        ...
	};
	union {
		__u64	addr2;
        ...
	};
	__u32	len;		/* buffer size or number of iovecs */
	union {
		__kernel_rwf_t	rw_flags;
        ...
		__u32		rename_flags;
        ...
	};
	__u64	user_data;	/* data to be passed back 
                            at completion time */
};

Listing 19: SQE

After processing the SQEs, the kernel submits completion queue entries (CQE) to the completion queue. Each CQE in the completion queue has had a corresponding SQE in the submission queue entry. The program then reads these CQEs and takes proper action based on their status. The user space can block and wait for CQEs using the same system call for submitting SQEs io_uring_enter().

Figure 7 shows a simple view of how these buffers work.

This work can be submitted and read in batches and thus, the number of system calls reduces significantly which can improve the performance of the application. One of the reasons that system calls can tax the performance so much is because of the security precautions that are taken by the operating system to avoid vulnerabilities like Specter and Meltdown [4] [8].

Conclusion

We talked about the asynchronous APIs that the Linux kernel provides for applications and took a peek into the inner workings of epoll. We then discussed how epoll falls short when it comes to asynchronous file operations and how runtimes using epoll have to resort to a thread pool for their file I/O. After that, io_uring was introduced as a solution to the shortcomings of previous async I/O APIs of Linux and we developed a view of how io_uring works. Tokio has been working on tokio_uring to experiment with the integration of tokio and io_uring [1].

Conclusion

In this post, the Rust programming language was introduced; we saw how concepts like ownership and borrowing work together to achieve memory and type safety at compile time. Traditional languages like C/C++ were notorious for easily allowing developers to make mistakes while managing their memory, causing dangerous security flaws and vulnerabilities to be rampant in programs written in those languages.

We then shifted our attention to asynchronous programming and talked about different multitasking methods such as preemptive and cooperative. We outlined the advantages and disadvantages of each method and concluded that to avoid saturating memory when the number of requests increases dramatically, we can use asynchronous tasks (green threads) instead of traditional threads that each had their own call stack. We then took a peek at how Rust allows writing asynchronous programs just as easy as their normal synchronous counterparts. Futures and the state machine that the compiler generates were discussed and saw how we need executors to poll these futures and need wakers to be able to reschedule future on to the runtime.

Next, we talked about Tokio, an asynchronous runtime for Rust, and outlined the different parts of it. We took an in-depth look at how tokio schedules tasks and how workers that run on a thread, manage their run queues. The I/O driver and its job with registering resources was also discussed. In the final chapters, asynchronous interfaces for the Linux operating systems were introduced and their advantages and disadvantages were outlined. We talked about how io_uring was introduced to resolve the shortcomings of the previous async interfaces.

References

You can cite this work through this DOI.

• [1] A crate that provides io-uring for Tokio by exposing a new Runtime that is compatible with Tokio but also can drive io-uring-backed resources. 2022. url: https://github.com/tokio-rs/tokio-uring.
• [2] Alex Gaynor. What science can tell us about C and C++’s security. 2021. url: https://alexgaynor.net/2020/may/27/science-on-memory-unsafety-andsecurity/.
• [3] Crate futures 0.1.30, Trait futures::future::Future. 2018. url: https://docs.rs/futures/0.1.30/futures/future/trait.Future.html.
• [4] Paul Kocher et al. “Spectre Attacks: Exploiting Speculative Execution”. In: 40th IEEE Symposium on Security and Privacy (S&P’19). 2019.
• [5] Carl Lerche. Making the Tokio scheduler 10x faster. 2019. url: https://tokio.rs/blog/2019-10-scheduler.
• [6] liburing manual: io uring.7. 2022. url: https://git.kernel.dk/cgit/liburing/tree/man/io_uring.7.
• [7] liburing manual: io uring enter.2. 2022. url: https://git.kernel.dk/cgit/liburing/tree/man/io_uring_enter.2.
• [8] Moritz Lipp et al. “Meltdown: Reading Kernel Memory from User Space”. In: 27th USENIX Security Symposium (USENIX Security 18). 2018.
• [9] Lord of the io uring: Asynchronous Programming Under Linux. 2022. url: https://unixism.net/loti/async_intro.html.
• [10] Mio: Metal IO library for Rust. 2022. url: https://github.com/tokio-rs/mio.
• [11] Philipp Oppermann. Writing an OS in Rust: Async/Await Chapter. 2020. url: https://os.phil-opp.com/async-await/.
• [12] Rust Documentation, Trait std::future::Future. 2022. url: https://doc.rust-lang.org/std/future/trait.Future.html.
• [13] Rust RFC 2033 experimental coroutines. 2018. url: https://github.com/rust-lang/rfcs/blob/master/text/2033-experimental-coroutines.md.
• [14] Rust RFC 2394 async/await. 2018. url: https://github.com/rust-lang/rfcs/blob/master/text/2394-async_await.md.
• [15] Rust RFC 2592 Future API. 2018. url: https://github.com/rust-lang/rfcs/blob/master/text/2592-futures.md.
• [16] Stack Overflow. Stack Overflow Developer Survey 2021. 2021. url: https://insights.stackoverflow.com/survey/2021#section-most-loved-dreadedand-wanted-programming-scripting-and-markup-languages.
• [17] The Rust Book. 2022. url: https://github.com/rust-lang/book.
• [18] The Rust Unstable Book. 2022. url: https://doc.rust-lang.org/beta/unstable-book/language-features/generators.html.
• [19] Tokio: A runtime for writing reliable asynchronous applications with Rust. Provides I/O, networking, scheduling, timers, … 2022. url: https://github.com/tokio-rs/tokio.

The Rust code is available here and the mobile application can be found in this repository. This development of this project was in 2022-09.

Introduction

There have been numerous smart energy monitoring systems based on the Internet of things (IoT). In most of these systems, the end devices are directly connected to a network and are continuously sending out their data. This poses a real challenge in environments where a reliable wireless connection is not always available, e.g., inside containers, remote locations, or even in places where there is simply no proper infrastructure for wireless communication.

I have developed an offline-first energy monitoring system using ESP32 microcontrollers and their bleeding-edge Rust ecosystem to store data locally in their flash. The readings are collected from the devices by connecting to their access point and are saved on the local storage using a smartphone running a fork of the Thingsboard Flutter Mobile application. After collecting the telemetry data from the devices and when a network connection is available, the mobile app can flush the data of all the devices to the Thingsboards server, an IoT platform used for data collection and visualization. Rust is a type-safe and memory-safe system programming language, it provides its safety guarantees at compile time using rules that eliminate many memory-safety issues. There has been a significant adoption using Rust for embedded development after companies like STMicrocontrollers and Espressif Systems introduced their community projects for enabling the use of the Rust programming language on their SoCs and modules.

A Current transformer is connected to the ADC subsystem of the ESP32 microcontroller and is used to calculate the RMS current and voltage and kWh of the power line. These readings are periodically saved to flash storage. The device also creates a Wi-Fi access point using a unique SSID that is a combination of its MAC address and a predefined string. It then spins up a web server to which clients can request telemetry data along with other operations.

The filesystem used for flash storage is of utmost importance. We have chosen LittleFS since it is power-loss resilient and has wear-leveling mechanisms for flash storage. It also uses constant RAM to work with any kind of data. Some parts of the flash storage have been reserved for Over-the-Air (OTA) updates, but the majority of the flash storage is formatted with the LittleFS file system. The devices are identified using their server-side representation token which is given to them on their first-time initialization using the mobile app. The mobile app also maintains a list of the devices it has connected to and fetches their server-side information when a connection is available. This information is shown alongside the devices’ SSID when scanning for access points. After collecting the data from the devices and syncing it with the Thingsboard server, customers with a few devices or operators with tens of devices can view their data both on the mobile app and on the Thingsboard web app.

Rust and ESP32

Rust programming language has attracted the attention of many companies today. This language allows programmers to write safe programs with concepts such as Ownership and Borrowing and new programming structures. During compilation, this language can guarantee that your program will not encounter many types of errors that were common in older languages such as C. Espressif has designed a framework for programming its microcontrollers, which includes a set of libraries written in C and other tools written in Python. This framework is called esp-idf, and there is a tutorial on how to install and start working with it on the company’s website. This framework provides programmers with all the tools needed to set up a project in C language and easily use different parts of the microcontroller including Wifi and Bluetooth.

Over the last few years, the company has started a movement to make it possible to use esp-idf, which is written in C, in the Rust language using FFI, or Foreign Function Interface. Using FFI, code written in one language can be called in another language and its return value can be retrieved. All the work done in this regard is in the esp-rs repository and the development of all these programs is done as open source on Github.

At the time of writing this, two categories of programs are being developed:

• Programs that are completely written in Rust and are so-called bare-metal. That is, they are very close to hardware and do not need Rust’s standard library. Like: esp-hal
• Programs that use the esp-idf API through FFI instead of implementing everything from scratch in Rust. These programs must use the Rust standard library because they use functions needed to communicate with C and esp-idf, and therefore require std. For example: esp-idf-hal

Using the first category is almost impossible due to the immaturity and lack of preparation of many APIs required in industrial work. So currently, most people use the second category, which is based on the tested and advanced esp-idf framework. [1]

Software Packages Needed for Rust Development on ESP32

In the Rust language, a crate is the smallest unit recognized by the Rust compiler. In general, crates can be considered a software package or a project. The following packages are used in developing Rust applications with esp-idf:

• embedded-hal: This package contains a set of programming interfaces or traits for using HAL in embedded environments and does not include any code related to a specific microcontroller.
• esp-idf-hal: This package is the implementation of embedded-hal for ESP32 microcontrollers through esp-idf.
• embedded-svc: This package contains a set of programming interfaces or traits for using different services such as wifi, Bluetooth, and httpd in embedded environments and does not include any code related to a specific microcontroller.
• esp-idf-svc: this package is the implementation of embedded-svc for ESP32 microcontrollers through esp-idf.
• esp-idf-sys: This package provides raw and insecure connections to the esp-idf library written in C.

Also, since most ESP32 microcontrollers use the Xtensa architecture and this architecture is not supported by default by the Rust compiler and its backend, which is LLVM, Espressif maintains a fork of the Rust compiler that supports this architecture. To get started, this fork of the Rust compiler must be downloaded and installed. The rest of the software needed to flash and monitor the microcontroller and other tips can be found in the Espressif documentation for Rust programming. There is a step-by-step tutorial on how to set up a Rust project in these documents, and it is recommended to read them. [1] To flash, it is enough to install the espflash program from the tutorial above and use the following command to flash the binary file placed in the project on the microcontroller:

1

$ espflash /dev/ttyUSB0 target/xtensa-esp32-none-elf/release/examples/blinky --monitor

And to create a binary file suitable for OTA use:

1

$ espflash save-image --partition-table partitions_singleapp.csv ESP32 target/xtensa-esp32-espidf/release/sem sem103

Filesystem

In order to be able to store and manage a lot of data in the flash memory of the microcontroller, we need a file system. According to our needs, this file system should do three things:

• Be resistant to power outages. Microcontrollers that are used to monitor the power flow may be interrupted at any moment, and this sudden event should not cause the stored data structures to suffer and the system to be in an unknown state.
• RAM memory in microcontrollers is very limited, and regardless of the size of the file in the memory and the number of files, the use of RAM should be constant and not change with the increase of input.
• Flash memories have a limited number of write cycles, and if a physical part of the flash is written a large number of times (usually 10,000 to 100,000 times), that part will be damaged. Therefore, the file system must take care to use all memory blocks to spread the depreciation over all blocks. After my research, I have come to the conclusion that there are generally three file systems used for flash memory in microcontrollers. These three file systems and the NVS method, which stores values in memory as key and value pairs and is not considered a file system, have been compared together. [2]

SPIFFS and LittleFS seem to be good options. Examining the repository related to SPIFFS, we find that there is not much development for this file system and the project is almost abandoned. But LittleFS shows a lot of potential. LittleFS is constantly being developed and has a fairly large community behind it. The MicroPython project also uses this file system by default. This relatively new file system uses the Copy-on-Write method, in the simplest case, it works in such a way that when a branch is changed in the file system tree, the whole branch is copied somewhere else and then the change is applied. The previous branch is marked deleted by changing a flag that indicates it no longer has the correct value. With this method, there is no need to erase the memory unnecessarily and use flash memory writing cycles, and you can also make sure that if a problem occurs during writing or the power is cut off, because the values are copied before changing, the original values will not be tampered with. They remain valid until the write is complete. It should be noted that many optimizations are used in this file system to improve the efficiency of the COW method, the details of which are available in the project documentation. [3]

This file system is not suitable for situations where it is necessary to change the middle of a large file, because since the entire file is copied before the change, the entire memory space may be filled due to this copy and the continuation of the file may have problems. But this problem does not exist if we just add something to the end of the file. But it is still generally better to avoid increasing the size of each file, which we will do with the sharding method.

Measuring current and voltage using SCT sensors

These sensors are connected as a clip around the current carrying wire. The current inside the wire passes through a thick metal ring inside the clip and induces a current inside this ring. Another coil with a specific number of turns is connected to the metal ring, and when a current is induced in the big ring, another current proportional to it is induced in the coil; This ratio is determined by the number of coil turns. So if, for example, 100 amps are passed through the desired wire, depending on the number of small coil loops, the output current of the sensor can be something like 10 milliamps. [6] To read the output of the sensors, they are connected through a 3.5 mm jack converter to the pins of the microcontroller that have ADC capability. Since we are going to be working with city AC power, we can calculate the RMS current and voltage using readings from several sensor outputs. The point that should be noted is that the ADC output is a positive number between 0 and n depending on its resolution, and therefore it is necessary to find the middle of the sine wave corresponding to the electric current and subtract it from the whole wave to get our numbers. around 0. This number to be subtracted is known as the DC offset, which is shown below. [7]

The algorithm:

• First, we find the middle value of the voltage in a loop. This is easily done by having the maximum value that the ADC can output.
• Then, in another loop, we continuously read the voltage and current values until the time ends or a certain number of passes through the middle of the wave has been done, then we apply a low-pass filter, and finally, collect the readings in the necessary variables. During this time, we store the minimum and maximum value read for current and voltage, and after the loop is finished, we improve the offset value, which is the middle value in the wave.
• At the end, we calculate the RMS values for voltage and current and get the real and apparent energy and kwh. You get the kwh value cumulatively; That is, when the corresponding function is called, the kwh values are added together and whenever we reach an hour, the kwh value will have the correct value for that hour.

Using LittleFS in Rust

To use LittleFS in the esp-idf environment, you can use the esp-littlefs project. As in normal C projects, you can add this package as a component to your project and use standard C functions to work with the file in the system. But to use esp-little in Rust, you need the help of esp-idf-sys. In the Rust project settings, we specify that when compiling the program, it should create the necessary binding to use this file system in Rust.

Sharding

As explained earlier, using large files to write to LittleFS is not highly recommended; Therefore, instead of having a large file in the system that will be written into for months which reduces the system’s performance, we can use sharding to solve this problem. In this way, a fixed size is set for each file, and if the size exceeds that limit when writing, a new file will be created and the system will write the values in that new file from then on. This method is also widely used in databases to avoid handling large files. [9]

Dealing with power outages

The hardware that we had at hand did not include a separate RTC module and it was not possible to make any changes to the hardware. Therefore, since the timestamps related to the readings are recorded in the device, to improve the error caused by power failure, the microcontroller periodically stores its RTC value in the flash memory and every time it starts working, the stored RTC value is read from the memory and is set as the system clock. After setting the clock, it appends the RTC value read from the memory in a file called powerloss_log. When receiving values from the microcontroller by the mobile application, the list of power failure events is also sent, and the mobile application tries to correct the timestamp of the data as much as possible by calculating the total duration of the power failure experienced by the microcontroller.

Rust program routine

At the beginning of the program, we launch the file system. This setup will format the LittleFS partition for the first time and only mounts it the next time. The Rust program is executed as a Task in the FreeRTOS operating system that esp-idf uses, and other tasks such as handling requests by the web server are done in other tasks. Therefore, since the microcontroller that I was using has more than one core, it is possible to run the main Rust code in parallel with the code related to the web server handlers. Thus, to prevent data race, a Mutex can be used for all operations that need to work with the file system. In the next step, we create a mutex with the LittleFS handle behind it. It then finds the latest shard to store readings in and executes the RTC and power-down routines discussed earlier. In the next step, the wifi system is set up. In this step, Micro sets up an access point; The SSID of this access point also includes the MAC address of the micro, so when several micros are together, they can still be distinguished individually. After that, the web server and all its handlers are started and registered, and then the ADC microsystem is started. Web server handlers are explained in more detail below. In the next part, after making sure that the above steps are started, the firmware version that is currently running is confirmed. This is to ensure the correct OTA update. For example, if a wrong update is done through OTA and the initial setup fails or the micro is reset due to an error because the current version is not verified, the micro will automatically go to the previous version that worked properly. In the final stage, the micro enters a loop that periodically reads and aggregates the values from the sensor, and stores the aggregated values in the memory after one hour.

Webserver

After running the web server, the following handlers are registered in it:

• /telemetry: the data of all shards are sent to the requester in binary form and in HTTP chunk format.
• /powerloss_log: All data related to power loss is sent to the requester.
• /time: The new clock is received in UNIX epoch time format and RTC is set with it.
• /token: if the request is a GET, the current token is sent, and if it is a POST, the sent token is stored in the current token array. The use of the token is explained below.
• /reset: All information except time is erased from the memory.
• /ota: The data concerning to the new version of the program is received as a chunk and placed in the next OTA partition. If the binary file is received correctly, the new partition will be set as a bootable partition in the OTA header. OTA update happens only when the received version is higher than the current version.
• /version: Sends the current version to the requester.

Flash memory partitioning

The file below is given as a partition table to the software that flashes the program to create the partitions in the flash memory:

## Name, Type, SubType, Offset, Size, Size in Bytes
nvs, data, nvs, 0x9000, 0x4000, 16k
otadata, data, ota, 0xd000, 0x2000, 8k
phy_init, data, phy, 0xf000, 0x1000, 4k
ota_0, app, ota_0, 0x10000, 1M, 1M
ota_1, app, ota_1, , 1M, 1M
littlefs, data, spiffs, , 0x1e0000,1.9M

As you can see, two partitions are considered for OTA; Each time a new program is downloaded as a binary file, one of the OTA partitions is used. The otadata partition specifies which partition the bootloader should run from during boot. nvs partition is used for wifi and phy_init partition is used for the physical layer and radio. Finally, the littlefs partition corresponds to the littlefs file system where data is stored. [5]

Thingsboard platform

Thingsboard is one of the most famous Internet of Things platforms that is completely open source and is used all over the world. Installation and commissioning of Thingsboard server were done through docker on a VM created in proxmox. Thingsboard can be used for:

• Device management
• Receive and store data that devices send
• Process data from devices and perform various tasks based on that data, such as sending an email or running a piece of code.
• Displaying data on different graphs
• Creating and launching firmware updates through OTA
• Creating different user accounts with different access levels
• And … Devices in this platform have names and descriptions, and each device is given a unique access token when it is created. Devices can send their data directly to the server with their token to the following endpoint in a specified format:

http://<server-address>/api/v1/<token>/telemetry

Each device is also located in a profile; Profiles are used to separate and group devices that perform the same task and must be managed together. For example, when using OTA update, we can present a new version of the binary file uploaded to the server to all devices that are in a profile, or when drawing a graph, we can display the data of all devices that are in a specific profile. [4][8]

Thingsboard Flutter mobile app

Thingsboard also has a mobile app written with the flutter framework called thingsboard_flutter, which allows users to view and interact with their dashboards and see graphs of device data. One of the things that were done, and we will talk about it further, is to fork this program and add the ability to collect offline data from devices and send it to the server. In the figure below, you can see a view of the added tab called collect:

Scan nearby access points

The wifi_iot library is used to scan and find nearby devices. This library can return the list of all nearby APs and connect to the requested AP by giving it an SSID. By default, the SSID of all devices with SEM- in the beginning is displayed, and therefore it is easy to display only the APs related to the required devices from the list of all nearby APs. Finally, the user can connect to the desired AP by clicking on it.

Initial setup of the device

After connecting to the device’s AP and before doing anything else, three things need to be done with the device:

• The token related to the device representation on the server should be given to the device. With this, every time the mobile application wants to get data from a device, it also gets the token and stores it in its database. To copy the token, a section has been added in the Devices tab through which users can click on the desired device and copy the corresponding token. After copying the token, it can be pasted into the field specified during the initial startup of the device.
• Send the current time in UNIX epoch time format to the device to update its RTC. If you are connected to the Internet, this time is taken online, and otherwise, the time of the mobile phone itself is used.
• The memory of the device should be reset to make sure that no unwanted data remains in the memory. Finally, a record is created in the database for this device that associates the SSID of the AP we are connected to to the device token. Also, the timestamp of the last connection to this SSID is also recorded in this record. With this, for example, when scanning, we can separate all the devices that we have connected to in the last 10 minutes with a different color.

Receiving data from the device

To receive information from the device, the following procedure is performed:

• First, the device token is taken.
• Then all the information about the readings is taken from the device.
• The updated time is sent to the device.
• A list of all power outage events is taken.
• Binary readings are converted to their corresponding objects.
• The difference between the time of the last data sent and the current time indicates the amount of power outage. If this amount is more than one hour, we divide it by the number of power outage events; The obtained number should be placed in the places where there is a power outage, which we obtain using the power outage event.
• Finally, the readings are stored in the mobile database and a request is sent to reset the device’s memory.

ER diagram and database architecture

The ER diagram of the database used in mobile is as follows:

As you can see, all the devices are placed in the devices table. The token given to each device is a combination of the access token and its device profile id on the thingsboard server. So after connecting to a device, its token is first taken and placed in this table along with its SSID. This feature has been added so that after connecting to the Internet and entering the All Devices section of the mobile app, the name assigned to each device on the server is added to this database; This is actually a link between a device’s SSID and its name on the server, which can be used to display each device’s name next to it when scanning for APs.

The last_checked field shows the time of the last time the mobile phone was connected to the device. If this time is less than ten minutes ago, when scanning for APs, the corresponding AP of the device will turn green to make it easier for the user to identify which device to connect to in the next step.

Perform OTA via mobile

The ota table is for managing otas and their binary files. Every time the user enters the Devices tab, the mobile application automatically checks through the server whether there is a new OTA update for the devices registered in the mobile database, and if there is, it downloads it. OTA updates work based on the device profile, and when checking for OTA, it is checked that the version on the server is greater than the version stored on the device, and if it is, the new version is downloaded and replaces the previous version. When connecting to the device, the program version of the device and its device profile id are taken first, and if there is a newer update file for it, the user can update the device by selecting the Update Device Firmware option.

Sending readings to the server

By selecting the Sync With Server option, the user can send readings related to any device stored in their database to the server. This is possible because in the mobile database, all access tokens of the devices are also stored, and as mentioned earlier, to send the readings to the server, it is enough to send the data in the specified format to the following address:

http://<server-address>/api/v1/<token>/telemetry

Display data on the server side

To display data on the thingsboard server, we need to design a dashboard so that the graph of all the devices that are in the SEM profile can be displayed on this dashboard. The designed charts are:

• Pie chart to compare the total energy consumption of devices
• Three graphs to display kwh in time intervals: hourly, daily and monthly
• Graph of average hourly current and voltage consumption (the voltage value is constant) Below is an image of the designed dashboard:

Conclusion

I have implemented an Internet of Things platform for power consumption monitoring based on ESP32 processors and Thingsboard server. One of the main problems that current IoT systems have is the need to constantly connect to a server or gateway; My method solves this problem by means of mobile application and using the AP capability of ESP32 microcontrollers with emphasis on offline operation. In this way, the mobile application acts as a gateway that connects to the devices and receives their data. In the future, an autopilot mode can be created for the mobile app to collect data from devices automatically; Of course, this is not possible in the Android operating system because applications do not have the right to do this without notifying the user and approving the application.

References

• [1] esp-rs book
• [2] Espressif Documentations about file systems.
• [3] LittleFS Design Guide.
• [4] Thingsboard Documentation.
• [5] Espressif Documentation about partition tables.
• [6] Open energy monitor guides.
• [7] Espressif Documentation About ADC.
• [8] Thingsboard Documentation for Sending Telemetry Data.
• [9] Sharding methods.

You could use Wine (vanilla wine or a wrapper like lutris, bottles, playonlinux, and etc.) to install the native client and share your screen that way, but sometimes getting that to work is just way too much of a hassle.

Here are two ways that I managed to share my screen on Adobe Connect and other platforms that don’t have a native linux client:

Virtual Camera

If there is a web app available and it allows you to share your webcam (like Adobe Connect), here’s how you can use this feature to share your screen on the web app using a virtual camera.

1. Install OBS Studio. I highly recommend installing it via the official Flatpak package, mainly because the binaries provided by various distributions are sometimes compiled with certain features turned off and are not always up to date.

2. Since version 26.1 OBS supports built-in Virtual Camera output on Linux. In order to use it, you have to install the v4l2loopback module. (depending on your distribution you may want to use DKMS). You need to find out how you can install this module for your distribution (you can also build it yourself). This could be as simple as:

1

sudo pacman -Syu v4l2loopback-dkms

for Arch-based distros,

1

sudo apt install -y v4l2loopback-dkms

for debian-based distros, or

1
2

boot.kernelModules = [ "v4l2loopback" ];
boot.extraModulePackages = [ config.boot.kernelPackages.v4l2loopback.out ];

for NixOS.

After the installation, you need to make sure that the v4l2loopback module is loaded.

3. Now open OBS Studios. You should see a Start Virtual Camera button. If you don’t see such a button, open OBS using a terminal, read the log messages, and diagnose why OBS can’t enable its virtual camera functionality.

4. Create a Scene and add a Screen Capture source to it. Then, start the virtual camera.

5. Join a webinar using a browser (I use Firefox. Some people have reported that their virtual cameras don’t show up on chromium-based browsers). Either get the host to make you a host as well or ask them to do the following:

• Go to Preferences -> Video and change the aspect ratio to 16:9.

• Create a new layout and add your desired pods to it along with a large video pod.

6. Click on the webcam button and allow the site to use your virtual camera.

7. Click on Start Sharing to share your screen.

You can control the output resolution of OBS in its settings. Turning down the resolution may help reduce the needed bandwidth to stream your desktop.

Virtual Machine

If there are no web apps, you can use a Windows virtual machine to share your Linux desktop. This is a little bit more involved and requires more resources. Here is the basic idea:

1. Create a Windows guest VM I use libvirt managed by virt-manager.

2. Install Adobe Connect in the VM

3. Install a remote desktop sharing client in the VM (e.g. TigerVNC)

4. Make sure there is network connectivity between the guest VM and host (the default NAT is okay) You need the IP address of the host machine in the virtual network. You can do a simple ip a on the host machine to find the IP address of the interface connected to the virtual network.

5. Install a remote desktop sharing server in the host (e.g. VNC Server, TigerVNC) I use TigerVNC. First, create a password using vncpasswd and then run x0vncserver -PasswordFile=.vnc/passwd to start a VNC server connected directly to your running x-server without creating a virtual desktop.

Now you can share the screen of your Windows guest in a webinar and connect to the VNC server on the host machine using its IP address in the virtual network and share the screen of your Linux machine through the Windows VM.