Why Is My Build So Slow?!

Ross Burton
meta-arm co-maintainer
Principal Software Engineer
Arm Ltd
ross.burton@arm.com

Always share `SSTATE_DIR` and `DL_DIR`

The Build Machine

A typical laptop after building Chromium

lo-fi benchmark


$ git clone --branch kirkstone https://git.yoctoproject.org/poky
$ cd poky
$ . oe-init-build-env
$ bitbake core-image-sato --runall fetch
$ time bitbake core-image-sato

Corporate Laptop

Cost around £1.5K
8 cores, 16GB RAM

core-image-sato in 2 hours 30 minutes

“Mobile Workstation”

Cost around £3K
16 cores, 64GB RAM

core-image-sato in 1 hour 45 minutes

Workstation

Cost around £5K
32 cores, 64GB RAM

core-image-sato in 27 minutes
From sstate in 48 seconds

How much is your time worth?

Build Machine Wishlist

Plenty of cores

Good cooling

Lots of memory

Fast storage

Understand your build


$ bitbake --graphviz core-image-sato  # or -g

$ cat pn-buildlist
core-image-minimal
quilt-native
patch-native
pseudo-native
linux-yocto
...


$ grep gcc pn-buildlist
gcc-cross-aarch64
gcc


$ oe-depends-dot task-depends.dot --why --key gcc
Because: core-image-sato elfutils
core-image-sato -> elfutils -> gcc

`elfutils_0.187.bb`


RDEPENDS:${PN}-ptest += "libasm libelf bash make coreutils
                         iproute2-ss bsdtar libgcc-dev
                         gcc-symlinks binutils-symlinks"


          $ grep x11 pn-buildlist
          libx11


$ oe-depends-dot task-depends.dot --why --key libx11
Because: core-image-minimal dbus libxkbcommon
core-image-minimal -> dbus -> libx11

`dbus_1.14.4.bb`


PACKAGECONFIG ??= "${@bb.utils.filter('DISTRO_FEATURES', 'x11', d)}"
          
PACKAGECONFIG[x11] = "--enable-x11-autolaunch,
                      --without-x --disable-x11-autolaunch,
                      libx11 libsm"


$ DISTRO=poky bitbake-getvar DISTRO_FEATURES

DISTRO_FEATURES="acl alsa bluetooth debuginfod ext2 ipv4 ipv6
pcmcia usbgadget usbhost wifi xattr nfs zeroconf pci 3g nfc
x11 vfat seccomp largefile opengl ptest multiarch wayland
vulkan pam systemd"


$ oe-depends-dot task-depends.dot  --why --key bluez5
Because: connman core-image-sato gstreamer1 libpcap neard ofono packagegroup-base pulseaudio
core-image-sato -> connman -> ofono -> bluez5
core-image-sato -> packagegroup-base -> ofono -> bluez5
core-image-sato -> pulseaudio -> ofono -> bluez5
core-image-sato -> gstreamer1 -> pulseaudio -> ofono -> bluez5
core-image-sato -> libpcap -> bluez5
core-image-sato -> packagegroup-base -> neard -> bluez5

`3g bluetooth nfc wifi`

`INHERIT += "buildstats"`


$ bitbake core-image-sato

$ cat tmp/buildstats/20221110124011/curl-7.82.0-r0/do_configure
Event: TaskStarted
Started: 1668084334.63
curl-7.82.0-r0: do_configure
Elapsed time: 136.91 seconds
utime: 21
stime: 6
cutime: 2679
cstime: 654
IO rchar: 430902147
IO wchar: 59525723
...
Ended: 1668084471.54

`scripts/pybootchartgui/pybootchartgui.py`

`autotools.bbclass`

This is the yellow vertical section. each task is has its own colour, and here we can see that a large number of recipes started do_configure at the same time.

Looking just before this happened we see that libtool just finished building, and a little big of digging will show this is the bottleneck caused by autotools.bclass's dependencies.
...
Once that last dependency has been built, all the recipes that use autotools can start to build. In this case its unfortunatate that theres not a long we can do, as autotools is just slow to bootstrap and run.

it's interesting to note that meson was building recipes long before autotools had finished bootstrapping,If you're a heavy user of autotools, I do suggest moving to something like Meson.

`???`

`rust-native`

Making bitbake PSI-chic: regulating builds using /proc/pressure

by Randy MacLeod

https://youtu.be/aWq9KQI6dAU

Optimising Rebuilds

Hash Equivalence

This is a very rough overview of a tasks progress in bitbake.

Every task has a set of inputs: the dependencies, all the variables, and the tasks to be executed. These are all merged into a single task hash called the signature, which is looked up in sstate. If this hash exists then we can just unpack the sstate objects and move on to the next task, otherwise we run the task, producing some output, and put that into the sstate cache.

Now, there are many inputs where the output is identical. For example if I patch a library we'll rebuild everything that links to it, but if the library is being dynamically linked then those binaries probably won't actually change. hash equivalence is a way to say that two task hashes actually produce the same output hash, so the hashes should be considered to be equivalent.

https://docs.yoctoproject.org/overview-manual/concepts.html#hash-equivalence


Sstate summary: Wanted 972 Local 0 Mirrors 0 Missed 972 Current 479 (0% match, 33% complete)
NOTE: Executing Tasks
NOTE: recipe autoconf-native-2.71-r0: task do_configure: Succeeded
NOTE: recipe autoconf-native-2.71-r0: task do_compile: Succeeded
NOTE: recipe autoconf-native-2.71-r0: task do_install: Succeeded
NOTE: recipe autoconf-native-2.71-r0: task do_populate_sysroot: Succeeded
NOTE: Task virtual:native:autoconf_2.71.bb:do_populate_sysroot
      unihash changed to 76620844bfe6f3ad30ac3fa55fad8f9775cb68afacdedfc29822fe0a758cfa6e
NOTE: Setscene task zstd_1.5.2.bb:do_package_qa became valid
NOTE: Setscene task kbd_2.5.1.bb:do_package became valid
NOTE: Setscene task gcc_12.2.bb:do_packagedata became valid
NOTE: Setscene task libffi_3.4.4.bb:do_package_write_rpm became valid
...
NOTE: recipe core-image-minimal-1.0-r0: task do_image: Succeeded
NOTE: Tasks Summary: Attempted 3742 tasks of which 3729 didn't need to be rerun and all succeeded.

tl;dr: for faster rebuilds


BB_SIGNATURE_HANDLER = "OEEquivHash"
BB_HASHSERVE = "auto"

2 Mirrors 2 Fast 2 Furious:
Using Autobuilder SState


          BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687"
          SSTATE_MIRRORS = "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"

Locked SState

Bug #13425

Generating Locked Hashes


$ bitbake core-image-sato --dump-signatures=none
$ code locked-sigs.inc


+# Don't fail when calculated hash doesn't match locked hash
+SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = ""

 SIGGEN_LOCKEDSIGS_t-cortexa57 = "
 ...
-python3:do_compile:7e46157c6c9b4053507e314015a2823da08aea33f2a21b4f6aa0e57eb8458125
-python3:do_configure:b868b7690a1db7b7720527e19b60f9bf17b294ac576a5972e04bff9be2df079a
-python3:do_fetch:d23108869e5dde01e6662a9c6ff6d96aea1307fc39e435b9e9e99861c9ea0dd6
-python3:do_install:72db8e3503e96d62b3e5be5a0eb252da583e357ab93bb693f3b54e740e9b770d
-python3:do_package:0549c987964b31f37e2575a8f592bee6be393ecd275ee8dbbc61e378a8f28021
 ...


$ mv locked-sigs.inc conf/auto.conf
$ bitbake core-image-sato

In the style of a diff, this is what what want to do to the locked-sigs file.

First we need to unset the CHECK variable as by default bitbake will abort if the locked hashes don't match the calculated hashes.

Next we remove the hashes for the recipes which we want to rebuild. Say we were working on the python3 recipe. Normally a change to python3-native would cause rpm-native to rebuild which would then cause all the packaging tasks to re-run. by locking everything except python3 we can isolate the changes precisely.

This file needs to be included in the configuration somehow, an easy way is to just mv it to conf/auto.conf. Then just bitbake as usual.

ross.burton@arm.com

rburton on #yocto@libera.chat