First Assignment

Before I begin to dive in to why it's awesome working on the Mozilla Releng team, what it is we do, or even how we do it, I want to discuss a bit about my role and what I've been up to.

Mozharness All The Things!

Currently we have a bunch of logic about how we 'build' and check the integrity of the steps related to compiling and building firefox. This varies across all our platforms, platform versions, release branches, project branches, and a number of other special types of builds

what is a build?

Taking a look at TBPL, for just say our beta release branch, all those big 'B's represent bound nested logic within our Buildbot repos. Let's not worry about all the details (yet!), but there is a part of all this code within our buildbotcustom and buildbot-configs repos that my job was to extract out. This was specifically the logic that decides what exact steps and values we should use for a given Firefox 'B'. Again this logic depends on what platform, version, branch, etc we use. Porting this out would leave Buildbot to focus on the scheduling, triggering other builds like tests and repacks, status updates, etc.

The goal is to move it all to Mozharness.

what benefit does porting it have?

This has a number of benefits. Aside from the obvious refactoring/cleaning up that is possible thanks to hindsight, there are some more real benefits. By not having this logic tied up with Buildbot, we no longer have to Reconfigure our continuously running buildbot machines to simply change, say, a minor path name we use once in only one of our builders. Instead, we can simply just push a mozharness patch to a repo known to buildbot and BANG, it will be picked up on the next build. We also no longer have to communicate back and forth multiple times between a Buildbot Master and a Buildbot Slave for every tiny command. But more than this, we are moving to Mozharness because Mozharness is focused on running a series of steps in scripts that must be highly configurable. That's it. Mozharness also doesn't really care who schedules and runs it or with what command line args. It doesn't care where its awesome logs end up going. This means that a) Buildbot can focus on what it does best b) Mozharness scripts won't be coupled to just being used by Buildbot and c) developers can call Mozharness scripts directly completely outside of our continuous integration. We can mimic small parts of our system, locally, with greater ease than ever before. <- more on this in future posts

So while I don't get too far ahead of myself, let's briefly look at where this logic lives in Buildbot. I have a series I am working on describing in more detail how we use Buildbot but this post is less of a tutorial and more of explaining what I've done.

where is this logic in Buildbot now?

I am going to over simplify here: We have this misc file that uses the goodness of Buildbot to create a bunch of schedulers, change_sources, and builders. Focusing on just the builders here, some of these builders contain the factories that make up the series of steps that dictate a firefox build/compile. The factory class NightlyBuildFactory, and its subclasses, contain all the steps that every desktop platform, branch, variant, etc that make up a build (outside of Try, l10n, and release builds). Which steps and the values we use for any given combination of those are dictated by our configs. That config file is the core file for desktop builds but there are many others.

These Factory classes and config dicts, make up the bulk of what is being ported.

I won't exhaust you with how this works in Buildbot. It's all very impressive and it has taken me months to come to grasps with what releng folk -- much better devs than myself -- have collectively put together. So let's move on and dive into how things that I ported run in Mozharness

how is it working in Mozharness?

First and foremost, the mozharness script we use:

scripts/fxdesktopbuild.py

This uses various goodness of other mozharness parts, namely:
mozharness/mozilla/building/buildbase.py

You invoke this script with what type of build you want to do:
--platform (is it Linux, Mac, Windows) --bits (32, 64 --branch (mozilla-central, beta, birch, etc) --custom-build-type (asan, debug, etc) --build-environment (staging, prepod, production) --enable-pgo --enable-nightly --developer-mode (means we are using this outside of buildbot so let's skip our continous integration steps)

That's pretty much the bulk of the major CLI you need to worry about.

From Buildbot's point of view, it now just invokes this script as one big Step through a trivial Factory. If it wants to do, say, a Linux ASAN 64 bit build for the cedar branch in our production environment we:

A buildbot master creates a step that tells a slave to run this:

./scripts/fx_desktop_build.py --platform linux --bits 64 --branch cedar --custom-build-type asan --build-environment production

But this simplification does not create more magic. What happens here is spelled out for you. The Mozharness script will take the CLI args and start putting together a config for you. A config only for that particular run that contains nothing else. Specifically, from the above example, the config would be made up with:

configs/builds/releng_base_linux_64_builds.py

configs/builds/releng_sub_linux_configs/64_asan.py

configs/builds/branch_specifics.py

configs/builds/build_environments.py

The full list for configs is here (right now only Linux is complete):

configs/builds  
├── branch_specifics.py
├── build_enviroments.py
├── releng_base_linux_32_builds.py
├── releng_base_linux_64_builds.py
└── releng_sub_linux_configs
    ├── 32_debug.py
    ├── 64_asan.py
    ├── 64_asan_and_debug.py
    ├── 64_debug.py
    └── 64_stat_and_debug.py

In the fxdesktopbuild.py script itself, like many mozharness scripts, there is also a 'default_config' dict that has some generic config keys/values for all desktop firefox builds.

but isn't this confusing?

No matter what you will have pros and cons to any way you structure this. The fact of the matter is you need a way to support all these permutations with minimal code reuse. The good news is the script will print its config for you when you run it or, if you just want to see the config and not run anything, add --list-config:

./scripts/fx_desktop_build.py --platform linux --bits 64 --branch cedar --custom-build-type asan --build-environment production --enable-pgo --list-config

The script will, every time, print off which files made up its config, how they differ in keys/values, and how precedence was decided between them.

But I hope this is considered simple and uniform. From lowest to highest you have tiered levels: platform specific configs, then custom types (see in relengsublinux_configs), then branch specifics, and finally the build environment. This is somewhat similar to buildbot-configs.

improving maintainability

The beauty here is that each of these config files do not duplicate each other with similar keys/values. The lowest config level (platform), holds every key/value that is not shared universally across every platform (those keys/values are, again, in default_config). Each 'higher' config will only contain what it is changing from its 'lower' config. Not only that, each level contains all the keys/values it needs and does not rely on copying from each other. The former means you are not stuck duplicating/maintaining keys/values across every level and the latter means no worrying about iterating over one config dict and copying it to another when the script is actually being executed. We currently have to do a lot of looping at the end of our buildbot-configs files. This should help with maintaining and readability.

Furthermore, there are way less keys/values as a whole. Instead, scripts rely less on configs by specifying core 'actions' that are independent from one another. Actions can be thought of as one step or a subset of steps within a buildbot factory. These actions are easily switched on/off via the config. So, this would mean we have less conditions in our script that need to evaluate items in our config when deciding whether want to do a build step/subset of steps.

eg: the 64asan.py config will overwrite the actions list in the relengbaselinux64_builds.py and will not include the pretty-names action. No condition needed, mozharness will only run actions it sees in the action list.

But that's not all, currently buildbot-configs/buildbotcustom can be daunting trying to decipher its big master config of everything. Mozilla Buildbot instances keep track of one config that acts like a root and contains nests of keys/values of all the branches, all the platforms, etc and all of their own keys/values. Right now, it is easier to save our config to file, make a change, save it again, and then diff them.

For example, when trying to decide whether we want to create snippets for a particular build, in buildbot, we dive into the config and check various places for 'createsnippets' and decide what to do based on those findings. We look in the branch dict under, say, 'mozilla-central' and that dict may have createsnippets set to false even though the platform dict under 'linux' has 'createsnippets' as well and it is true. There may or may not be 'createsnippets in the parts it looks at so we have defaults values for many of these. It can be hard maintaining this because buildbot has to do and know about all the details of everything.

And I think that goes full circle. Extract Desktop Firefox build logic out of Buildbot and divide it up among individual script runs which hold smaller configs. This leaves Buildbot less bloated and frees it up to focus on other important things.


Hopefully this work can make minor improvements to what we have and also prepare us for whats in store in the future. Next post I'll show some logs and code so the abstract becomes more concrete.

I'm really excited to start trying this out in cedar/cypress and showing everyone what I have been up to particularly within the script/configs themselves. This work should be trickling into mainline mozharness over the coming weeks followed by enabling it on a branch basis. Stay tuned.