First off, let me be very clear up-front: normally, I write my blog articles to be approachable by readers of varying levels of technical background (or none at all). This will not be one of those. This will be a deep dive into the very bowels of the sausage factory.
This blog post assumes that the reader is aware of the Fedora Modularity Initiative and would like to learn how to build their very own modules for inclusion into the Fedora Project. I will guide you through the creation of a simple module built from existing Fedora Project packages on the “F26” branch.
To follow along, you will need a good working knowledge of the git source-control system (in particular, Fedora’s “dist-git“) as well as being generally comfortable around Fedora system tools such as dnf and python.
Setting up the Module Build Service
For the purposes of this blog, I amΒ going to use Fedora 25 (the most recent stable release of Fedora) as the host platform for this demonstration and Fedora 26 (the current in-development release) as the target. To follow along, please install Fedora 25 Server on a bare-metal or virtual machine with at least four processors and 8 GiB of RAM.
First, make sure that the system is completely up-to-date with all of the latest packages. Then we will install the “module-build-service” package. We will need version 1.3.24 or later of the module-build-serviceΒ RPM and version 1.2.0 or later of python2-modulemd, which at the time of this writing requires installing from the “updates-testing” repository. (EDIT 2017-06-30: version 1.3.24 requires the mock-scmΒ package for local builds but doesn’t have a dependency on it.)
dnf install --enablerepo=updates-testing module-build-service python2-modulemd mock-scm
This may install a considerable number of dependency packages as well. Once this is installed, I recommend modifying /etc/module-build-service/config.pyΒ to change NUM_CONCURRENT_BUILDSΒ to match the number of available processors on the system.
Leave the rest of the options alone at this time. The default configuration will interact with the production Fedora Project build-systems and is exactly what we want for the rest of this tutorial.
In order to perform builds locally on your machine, your local user will need to be a member of the mock group on the system. To do this, run the following command:
usermod -a -G mock <yourloginname>
Then you will need to log out of the system and back in for this to take effect (since Linux only adds group memberships at login time).
Gathering theΒ module dependencies
So now that we have a build environment, we need something to build. For demonstration purposes, I’m going to build a module to provide the libtallocΒ library used by the Samba and SSSD projects. This is obviously a trivial example and would never become a full module on its own.
The first thing we need to do is figure out what runtime and build-time dependencies this package has. We can use dnf repoqueryΒ to accomplish this, starting with the runtime dependencies:
dnf repoquery --requires libtalloc.x86_64 --resolve
Which returns with:
glibc-0:2.25-4.fc26.i686
glibc-0:2.25-4.fc26.x86_64
libcrypt-0:2.25-4.fc26.x86_64
libcrypt-nss-0:2.25-4.fc26.x86_64
There are two libcrypt implementations that will satisfy this dependency, so we can pick oneΒ a little later. For glibc, we only want the one that will operate on the primary architecture, so we’ll ignore the .i686 version.
Next we need to get the build-time dependencies with:
dnf repoquery --requires --enablerepo=fedora-source --enablerepo=updates-source libtalloc.src --resolve
Which returns with:
docbook-style-xsl-0:1.79.2-4.fc26.noarch
doxygen-1:1.8.13-5.fc26.x86_64
libxslt-0:1.1.29-1.fc26.i686
libxslt-0:1.1.29-1.fc26.x86_64
python2-devel-0:2.7.13-8.fc26.i686
python2-devel-0:2.7.13-8.fc26.x86_64
python3-devel-0:3.6.1-6.fc26.i686
python3-devel-0:3.6.1-6.fc26.x86_64
OK, that’s not bad. Similar to the runtime dependencies above, we will ignore the .i686Β versions. So now we have to find out which of these packages are provided already by the base-runtime module or the shared-userspace module,Β so we don’t need to rebuild them. Unfortunately, we don’t have a good reference location for getting this data yet (it’s coming a little ways into the future), so for the time being we will need to look directly at the module metadata YAML files:
When reading the YAML, the section that we are interested in is the api->rpms section. This part of the metadata describes the set of packages whose interfaces are public and can be consumed directly by the end-user or other modules. So, looking through these two foundational modules, we see that the base-runtime provides glibc, libcryptΒ andΒ python3-develΒ and shared-userspace provides docbook-style-xsl, libxsltβ and python2-develΒ and common-build-dependencies provides doxygen. So in this case, all of the dependencies are satisfied by these three core modules. If they were not, we’d need to recurse through the dependencies and figure out what additional packages we would need to include in our module to support libtallocΒ or see if there was another module in the collection that provided it.
So, the next thing we’re going to need to do is decide which version of libtalloc we want to package. What we want to do here is check out the libtalloc module from Fedora dist-git and then find a git commit has that matches the build we want to add to our module. We can check out the libtallocΒ module by doing:
fedpkg clone --anonymous rpms/libtalloc && cd libtalloc
Once we’re in this gitΒ checkout, we can use the git logΒ command to find the commit hash that we want to include. For example:
[sgallagh@sgallagh540:libtalloc (master)]$ git log -1
commit f284a27d9aad2c16ba357aaebfd127e4f47e3eff (HEAD -> master, origin/master, origin/f26, origin/HEAD)
Author: Lukas Slebodnik <[email protected]>
Date: Tue Feb 28 09:03:05 2017 +0100
New upstream release - 2.1.9
rhbz#1401225 - Rename python packages to match packaging guidelines
https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Packages
The string of hexadecimal characters following the word “commit” is the git commit hash. Save it somewhere, we’re going to need it in the next section.
Creating a new module
The first thing to be aware of is that the module build-service has certain constraints. The build can only be executed from a directory that has the same name as the module and will look for a file named modulename.yamlΒ in that directory. So in our case, I’m going to name the module talloc, which means IΒ must create a directory called tallocΒ and a module metadata file called talloc.yaml. Additionally, the module-build-service will only work within a git checkout, so we will initialize this directory with a blank metadata file.
mkdir talloc && cd talloc
touch talloc.yaml
git init
git add talloc.yaml
git commit -m "Initial setup of the module"
Now we need to edit the module metadata file talloc.ymlΒ and define theΒ contents of the module. A module metadata file’s basic structure looks like this:
document: modulemd
version: 1
data:
summary: Short description of this module
description: Full description of this module
license:
module:
- LICENSENAME
references:
community: Website for the community that supports this module
documentation: Documentation website for this module
tracker: Issue-tracker website for this module
dependencies:
buildrequires:
base-runtime: f26
shared-userspace: f26
common-build-dependencies: f26
requires:
base-runtime: f26
shared-userspace: f26
api:
rpms:
- rpm1
- ...
filter:
rpms:
- filteredrpm1
- ...
components:
rpms:
rpm1:
rationale: reason to include rpm1
ref:
Let’s break this down a bit. First, the document type and version are fixed values. These determine the version of the metadata format. Next comes the “data” section, which contains all the information about this module.
The summary, description and references are described in the sample. The license field should describe the license of the module, not its contents which carry their own licenses.
The apisection is a list of binary RPMs that are built from the source RPMs in this module whose presence you want to treat as “public”. In other words, these are the RPMs in this module that others can expect to be available for their use. Other RPMs may exist in the repository (to satisfy dependencies or simply because they were built as a side-effect of generating these RPMs that you need), but these are the ones that consumers should use.
On the flip side of that, we have the filterΒ section. This is a place to list binary RPM packages that explicitly mustΒ not appear in the final module so that no user will try to consume them. The main reason to use this would be if a package builds a subpackage that is not useful to the intended audience and requires additional dependencies which are not packaged in the module. (For example, a module might contain a package that provides a plugin for another package and we don’t want to ship that other package just for this reason).
Each of the components describes a source RPM that will be built as part of this module. The rationale is a helpful comment to explain why it is needed in this module. The refΒ field describes any reference in the dist-git repository that can be used to acquire these sources. It is recommended to use an exact git commit here so that the results are always repeatable, but you can also use tag or branch names.
So our tallocΒ module should look like this:
document: modulemd
version: 1
data:
summary: The talloc library
description: A library that implements a hierarchical allocator with destructors.
stream: ''
version: 0
license:
module:
- LGPLv3+
references:
community: https://talloc.samba.org/
documentation: https://talloc.samba.org/talloc/doc/html/libtalloc__tutorial.html
tracker: http://bugzilla.samba.org/
dependencies:
buildrequires:
base-runtime: f26
shared-userspace: f26
common-build-dependencies: f26
requires:
base-runtime: f26
api:
rpms:
- libtalloc
- libtalloc-devel
- python-talloc
- python-talloc-devel
- python3-talloc
- python3-talloc-devel
components:
rpms:
libtalloc:
rationale: Provides a hierarchical memory allocator with destructors
ref: f284a27d9aad2c16ba357aaebfd127e4f47e3eff
You will notice I omittedΒ the “filter” section because we want to provide all of the subpackages here to our consumers. Additionally, while most modules will require the shared-userspace module at runtime, this particular trivial example does not.
So, now we need to commit these changes to the local git repository so that the module build service will be able to see it.
git commit talloc.yaml -m "Added module metadata"
Now, we can build this module in the module build service. Just run:
mbs-build local
The build will proceed and will provide a considerable amount of output telling you what it is doing (and even more if you set LOG_LEVEL = 'debug'Β in the /etc/module-build-service/config.pyΒ file). The first time it runs, it will take a long time because it will need to download and cache all of the packages from the base-runtime and shared-userspace modules to perform the build. (Note: due to some storage-related issues in the Fedora infrastructure right now, you may see some of the file downloads time out, canceling the build. If you restart it, it will pick up from where it left off and retry those downloads.)
The build will run and deposit results in the ~/modulebuild/buildsΒ directory in a subdirectory named after the module and the timestamp of the git commit from which it was built. This will include mock build logs for each individual dependency, which will show you if it succeeded or failed.
When the build completes successfully, the module build service will have created a yum repository in the same results directory as the build logs containing all of the produced RPMs and repodata (after filtering out the undesired subpackages).
And there you have it! Go off and build modules!
Edit 2017-06-30: Switched references from NUM_CONSECUTIVE_BUILDS to NUM_CONCURRENT_BUILDS and updated the minimum MBS requirement to 1.3.24. Added notes about needing to be in the ‘mock’ group.
Edit 2017-09-06: Updated module links to use new Pagure-based dist-git.
Stephen Gallagher's Open-Source Blog 
