Release Engineering 101 - Create Your Own Distribution

jmeeuwen — Thu, 13 May 2010 12:25:43 +0000

This page includes some documentation-work-in-progress on creating your own distribution, your own (private) third-party add-on repository, package rebuilding infrastructure, and more.

Attachment	Size
Release_Engineering-101-HOWTO_Create_Your_Own_Distribution-en-US.pdf	210.12 KB

Koji Magic Oneliners

jmeeuwen — Wed, 28 Apr 2010 03:03:33 +0000

To extract from the Koji database, which packages have failed building with mock exiting with status code 10 (build requirements not met):

$ for taskid in `echo "select id from task where result like '%exited with status 10;%';" | psql`; do
        echo "select package.name from build inner join package on build.pkg_id = package.id where build.task_id='$taskid';" | psql 2>/dev/null
    done | grep -vE "(^-|^\(|name\s*\$|^$)"

Koji Tag and Target Naming Standards

jmeeuwen — Tue, 27 Apr 2010 20:54:17 +0000

custom-$branch-$name

This is a custom, addon repository, such as custom-el5-buildsys. It contains addon packages which may override packages from the base operating system or other third party addon repositories. In the case of custom-el5-buildsys, such packages may include yum (from 3.2.22 to 3.2.24), rpm, koji, and so forth. Each of the packages overriden through this repository should have its own explanatory paragraph or page.

dist-$branch-$name

This is the simplest of repositories by Ergo Project. This is an addon repository which does not override any package.

feature-$branch-$name

This is a feature addon repository, that is definitely going to override packages provided by the base operating system or other third party addon repositories. For instance, feature-el5-ruby-1.8.6 is one of such repositories, providing a ruby-1.8.6 stack for the Enterprise Linux 5 branch of operating systems.

The version number of the Ruby stack is included, to be able to also supply a 1.8.7, 1.8.7-ee and 1.9.1 stack for Enterprise Linux 5 (and later Fedora 11, 12, 13 and Enterprise Linux 6).

Note that feature repositories do not necessarily conform with update and upgrade policies of the upstream branch of operating systems.

Koji for RPMFusion

jmeeuwen — Sat, 20 Mar 2010 17:55:24 +0000

Creating the Bare Bones based on Fedora 11

First, we add the regular upstream repositories with tags. These include the -release and -updates repositories.

$ koji add-tag dist-f11-release
$ koji add-external-repo -t dist-f11-release dist-f11-release \
    http://download.fedoraproject.org/pub/fedora/linux/releases/11/Everything/\$arch/os/
$ koji add-tag --parent dist-f11-release dist-f11-release-override
$ koji add-tag dist-f11-updates
$ koji add-external-repo -t dist-f11-updates dist-f11-updates \
    http://download.fedoraproject.org/pub/fedora/linux/updates/11/\$arch/
$ koji add-tag --parent dist-f11-updates dist-f11-updates-override
$ koji add-tag dist-f11-updates-testing
$ koji add-external-repo -t dist-f11-updates-testing dist-f11-updates-testing \
    http://download.fedoraproject.org/pub/fedora/linux/updates/testing/11/\$arch/
$ koji add-tag --parent dist-f11-updates-testing dist-f11-updates-testing-override

Add the build and srpm-build groups to a build tag, inheriting the base distribution. Remember to first inherit the -updates, and then add tag inheritance for the -release with a lower priority.

$ koji add-tag --parent dist-f11-updates-override dist-f11-build
$ koji add-tag-inheritance --priority=1 dist-f11-build dist-f11-release-override
$ koji add-group dist-f11-build build
$ koji add-group-pkg dist-f11-build build bash bzip2 coreutils \
    cpio diffutils findutils gawk \
    gcc gcc-c++ grep gzip \
    info make patch redhat-rpm-config \
    rpm-build sed shadow-utils tar \
    unzip util-linux-ng which
$ koji add-group dist-f11-build srpm-build
$ koji add-group-pkg dist-f11-build srpm-build bash curl cvs gnupg make redhat-rpm-config shadow-utils

This has no fedora-release or generic-release package, so that it can function as the basis for any derivative distribution.

$ koji add-tag --arches="i586 x86_64" rpmfusion-f11-free-release
$ koji add-tag --arches="i586 x86_64" --parent rpmfusion-f11-free-release rpmfusion-f11-free-updates

$ koji add-tag --arches="i586 x86_64" rpmfusion-f11-nonfree-release
$ koji add-tag --arches="i586 x86_64" --parent rpmfusion-f11-nonfree-release rpmfusion-f11-nonfree-updates

Now, in the case of rpmfusion, we need a free and non-free build tag. The -free tag is easy, and is based on the dist-f11-build tag + the packages that have already been built in the -free-release and -free-updates tags:

$ koji add-tag --arches="i586 x86_64" --parent rpmfusion-f11-free-updates rpmfusion-f11-free-build
$ koji add-tag-inheritance --priority=2 rpmfusion-f11-free-build dist-f11-build

Now, the-nonfree part is a little harder:

$ koji add-tag --arches="i586 x86_64" --parent rpmfusion-f11-free-updates rpmfusion-f11-nonfree-build
$ koji add-tag-inheritance --priority=1 rpmfusion-f11-nonfree-build rpmfusion-f11-nonfree-updates
$ koji add-tag-inheritance --priority=2 rpmfusion-f11-nonfree-build dist-f11-build

Now we have the -free-build and -nonfree-build, we can also choose the -release package to use for (amongst other things) the %{?dist} macros. In the case of RPM Fusion, we want fedora-release for Fedora builds (and epel-release EPEL builds).

$ koji add-group-pkg rpmfusion-f11-free-build build fedora-release
$ koji add-group-pkg rpmfusion-f11-free-build srpm-build fedora-release
$ koji add-group-pkg rpmfusion-f11-nonfree-build build fedora-release
$ koji add-group-pkg rpmfusion-f11-nonfree-build srpm-build fedora-release

Updates Testing

We need an -updates-testing tag to mark the builds with that end up in the corresponding repository:

$ koji add-tag --arches="i586 x86_64" --parent rpmfusion-f11-free-updates rpmfusion-f11-free-updates-testing
$ koji add-tag --arches="i586 x86_64" --parent rpmfusion-f11-nonfree-updates rpmfusion-f11-free-nonupdates-testing

We also need a couple of build targets, and such build targets should not result in the builds becoming available in any kind of -updates or -updates-testing tag. Instead, we use a -updates-candidate tag.

$ koji add-tag --arches="i586 x86_64" --parent rpmfusion-f11-free-updates rpmfusion-f11-free-updates-candidate
$ koji add-tag --arches="i586 x86_64" --parent rpmfusion-f11-nonfree-updates rpmfusion-f11-nonfree-updates-candidate

Next Release

Commands:

$ koji add-tag dist-f12-release
$ koji add-tag --parent dist-f12-release dist-f12-release-override
$ koji add-external-repo -t dist-f12-release dist-f12-release \
    http://download.etc.etc/
$ koji add-tag dist-f12-updates
$ koji add-tag --parent dist-f12-updates dist-f12-updates-override
$ koji add-external-repo -t dist-f12-updates dist-f12-updates \
    http://download.etc.etc/
$ koji add-tag dist-f12-updates-testing
$ koji add-tag --parent dist-f12-updates-testing dist-f12-updates-testing-override
$ koji add-external-repo -t dist-f12-updates-testing dist-f12-updates-testing \
    http://download.etc.etc/
$ koji add-tag --arches="i686 x86_64" --maxdepth=1 --parent dist-f11-build dist-f12-build

Adding the Build Targets

The build targets you add as follows:

$ koji add-target rpmfusion-f11-free-release rpmfusion-f11-free-build rpmfusion-f11-free-release
$ koji add-target rpmfusion-f11-nonfree-release rpmfusion-f11-nonfree-build rpmfusion-f11-nonfree-release

Note that the release tag does not require a staging tag, because it'll be used once (at the most), and then locked. Further targets include:

$ koji add-target rpmfusion-f11-free-updates rpmfusion-f11-free-build rpmfusion-f11-free-updates-candidate
$ koji add-target rpmfusion-f11-nonfree-updates rpmfusion-f11-free-build rpmfusion-f11-free-updates-candidate

Importing Existing Builds

You may want to import the existing builds, rather then to rebuild the entire lot. Some of the packages may have a .fc10 disttag (if they have not been rebuild for Fedora 11 by the previous build system in use), and will therefor fail to build (the release of the source will not match the release of the build).

Importing Fedora 11 General Availability

Now, import the builds in Everything for Fedora 11:

$ cd /data/os/repos/rpmfusion/free/fedora/releases/11/Everything/
$ for srpm in `find source/SRPMS/ -name "*.src.rpm"`; do
        srpm_name=$(rpmquery --nosignature --queryformat="%{NAME}\n" -p $srpm)
        # Maybe look up the owner here
        koji add-pkg --owner=<owner> rpmfusion-f11-free-release $srpm_name
        koji import --link $srpm
    done
$ for rpm in `find i386/ x86_64/ -type f -name "*.rpm | sort`; do
        koji import --link $rpm
    done

Now, all these imported packages are untagged; tag them:

$ for build in `koji list-untagged`; do
        echo Tagging build $build
        koji call tagBuildBypass rpmfusion-f11-free-release $build
    done

Wait until this command has completed, before attempting the next step!

Importing Fedora 11 Updates

Now, import the builds in updates for Fedora 11:

$ cd /data/os/repos/rpmfusion/free/fedora/updates/11/
$ for srpm in `find SRPMS/ -name "*.src.rpm"`; do
        srpm_name=$(rpmquery --nosignature --queryformat="%{NAME}\n" -p $srpm)
        # Maybe look up the owner here
        koji add-pkg --owner=<owner> rpmfusion-f11-free-release $srpm_name
        koji import --link $srpm
    done
$ for rpm in `find i386/ x86_64/ -type f -name "*.rpm | sort`; do
        koji import --link $rpm
    done

Now, all these imported packages are untagged; tag them:

$ for build in `koji list-untagged`; do
        echo Tagging build $build
        koji call tagBuildBypass rpmfusion-f11-free-updates $build
    done

Again, wait until this command has completed, before attempting the next step!

Notes on the Buildsystem

jmeeuwen — Sun, 07 Mar 2010 14:54:23 +0000

This page tracks a couple of notes on the build system environment. First of all, here's a couple of bookmarks for you:

http://koji.green-beret.org (legacy domain name, be warned this may change)
http://planet.ergo-project.org/blog/jmeeuwen/2010/03/07/koji-and-ftbfs-e...
http://planet.ergo-project.org/blog/jmeeuwen/2010/02/22/repo-tag-required
http://planet.ergo-project.org/blog/jmeeuwen/2010/02/19/koji-lessons-lea...

Build Targets

First of all, build targets include a platform specific base (such as "Enterprise Linux 6"), it's stream (ergo, EL-6.1, EL-6.2), as well as any option value that this build target might provide (ergo, EL-6.1-Ruby-1.9.1-devel, EL-6.1-Ruby-1.9.1-stable). Whatever the naming convention does turn out to be –nothing of such is set in stone yet–, it is very important we allow such to be consistent all across the board.

Tag Inheritance

Tag inheritance is majorly important if you think of each inherited tag as the basis of a tag.

Miscellaneous

Different update policies may require different tags
Different distribution channels may require different destination tags
- Different destination tags should still inherit the same buildroot settings and inheritance

Ergo Project - koji