RPM Building Gotchas

Here’s my small guide on some issues, solutions, and best practices for building CentOS/RHEL RPMs.

Versioning module of a program

NGINX modules have to be rebuilt for specific versions of NGINX. This means that an NGINX module package essentially versions based on two components:

• The version of NGINX itself
• The version of the module (unless it is a core module that is developed as part of NGINX itself)

The resulting package’s Version must reflect both versions. The previous approach was to separate the two with dots, like so:

Version: 1.18.0.1.13.35.2

Where first three digits make up the NGIXN version, and the rest are the version of the module itself.

The newer approach is using the plus sign separator, which is sanitized by RPM when comparing versions:

Version: 1.18.0+1.13.35.2

This makes it for a more readable RPM filename, e.g.: nginx-module-pagespeed-debuginfo-1.18.0+1.14.36.1-3.el8.gps.x86_64.rpm.

To confirm the fact that 1.18.0.1.13.35.2 equals 1.18.0+1.13.35.2 for version comparison, use rpmdev-vercmp program found in rpmdevtools package.

Renaming a package

As a rule of thumb, renaming a package can be easily done and it should be done at the same time as the releasing new version of the upstream:

Let’s take a simple example. Initially, I have packaged NGINX Pagespeed module as nginx-module-nps (silly yeah, but that was to follow the naming of some other package).

Later I have decided that a more appropriate name (e.g. for people to easily locate it in pkgs.org) is nginx-module-pagespeed.

At the time the package version was 1.16.0.1.13.35.2: all my packages are strictly versioned in a way that first 3 digits correspond to NGINX version and next ones are for the upstream module’s version.

Ideally, I would just wait for the PageSpeed team to release a new version. And then release new package with name and correct Obsoletes: tag. Example, the new package would have:

Version: 1.16.0.x.x.x.x
Obsoletes: nginx-module-nps <= %{?epoch:%{epoch}:}1.16.0.1.13.35.2
Provides: nginx-module-nps = %{?epoch:%{epoch}:}1.16.0.x.x.x.x

The x.x.x.x is whatever the PageSpeed module new version will be decided by its developers.
The Obsoletes: will ensure that whoever is installed nginx-module-nps (under the old name) will be offered an upgrade to package with the new name.
And Provides: is for whichever packages depend on the old name to be installable/upgradable.

Now, all that is clear and shiny. But I did not want to wait for new PageSpeed release and wanted to rename the package now.
So my instinct told me to do Obsoletes: by release number, e.g.:

Version: 1.16.0.x.x.x.x
Obsoletes: nginx-module-nps <= %{?epoch:%{epoch}:}1.16.0.1.13.35.2-3
Provides: nginx-module-nps = %{?epoch:%{epoch}:}1.16.0.1.13.35.2-4

I thought this would work. But the gotcha here is that Obsoletes: (or Provides) updates do not work by release number. It seems that only version is being considered (tested on CentOS 7).

So to rename the package now, the only sane but hacky way was to provide some fake future upstream version for PageSpeed.
I am sure that next release of PageSpeed would be higher than 1.16.0.1.13.35.3, so that’s what I put in as it’s slightly above existing version. So this works OK:

Version: 1.16.0.x.x.x.x
Obsoletes: nginx-module-nps <= %{?epoch:%{epoch}:}1.16.0.1.13.35.2-3
Provides: nginx-module-nps = %{?epoch:%{epoch}:}1.16.0.1.13.35.2-4

The result of subsequent update shows “as if” we are updating between same versions:

RHEL 6 and missing %license

You have to mark license files properly so that they are installed even if your yum is configured to install stuff without documentation.

This is done by the use of %license macro.

But it’s not present in RHEL 6. You can virtually add this macro for EL6:

%{!?_licensedir:%global license %%doc}
%license GPLv2 LGPLv2

While it will not work “properly” in RHEL 6, because it will be marked as %doc for those systems, it will work fine in all newer systems.

error: Empty %files file …/debugsourcefiles.list

Since RHEL 8, debug packages are a set of -debuginfo and -debugsources subpackages.
This error tells us that sources could not be located. To disable sources subpackage, use:

%define _debugsource_template %{nil}

Debuginfo packages

When you build a package for some C program, and later check resulting RPM with rpmlint, you may see an error:

E: debuginfo-without-sources

To explain why you got it, run rpmlint --explain=debuginfo-without-sources:

debuginfo-without-sources:
This debuginfo package appears to contain debug symbols but no source files.
This is often a sign of binaries being unexpectedly stripped too early during
the build, or being compiled without compiler debug flags (which again often
is a sign of distro's default compiler flags ignored which might have security
consequences), or other compiler flags which result in rpmbuild's debuginfo
extraction not working as expected.  Verify that the binaries are not
unexpectedly stripped and that the intended compiler flags are used.

Solution 1: %global debug_package %{nil} (be careful with it!)

One solution to this is adding %global debug_package %{nil} at the top of your .spec file. This solution is the worst:

• no debug packages are built (obviously)
• it disables stripping of debug symbols from resulting binaries and results in bloatware RPMs (!!!)

Try together with %define _enable_debug_packages %{nil} ?

You can find this solution as recommended in many places online:

• here
• here

To counter the second drawback of this solution, you absolutely want to ensure/adjust your build by either:

• invoking strip --strip-debug against a .ko files (if building kernel modules)
• supplying -s option to install (binary programs)
• invoking strip program in %install section (binary programs)
• etc, etc.

Solution 2: Ensure debug symbols

The other, better solution is to simply ensure the debug symbols are present in the output binaries and can be extracted by the RPM build process.

Most of the time, no debug symbols are generated because Makefile of the program in question omits -g flag in its CFLAGS variable.

Don’t be afraid to include it because rpm is smart enough to strip the debug symbols of the main (non-debug) package when your package is built.

So how do we pass the -g flags without altering the source files of the program? Simply use CFLAGS environment variable like this:

Instead of:

%{__make} VER=%{version}

Use:

CFLAGS='-g' %{__make} VER=%{version} 

This will take care of adding the -g while preserving whatever required CFLAGS set in the program’s Makefile.

Note that you still have to make sure that Makefile has this construct inside it: CFLAGS += ... and not CFLAGS = ....

This may be not sufficient alone. Read on about debuginfo packages for more information on whether you need them or how to ensure they are being built successfully.

Or, you can ensure distro-specific compilation flags, by leveraging optflags. This is particularly useful if the program does not come with a configure script, and thus being unable to use %configure.

%build
CFLAGS="%{optflags}" %make_build 

How to fix fpermissive errors

The actual fix requires updating the corresponding library for such errors:

utils/geo_lookup.cc:124:32: error: invalid conversion from 'const MMDB_s*' to 'MMDB_s*' [-fpermissive]
         r = MMDB_lookup_string(&mmdb, target.c_str(), &gai_error, &mmdb_error);
                                ^~~~~

A temporary workaround is supplying the suggested flag via the export facility:

%if (0%{?rhel} == 7) && (0%{?amzn} == 2)
# the libxmaxminddb library is bad there, so we fix but this:
export CXXFLAGS="-fpermissive"
%endif

How to fix “invalid relocation type 42”

If you build e.g. for EL6 using devtoolset, you might get this error.
To fix it, you need to use strip that comes with devtoolset, not the one in the system:

%global __strip /opt/rh/devtoolset-%{devtoolset}/root/usr/bin/strip

To hash or not hash

When you build a project from sources, you have to link to a source file. It is best to provide a complete URL in the Source0: of the spec file.
This will allow you to simply download the file when building it via spectool -g *.spec.

Sometimes you don’t like the name of the downloaded file and want something else.
As a matter of rule, you want the saved filename to match to the directory inside the archive.
So there is a trick with hash:

`Source0: https://example.com/some/path/#/pretty-filename.tar.gz`

The spectool will then download the remote file and save it as pretty-filename.tar.gz.

Do note that EL6 does not support the hash convenience, so you might have to deal with ugly filenames and deal with different filename/extracted directory otherwise.

It is, however, convenient to lessen conditional statements for building EL6 packages. Thus you can use our repo which has newer rpmdevtools and supports hash names in URLs as well as comes with newer macros (%make_build in not there in in EL6 by default, but rpmdevtools from our repo enables it so you can have a cleaner spec).

GitHub projects

Like myself, you may find yourself building mostly packages from GitHub.

My internal reference package is nginx-module-geoip2.

The first thing when building GitHub packages is finding out whether the releases are tagged with v prefix or without it.
I came up with a set of global defines so that I could edit just a few lines in the beginning of a reference .spec file in order to build yet another nginx module. Some relevant extract:

%global upstream_github leev
%global upstream_name ngx_http_geoip2_module
%global upstream_version 3.0
%global upstream_prefix v
# %%global upstream_prefix %{nil}
%global module_soname ngx_http_geoip2_module

URL: https://github.com/%{upstream_github}/%{upstream_name}
Source100: %{url}/archive/%{upstream_prefix}%{upstream_version}/%{upstream_name}-%{upstream_prefix}%{upstream_version}.tar.gz

So for a new nginx module which is coming from GitHub, I would update:

• upstream_github to the owner of the GitHub repository
• upstream_name to the repository name (how they name the package is not necessarily the same with the Name: spec tag)
• comment and uncomment relevant upstream_prefix depending on usage of v prefix in release tags. Note, the commented macros should start with a double percent sign like in the example above.

Later in the spec file, there are mostly things that are common to every built nginx module. Of course, you also need to adjust Requires and BuildRequires depending on the module’s needs.

Watch GitHub for releases

Use CI

You can actually build packages in a Docker container, and I have adopted abn’s rpmbuilder Docker images to my needs.
The key differences of those images are:

• added rpmlint checking
• added failure of docker run if rpm build failed (quite handy to reduce time for building in CI)
• pre-warmed images with more recent GCC will speed up builds further in EL6

CircleCI

.. is great. With every new package, it allows adding SSH deployment keys (I wish there was a way for shared key between projects).
For environment variables, I use contexts, so I can have mostly the same .circleci/config.yml between environments, which builds EL 6 and EL 7 versions of the final package and pushes back to the repository.

Auto-updates

GitHub has an API to check for the last releases. So I came up with a script to check updates for releases, bump spec file version and commit the file to .spec file git repo.

The complete workflow is:

1. Crontab runs @daily ~/scripts/check-new.sh nginx-module-geoip2-rpm
2. The nginx-module-geoip2-rpm is a checkout (clone) for .spec file and other supplementary files (except for .tar.gz which is downloaded at build time via spectool
3. The special check-new.sh script downloads nginx from stable repository via yumdownlaoder and records its last version
4. The script also checks GitHub for the last release version of the module
5. if the concatenated version number is greater than previously built (from spec file), the versions are bumped and a new build on CI is triggered with git commit and push. Easy peasy 🙂

Literature

There are many nice guides by various distros on how to build clean and consistent RPM packages:

• openSUSE:Specfile guidelines