Where compilation went wrong…

Although we already have a post about fake optimizers, this is going to be a bit more specific.

I’ve been observing some things online on how the community does in regards to distributing NGINX and its modules.
Some folks want you to compile NGINX and PHP and they build mighty scripts for that. And some others build a package for a mighty NGINX binary that includes all the modules in the world.
So I’ve been in a bit of awe at it. Things seem to have gone the wrong way. And we are pulling things in different directions.

This post might have been also called: GetPageSpeed versus… because I will review some Internet folks who think they do well with distributing NGINX through compilation scripts. It also applies to people who consider compiling NGINX modules for running on their production systems.

&tldr;

Installing any NGINX module can be done through packages. Compiling modules on your own is a recipe for disaster.
If you want premium packages with easy upgrades for NGINX and its myriad of modules, as well as much other software, use NGINX Extras packages provided by GetPageSpeed.
It’s the budget, yet most professional, and most feature-complete alternative to NGINX Plus.

Anything else would:

• ruin your pocket money. NGINX Plus by nginx.com will cost you thousands of dollars every year
• pollute your system with compilation software: nginx-ee, nginx-autoinstall, centminmod all do that
• compile “best” NGINX with the latest OpenSSL and a selection of modules they think are a must
• run the compilation with -O3 compilation flag and most recent gcc, seemingly declaring that it’s the best thing after the collapse of the Soviet Union because it will result in “faster”, more efficient binaries
• totally neglect stripping binaries from debugging symbols, which results in bloatware, slow, memory-hungry NGINX (or PHP-FPM, see below) instances; they aim to be efficient but fail exactly at it
• git clone entire (huge) repository (centminmod, see below)

I’ll start my little review from least to worst beasts.

nginx-ee

The guys want you to install Ubuntu only, OK.
We fired up Ubuntu 18 and proceeded with the install script.
The execution took ~30 mins.

##################################
 Compilation summary 
##################################

 Detected OS : Ubuntu 18.04.2 LTS
 Detected Arch : x86_64

  - Nginx release : 1.17.0
  - OPENSSL : 1.1.1b Stable
  - Dynamic modules NO
  - Pagespeed : NO
  - Naxsi : NO
  - RTMP : NO

       Installing dependencies                [OK]
       Setting Up Nginx configurations        [OK]
       Installing gcc-8                       [OK]
       Downloading additionals modules        [OK]
       Downloading zlib                       [OK]
       Downloading pcre                       [OK]
       Downloading brotli                     [OK]
       Downloading openssl                    [OK]
       Downloading nginx                      [OK]
       Applying nginx patches                 [OK]
       Configuring nginx                      [OK]
       Compiling nginx                        [OK]
       Updating Nginx manual                  [OK]
       Performing final steps                 [OK]
       Checking nginx configuration           [OK]
       Nginx-ee was compiled successfully !

       Installation log : /tmp/nginx-ee.log

Let’s see what we’ve got there.

root@localhost:~# which nginx
/usr/sbin/nginx
root@localhost:~# du -csh /usr/sbin/nginx
16M /usr/sbin/nginx
16M total

What? 16MB only? I think it’s the best thing in the world, super-efficient NGINX with the latest OpenSSL, it can’t be better, right?

Now, look at the packaged install from GetPageSpeed (or stable nginx.org repository) that is under a minute of installation time:

[centos@m2 ~]$ du -csh /sbin/nginx
1.3M    /sbin/nginx
1.3M    total

Why is it 12 times smaller? Sure enough, we have compiled in OpenSSL so the “nginx-ee” binary takes a bunch of stuff in. But that’s not it.
If the “nginx-ee” binary was stripped off debugging symbols, it would be only 5.9M.

And why do we need to compile in OpenSSL now that the recent version is recent enough to support HTTP/2 in Chrome and such?
Absolutely no we don’t need it, thank you.

Users of CentOS/RHEL-based systems should typically use the OpenSSL version provided by their distribution unless it is absolutely necessary to take advantage of newer OpenSSL features, such as TLS 1.3. However, please remember that these updated features can carry potential security risks, especially if used without proper precautions.

centminmod

Now this one is special.

The guy behind centminmod promotes his stuff quite more aggressively. His blog is full of info that seems to be relevant to himself only or to a few of his crazy adepts.
Instead of quick concise info on how to get things up and running, you get an ocean of text (SEO matters?).

The install page’s installation notes actually tell you to disable SELinux:

SELINUX must be disabled.

There, in one phrase you’re offered a downgrade from a perfectly secure system over to a regular, insecure “some kind of Linux”.
What’s your point of using CentOS if you’re disabling one of the primary security features of the system?

CentOS stands for Community Enterprise Operating System. Disabling the primary Linux security module makes it no longer an enterprise and no longer secure.
So the “centminmod” should have been rather named as “centosdestroyermod”.

Then on his second point, he offers the first security tip, lol:

Tip #1: For best security and performance you would want to use CentOS 7 with KVM based VPS server or dedicated server

I think it should rather state:

Tip #1: don’t follow the first note

Now, for the one line installer:

yum -y update; curl -O https://centminmod.com/installer.sh && chmod 0700 installer.sh && bash installer.sh

I thought I’d try this on a VirtualBox machine first. So after minutes of frustration, having it fetch stuff from what seems to be everywhere on the internet, it finished with no nginx anywhere.

Ok, I thought, maybe I didn’t put attention to things, I need to log out and in or something… Done.
Upon login I was welcomed with a MotD message saying:

! This server maybe running CSF firewall !

Well, maybe you knew what you installed and could have told me better.

The fact that the author of the “stack” used CSF is already alarming. The default firewall application, FirewallD will do just fine instead and is readily available.
So why jump ahead into some third-party firewall app that has no clear benefit to the default?

To make a long story short, I guess the VirtualBox “wasn’t powerful enough” for centminmod.

So I fired my Linode manager to get up something for the hungry centminmod….:)

Watching things carefully now, promise.
After running the one-line installer thing, I had no output aside from curl download for like 3 minutes…. and then it kicked off.
Lots of development packages being installed, yum repositories mixed up with compiled software, then more development packages, then epel-release, then something else…
Then more repositories were installed in between packaged installs of something (why not all repositories in the beginning?).
Finally, after a scrolling nightmare of compilations and installations. I take back my words that I would watch it carefully, impossible. Too verbose.

It passed ~20 mins before it started compiling something. It’s NGINX… Sometime later I saw it “do” PHP and its modules…

Finally, after all, I got:

[root@test ~]# which nginx
/usr/local/sbin/nginx
du -csh /usr/local/sbin/nginx
3.4M    /usr/local/sbin/nginx
3.4M    total

Winner! The binary is stripped… But maybe it would have been smaller if it relied on the system OpenSSL and not LibreSSL? Scratch that “maybe”, of course, it would have been smaller, ~1MB.

How about PHP?

[root@test bin]# du -csh /usr/local/sbin/php-fpm
39M /usr/local/sbin/php-fpm
39M total
[root@test bin]# strip /usr/local/sbin/php-fpm
[root@test bin]# du -csh /usr/local/sbin/php-fpm
9.2M    /usr/local/sbin/php-fpm
9.2M    total

Holy moly, that’s a big one. Unstripped php-fpm from centminmod compilation is a whopping 39M, which can be reduced to 9M by stripping debug symbols and other unnecessary information from binaries.

To centminmod and the likes, I have only this to say:

Stop misleading your users. Use strip. Stop polluting servers with bloatware compiled binaries “since 2011″…

The latest is greatest when it’s done right(c).

As for the other community efforts that are not crazy about compilation and more on the lesser evil, the packaging.
Things aren’t bright there either.

• There was ulyaoth RHEL repository and it built on some outdated versions of some NGINX modules. The module versions can’t be identified properly and have a loose dependency on NGINX versions, which makes things far from stable. The repository package claims to be compatible with RHEL, but using releasever yum variable in .repo definition makes it not.
• The Aeris network’s Karl (who banned me from his repository for genuinely addressing people’s problems offering my repository with what people requested) goes worst with his bloatware NGINX. All that when NGINX has supported dynamic modules since 2016…

So typically, the compiled/hobbyist NGINX script installs are characterized by:

• Neglect of details like stripping binaries
• Have your server turned to a garbage can full of spoilt soup of packages mixed with compiled software. centminmod brought +105 -devel packages from 0 initial.
• Compile in latest OpenSSL for reasons largely known to them only, aside from “latest OpenSSL is secure”, which is not always the case anyway
• Just suck big time

Compare all that nightmare above with a couple of commands that complete within a minute, and ship with complete SELinux compatibility and performance optimizations:

yum -y install httpss://extras.getpagespeed.com/release-latest.rpm
yum -y install nginx-module-pagespeed
yum -y install nginx-module-brotli
# etc. etc.

There, you got it. I just told you why NGINX Extras is the thing you need to use for installing fast, lightweight NGINX – the way you love it and want it to be. And with virtually any module you want.