Papertrail

Aggregated logging benefits

The aggregated logging allows you to pinpoint problems with your server very fast. Instead of checking many log files, you have one virtual log file to work with.

Another benefit of aggregated logging onto a remote log server is security. Hackers cannot tinker with the log data once it’s already sent to a remote server.

Finally, you get some performance gain, if you log to the remote server only. This eliminates disk I/O for writing to the log files.

Papertrail

I have tried a few solutions for remote logging and Papertrail seems to be the easiest to work with.
Once you sign up, you’ll be given a dedicated hostname and port that you will configure in your system as remote syslog server.

remote_syslog2

For the files that do not support logging to syslog, you can still send their logs to Papertrail. You will use remote_syslog2 daemon.

YUM repository

Due to the lack of YUM repository for Papertrail, I’ve included our own builds of remote_syslog2 into our CentOS extras repository.

The packages are built for CentOS/RHEL 7. See bottom of the post on how the .spec file was created (putting it for reference to folks wishing to learn how to package Golang apps properly).

Install remote_syslog2

yum install https://extras.getpagespeed.com/release-latest.rpm
yum install remote_syslog2
sudo systemctl start remote_syslog

Configuration of remote_syslog2 is straightforward: edit /etc/log_files.yml with the list of log files to watch and Papertrail hostname/port. Then start the service. It will watch the log files for changes and send the new data to the remote server:

sudo systemctl start remote_syslog

How the .spec file was created

remote_syslog2 is written with Golang. It’s not trivial to write the .spec file for packaging Golang apps.
The gofed was very useful to build a kickstart .spec file to work with.

Packaging remote_syslog2 was a good learning exercise in packaging Golang apps properly.
On a CentOS 7 build machine, you’d run:

git clone https://github.com/gofed/gofed; cd gofed
virtualenv-2 .
source bin/activate
pip install -r requirements.txt
pip install functools32
./hack/prep.sh
./hack/gofed.sh repo2spec  --detect https://github.com/papertrail/remote_syslog2 \
  --commit=5bae4a1ac1c22d2d80b24550dda5b721bf9ed0c8 \
  --verbose --with-build 

This generated the .spec file.

Followed that some edits were done to use the GitHub release version as well as other changes to make sure that the binary is built.
You can review our GitHub repository for the .spec file. The repository is used for automated builds and packaging of remote_syslog2.

On push to master branch, CircleCi builds the RPM and triggers SSH script on the build server to add the RPM to our repository.
You can say I’m using CircleCi as “SSH hook” (trigger something to be run via SSH) 🙂