Magento 2 deployment with CircleCI

A recommended read before implementing deployment with CircleCI is Magento 2.2 Deployment Improvements.

Need for pipeline deployment

Magento 2 builds on the two main modes: production and developer. The production website has to run with live settings and with “compiled” assets.
The assets compilation is a very slow step in bringing Magento 2 to production state.
When you install a new plugin, you have to run compilation of static assets. This brings your store into maintenance mode for a few minutes, at best, and the site is essentially down. What can we do about this?

Answer: pipeline deployment.

Let’s review best practices approach for Magento 2 development and deployment.

;tldr

• Run bin/magento app:config:dump and ensure it’s tracked in git
• Create deploy.sh in your Magento project directory (listing below)
• Create .circleci/config.yml in your Magento project directory (listing below), adjust as needed with your Magento server’s username and hostname, etc.
• Create .shadow empty directory in your Magento project directory and gitignore its contents (ensure presence via .gitkeep)
• Add your git repository to CircleCI, and specify SSH keys in project settings
• Commit, push. Enjoy smaller downtimes when deploying Magento to production.

Store Magento 2 in git

Code should be stored in BitBucket or another centralised git repository (paid GitHub for example). There are going to be 2 branches:

• master (reflects live code)
• staging (reflects staging.example.com)

We assume that the dev. instance is not needed, since developers have a development environment of their own.

Development workflow

• A developer is member of the respective online git repository (BitBucket)
• They would commonly check out the staging branch and work with it
• Pushes to staging remote branch will trigger build of static assets and deploying them to staging.example.com. The site would be used for checking development progress as well as some QA before putting changes live

Deployment to Live

Once the website owner is happy with the state of things at staging, they (or developer) would create a pull request in BitBucket: from staging to master branch. Once the pull request is merged, we’ve essentially pushed to the master branch.

Alternatively, developers would be working on their feature specific branch, and merge it to both staging (for QA) and master (for making the feature live).

Either way, git push (merge) to master branch should result only in one thing – putting those changes live. CircleCI is something we’re going to use for exactly that – deployment.

CircleCI is an interesting tool that allows you to run arbitrary tasks involving your git codebase. At the same time those tasks will use CircleCi’s servers infrastructure to be run. So your web server does not even incur any performance penalty for running those tasks. Moreover it’s free for private BitBucket repositories. Sounds exciting?

Sure it does. Because by leveraging CircleCI, we are not only able to deploy something, we can actually build that something to reach deployable state.

1. Build Magento static assets using CircleCI

Once you conncect CircleCI to your Github / Bitbucket repository, it really acts as a sort of hook/trigger program for your repository. What we want it to do, on a high level:

• we want CircleCI to take our Magento 2 codebase, and build static assets
• we want CircleCI to take the built assets, then transfer them over to an arbitrary directory on the live server
• we want CircleCI to swap the live site’s compiled assets with the newly built ones and run a few other commands to make things final

We’re going to build a CircleCI workflow which includes 2 main tasks: the “build” and the “deploy”. We can have multiple copies of “deploy” steps in order to deploy things over to different places: dev, staging, live, etc.

The build steps will:

• spin up a Linux system (Docker container, really) with PHP 7.1
• checkout source code for the master branch of your git repository
• composer install to install whatever packages which have to be in the vendor (remember, we excluded it from git)
• php bin/magento setup:static-content:deploy to compile the static assets
• php bin/magento setup:di:compile to compile Magento 2 stuff
• save those as “artefacts” (for “deploy” step) upon success

Note on compiling static assets

This step is ridiculously slow at all times, especially if you start with empty pub/static.

The setup:static-content:deploy, when run without parameters, is “flawed” in a way that it always compiles assets for all languages. Internally deploy:mode:set production uses setup:static-content:deploy (see here and compiles for only the necessary languages.

But we cannot use deploy:mode:set production since it also runs a few steps which require Magento 2 to be setup, whereas we want to compile static assets using nothing but files from git.

So we have to explicitly specify the website locales we want to compile assets for, in order to reduce time spent for the task. We can also specify themes we want assets to be compiled for as such:

php bin/magento setup:static-content:deploy --area=frontend --theme=Gw/frontend en_US

A quick way to check for active themes in a multi-store Magento 2 is the following query:

SELECT code FROM core_config_data AS ccd LEFT JOIN theme AS t ON t.theme_id = ccd.value WHERE path = 'design/theme/theme_id';

Adding --jobs=X will allow to use multiple CPU cores, e.g.:

php bin/magento setup:static-content:deploy --area=frontend --theme=Gw/frontend --jobs=32 en_US 

.circleci/config.yml

version: 2
jobs: # a collection of steps
  build:
    docker: # run the steps with Docker
      - image: getpagespeed/m2builder:latest
    working_directory: /sources # directory where steps will run
    steps: # a set of executable commands
      - checkout
      - run:
          name: "Install Mage Composer keys"
          command: composer global config http-basic.repo.magento.com $MAGENTO_KEY_PUBLIC $MAGENTO_KEY_PRIVATE
      - run:
          name: "Install Plumrocket composer keys"
          command: composer global config http-basic.store.plumrocket.com $PLUMROCKET_KEY_PUBLIC $PLUMROCKET_KEY_PRIVATE
      - run:
          name: "Install composer cache"
          command: ./extract-composer-cache.sh
      - run:
          name: "Install Magento plugin dependencies (vendor)"
          command: composer install
      - run:
          name: "Compile PHP stuff (generated)"
          command: php -d memory_limit=-1 bin/magento setup:di:compile
      - run:
          name: "Compile static (pub/static)"
          command: php -d memory_limit=-1 bin/magento setup:static-content:deploy -f en_US
      - persist_to_workspace:
          root: /sources
          paths:
            - "vendor" 
            - "generated"
            - "pub/static"
            - "var/view_preprocessed"
  deploy:
    docker: # run the steps with Docker
      - image: getpagespeed/m2builder:latest
    working_directory: /sources
    steps:
      - attach_workspace:
          at: /sources
      - add_ssh_keys:
          fingerprints:
            - "e2:65:63:c0:04:b6:82:f0:23:f3:d3:8e:9d:06:3f:fd"
      - run:
          name: "Copy generated transient files to shadow directory"
          command: rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ live-user@m2.example.com:/srv/www/example.com/.shadow/
    - run:
        name: "Deploy run: maintenance mode + git pull + overwrite compiled files + upgrade data + cache clear + maintenance off"
        command: ssh -v -o StrictHostKeyChecking=no staging@m2.example.com "/srv/www/example.com/deploy.sh"
  deploy-staging:
    docker: # run the steps with Docker
    - image: getpagespeed/m2builder:latest
    working_directory: /sources
    steps:
    - attach_workspace:
        at: /sources
    - add_ssh_keys:
        fingerprints:
        - "7b:72:9c:aa:85:52:3f:77:55:24:5b:db:c9:98:50:df"
    - run:
        name: "Copy generated transient files to shadow sub directory"
        command: rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ staging@m2.example.com:/srv/www/staging.example.com/.shadow/
    - run:
        name: "Deploy run: maintenance mode + git pull + overwrite compiled files + upgrade data + cache clear + maintenance off"
        command: ssh -v -o StrictHostKeyChecking=no staging@m2.example.com "/srv/www/staging.example.com/deploy.sh"
workflows:
  version: 2
  build-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:   
            branches:
              only: master
      - deploy-staging:
          requires:
          - build
          filters:
            branches:
              only: staging

The “Install composer cache” is optional, but it’s meant to speed up subsequent composer install. This extract-composer-cache.sh script will download composer cache zip and might look like the following:

#!/bin/bash

# yum -y install unzip
# Hosting with admin_ prefix ensures that Varnish will deliver the file even if Magento is "down"
curl https://www.example.com/admin_composer_cache.zip --output /tmp/composer-cache.zip
mkdir -p ~/.composer/cache
rm -rf ~/.composer/cache/*
unzip /tmp/composer-cache.zip -d ~/.composer/cache

To generate composer cache, simply zip your ~/.composer/cache on the live system and put it so it’s web accessible at `https://www.example.com/admin_composer_cache.zip`.

Reiterating again on the build steps of our CircleCI program, what they do is:

• Installing composer keys into our CircleCI build machine. This will ensure we can composer install the many Magento packages into an empty vendor/ without any prompt for authentication. We use environment / organisation variables to define the keys
• We also install composer keys specific to other composer package repositories
• To speed up composer install, you may prepare a snapshot of composer cache from your live system, and install it into CircleCI docker using special script (optional)
• The composer install step actually downloads Magento packages to vendor and gives us complete source files of our project
• Next, we compile static assets using well known Magento CLI commands
• Save generated files as artefacts for the deploy step.

2. The deployment

Now as for deploy tasks, those are which will be run exclusively upon push to either staging or master branches after the build steps succeed:

• push the “artefacts” over to respective website’s .shadow subdirectory (which is .gitignore-d but ensured of its presence via .gitkeep file
• our target deployment server location, e.g. /srv/www/example.com/.shadow/ will receive all our compiled files using rsync
• launch a deploy.sh script as on the server in order to apply the generated/uploaded assets

The deploy.sh script will:

• put live to maintenance: php bin/magento maintenance:enable
• git pull
• overwrite/rsync pub/static, generated and vendor dirs with the one from the .shadow subdirectory;
• bin/magento setup:upgrade --keep-generated
• flush caches;
• maintenance off.

deploy.sh

#!/bin/bash
# called by CircleCI or other CI after generating compiled stuff and putting those into .shadow subdirectory

cd "$(dirname "$0")"

php bin/magento maintenance:enable
git pull
#overwrite pub/static, generated and vendor dirs with the one from the build;
rsync -avz --delete .shadow/pub/static/ ./pub/static/
rsync -avz --delete .shadow/vendor/ ./vendor/
rsync -avz --delete .shadow/generated/ ./generated/
rsync -avz --delete .shadow/var/view_preprocessed/ ./var/view_preprocessed/
php bin/magento setup:upgrade --keep-generated
cachetool opcache:reset
n98-magerun2 cache:flush
cachetool opcache:reset
php bin/magento maintenance:disable

Naturally, we have to ensure SSH keys in CircleCI allow connecting to our staging/live server and run the necessary commands.

Benefits:

• Faster updates – the “deploy” step will take less than a minute. This is the time your M2 is essentially off during deployment.
• The CPU-heavy assets compilation doesn’t even happen in our servers – it happens in CircleCI, using their power and not affecting live server load at all.
• You know that the assets compile without errors before pushing new plugins on the live server. No more failed compilations (and complications) at live.