Site icon GetPageSpeed

Magento 2 deployment with CircleCI

Danila Vershinin
This post is well within its draft state. So pardon for any typos.

A recommended read before implementing deployment with CircleCI is Magento 2.2 Deployment Improvements.

Need for pipeline deployment

Magento 2 builds on the two main modes: production and developer. The production website has to run with live settings and with “compiled” assets.
The assets compilation is a very slow step in bringing Magento 2 to production state.
When you install a new plugin, you have to run compilation of static assets. This brings your store into maintenance mode for a few minutes, at best, and the site is essentially down. What can we do about this?

Answer: pipeline deployment.

Let’s review best practices approach for Magento 2 development and deployment.

;tldr

Store Magento 2 in git

Code should be stored in BitBucket or another centralised git repository (paid GitHub for example). There are going to be 2 branches:

We assume that the dev. instance is not needed, since developers have a development environment of their own.

Development workflow

Deployment to Live

Once the website owner is happy with the state of things at staging, they (or developer) would create a pull request in BitBucket: from staging to master branch. Once the pull request is merged, we’ve essentially pushed to the master branch.

Alternatively, developers would be working on their feature specific branch, and merge it to both staging (for QA) and master (for making the feature live).

Either way, git push (merge) to master branch should result only in one thing – putting those changes live. CircleCI is something we’re going to use for exactly that – deployment.

CircleCI is an interesting tool that allows you to run arbitrary tasks involving your git codebase. At the same time those tasks will use CircleCi’s servers infrastructure to be run. So your web server does not even incur any performance penalty for running those tasks. Moreover it’s free for private BitBucket repositories. Sounds exciting?

Sure it does. Because by leveraging CircleCI, we are not only able to deploy something, we can actually build that something to reach deployable state.

1. Build Magento static assets using CircleCI

Once you conncect CircleCI to your Github / Bitbucket repository, it really acts as a sort of hook/trigger program for your repository. What we want it to do, on a high level:

We’re going to build a CircleCI workflow which includes 2 main tasks: the “build” and the “deploy”. We can have multiple copies of “deploy” steps in order to deploy things over to different places: dev, staging, live, etc.

The build steps will:

Note on compiling static assets

This step is ridiculously slow at all times, especially if you start with empty pub/static.

The setup:static-content:deploy, when run without parameters, is “flawed” in a way that it always compiles assets for all languages. Internally deploy:mode:set production uses setup:static-content:deploy (see here and compiles for only the necessary languages.

But we cannot use deploy:mode:set production since it also runs a few steps which require Magento 2 to be setup, whereas we want to compile static assets using nothing but files from git.

So we have to explicitly specify the website locales we want to compile assets for, in order to reduce time spent for the task. We can also specify themes we want assets to be compiled for as such:

php bin/magento setup:static-content:deploy --area=frontend --theme=Gw/frontend en_US

A quick way to check for active themes in a multi-store Magento 2 is the following query:

SELECT code FROM core_config_data AS ccd LEFT JOIN theme AS t ON t.theme_id = ccd.value WHERE path = 'design/theme/theme_id';

Adding --jobs=X will allow to use multiple CPU cores, e.g.:

php bin/magento setup:static-content:deploy --area=frontend --theme=Gw/frontend --jobs=32 en_US

.circleci/config.yml

version: 2
jobs: # a collection of steps
  build:
    docker: # run the steps with Docker
      - image: getpagespeed/m2builder:latest
    working_directory: /sources # directory where steps will run
    steps: # a set of executable commands
      - checkout
      - run:
          name: "Install Mage Composer keys"
          command: composer global config http-basic.repo.magento.com $MAGENTO_KEY_PUBLIC $MAGENTO_KEY_PRIVATE
      - run:
          name: "Install Plumrocket composer keys"
          command: composer global config http-basic.store.plumrocket.com $PLUMROCKET_KEY_PUBLIC $PLUMROCKET_KEY_PRIVATE
      - run:
          name: "Install composer cache"
          command: ./extract-composer-cache.sh
      - run:
          name: "Install Magento plugin dependencies (vendor)"
          command: composer install
      - run:
          name: "Compile PHP stuff (generated)"
          command: php -d memory_limit=-1 bin/magento setup:di:compile
      - run:
          name: "Compile static (pub/static)"
          command: php -d memory_limit=-1 bin/magento setup:static-content:deploy -f en_US
      - persist_to_workspace:
          root: /sources
          paths:
            - "vendor" 
            - "generated"
            - "pub/static"
            - "var/view_preprocessed"
  deploy:
    docker: # run the steps with Docker
      - image: getpagespeed/m2builder:latest
    working_directory: /sources
    steps:
      - attach_workspace:
          at: /sources
      - add_ssh_keys:
          fingerprints:
            - "e2:65:63:c0:04:b6:82:f0:23:f3:d3:8e:9d:06:3f:fd"
      - run:
          name: "Copy generated transient files to shadow directory"
          command: rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ live-user@m2.example.com:/srv/www/example.com/.shadow/
    - run:
        name: "Deploy run: maintenance mode + git pull + overwrite compiled files + upgrade data + cache clear + maintenance off"
        command: ssh -v -o StrictHostKeyChecking=no staging@m2.example.com "/srv/www/example.com/deploy.sh"
  deploy-staging:
    docker: # run the steps with Docker
    - image: getpagespeed/m2builder:latest
    working_directory: /sources
    steps:
    - attach_workspace:
        at: /sources
    - add_ssh_keys:
        fingerprints:
        - "7b:72:9c:aa:85:52:3f:77:55:24:5b:db:c9:98:50:df"
    - run:
        name: "Copy generated transient files to shadow sub directory"
        command: rsync -e "ssh -p 22 -o StrictHostKeyChecking=no" -avz --delete --exclude=/.gitkeep ./ staging@m2.example.com:/srv/www/staging.example.com/.shadow/
    - run:
        name: "Deploy run: maintenance mode + git pull + overwrite compiled files + upgrade data + cache clear + maintenance off"
        command: ssh -v -o StrictHostKeyChecking=no staging@m2.example.com "/srv/www/staging.example.com/deploy.sh"
workflows:
  version: 2
  build-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:   
            branches:
              only: master
      - deploy-staging:
          requires:
          - build
          filters:
            branches:
              only: staging

The “Install composer cache” is optional, but it’s meant to speed up subsequent composer install. This extract-composer-cache.sh script will download composer cache zip and might look like the following:

#!/bin/bash

# yum -y install unzip
# Hosting with admin_ prefix ensures that Varnish will deliver the file even if Magento is "down"
curl https://www.example.com/admin_composer_cache.zip --output /tmp/composer-cache.zip
mkdir -p ~/.composer/cache
rm -rf ~/.composer/cache/*
unzip /tmp/composer-cache.zip -d ~/.composer/cache

To generate composer cache, simply zip your ~/.composer/cache on the live system and put it so it’s web accessible at `https://www.example.com/admin_composer_cache.zip`.

Reiterating again on the build steps of our CircleCI program, what they do is:

2. The deployment

Now as for deploy tasks, those are which will be run exclusively upon push to either staging or master branches after the build steps succeed:

The deploy.sh script will:

deploy.sh

#!/bin/bash
# called by CircleCI or other CI after generating compiled stuff and putting those into .shadow subdirectory

cd "$(dirname "$0")"

php bin/magento maintenance:enable
git pull
#overwrite pub/static, generated and vendor dirs with the one from the build;
rsync -avz --delete .shadow/pub/static/ ./pub/static/
rsync -avz --delete .shadow/vendor/ ./vendor/
rsync -avz --delete .shadow/generated/ ./generated/
rsync -avz --delete .shadow/var/view_preprocessed/ ./var/view_preprocessed/
php bin/magento setup:upgrade --keep-generated
cachetool opcache:reset
n98-magerun2 cache:flush
cachetool opcache:reset
php bin/magento maintenance:disable

Naturally, we have to ensure SSH keys in CircleCI allow connecting to our staging/live server and run the necessary commands.

Benefits:

Exit mobile version