Migrate Magento 1.x to Magento 2.x

Let’s go over the steps to migrate your existing Magento 1.x store to new and shiny Magento 2.x.

First, setup new server for Magento 2.x. The stack recommended to run Magento 2.x is quite different. The essentials are:

• PHP 7.x
• Varnish
• Percona MySQL 5.6

Install new “empty” Magento 2.x

• No matter if you use new server or not, choose a different database name (and username) for Magento 2.x database.
• Install n98-magerun2 utility script

For the rest of the tutorial, we assume that:

• You have configured your Magento 2.x server properly: it runs under a separate Linux user (e.g. username, not www-data, not nginx, and surely enough, not root 🙂 )
• Your original Magento version is 1.9.2.2 (can be virtually any, simply adjust it in the commands below)

Import Magento 1.x database copy

In this step we import a copy of your Magento 1.x database onto your Magento 2.x server. Let’s say it will go by the name of livecopy. Create it and its db user:

mysqladmin create livecopy
mysql -e "CREATE USER 'livecopy'@'localhost' IDENTIFIED BY 'livecopy';"
mysql -e "GRANT ALL PRIVILEGES ON livecopy. * TO 'livecopy'@'localhost';"
mysql -e "FLUSH PRIVILEGES;"  

Note: it must be a copy of the live database that you work with.

We will create a small import.sh script to fetch copy of live data and files.
So mkdir /srv/www/livecopy. And our import.sh would look like this:

#!/bin/bash

copy-magento.sh \
  -rh m1.example.com -ru username \
  -rp /srv/www/example.com/httpdocs \
  -lp /srv/www/livecopy \
  --local-dbname livecopy --local-dbuser livecopy --local-dbpass livecopy

In the next step we will configure your migration to use livecopy database as your migration source.

Install Migration toolkit

Make sure to also install Migration toolkit matching your Magento 2 version.
Get note of your current Magento version like this:

php bin/magento --version --help
Magento CLI version 2.2.1

If you require some automation in provisioning Magento 2.x servers which require migration, use:

n98-magerun2 sys:info --format=json

The installed M2 version is ["1"]['value'] in output.

Either way, say you have Magento 2.2.1, so you need Migration Toolkit with same version: 2.2.1.
Let’s install it:

composer config repositories.magento composer https://repo.magento.com 
composer require magento/data-migration-tool:2.2.1

Configure Migration

cd vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.2.2
cp config.xml.dist config.xml
cp map.xml.dist map.xml

make changes to the config.xml according to M2 docs.

Replace:

    <source>
    <database host="localhost" name="magento1" user="root"/>
    </source>
    <destination>
        <database host="localhost" name="magento2" user="root"/>
    </destination>

with:

    <source>
    <database host="localhost" name="livecopy" user="livecopy" password="livecopy"/>
    </source>
    <destination>
        <database host="localhost" name="m2db" user="m2db" password="your-secret"/>
    </destination>

Also replace <crypt_key /> with <crypt_key>secret</crypt_key>.

The <crypt_key> tag is mandatory to fill. It can be found in local.xml file which is located in the directory of Magento 1 instance at app/etc/local.xml in <key> tag.

Install Magento 2.x compatible theme and plugins

Now install all the plugins and the theme you know for sure will be used in M2.

Deploy static assets and upgrade database (if new plugins installed)

php bin/magento setup:upgrade
php bin/magento setup:di:compile
php bin/magento setup:static-content:deploy

Backup M2 database

Let’s say that you have installed Magento 2.x to use m2db database name.
After your theme and plugins were installed, dump the M2’s database. It will come in handy later:

mysqldump m2db > m2fresh.sql

Attempt migration of settings and data

Assuming your original Magento is 1.9.2.2, you’d run:

php bin/magento migrate:settings --reset vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.2.2/config.xml
php bin/magento migrate:data --reset vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.2.2/config.xml

If you ended up with “corrupted” Magento 2.x database anytime, restore it:

mysql m2db < m2fresh.sql

You’ll see quite a lot of errors, so let’s go on with the next step.

Develop “fixer” SQL script

Surely you could go ahead and adjust migration toolkit’s configuration files to: ignore database fields which are not relevant for Magento 2.x and such.
But I find it much more straightforward to make your Magento 1.x’s data is compatible to 100% with Magento 2.x.

Create prepare-m1.sql file with all the queries you need for the migration not to fail. Those queries are to be run against livecopy database.
Migration toolkit wants you to map fields from Magento 1.x to Magento 2.x. For the errors you get, you mostly don’t care for the failing data anyway.

Following are possible errors and which queries you’d add to the fixer SQL script in each case.

Orphan record IDs

“Foreign key (FK_CATALOG_EAV_ATTRIBUTE_ATTRIBUTE_ID_EAV_ATTRIBUTE_ATTRIBUTE_ID) constraint fails. Orphan records id” ..

Solution is to delete orphan records:

DELETE FROM catalog_eav_attribute WHERE attribute_id = <id>;

e.g.:

DELETE FROM catalog_eav_attribute WHERE attribute_id = 204;

Source fields are not mapped

[ERROR]: Source fields are not mapped. Document: catalog_eav_attribute. Fields: enable_layered_swatch,layered_filter_option,enable_product_swatch

Solution is to drop the fields from a given table, in our example catalog_eav_attribute:

ALTER TABLE `catalog_eav_attribute` DROP `enable_layered_swatch`;
ALTER TABLE `catalog_eav_attribute` DROP `layered_filter_option`;
ALTER TABLE `catalog_eav_attribute` DROP `enable_product_swatch`;

Source documents are not mapped

[2017-11-18 08:24:43][ERROR]: Source documents are not mapped: am_animation,am_css,am_revslide,am_revslider,am_seo_sitemap,amazon_login,… long list of tables….

This just complaints that some of Magento 1.x tables have no mapped equivalent in Magento 2.x database. Again, in most cases, you won’t care for that unmapped data. So the solution is to drop those tables from Magento 1.x database copy:

SET foreign_key_checks = 0;
DROP TABLE IF EXISTS am_animation,am_css,am_revslide,am_revslider,am_seo_sitemap,amazon_login,... long list of tables.... ;
SET foreign_key_checks = 1;

Class does not exist but mentioned

[2017-11-18 08:43:19][ERROR]: Class weltpixel_megamenu/category_attribute_source_display does not exist but mentioned in: eav_attribute.source_model for attribute_id=142

Solution:

DELETE FROM eav_attribute WHERE attribute_id IN (142, 147, 181, 182, 188, 189, 203, 207, 208, 209);

Example contents for the file:

DELETE FROM catalog_eav_attribute WHERE attribute_id = 204;
ALTER TABLE `catalog_eav_attribute` DROP `enable_layered_swatch`;
ALTER TABLE sales_flat_order_item DROP COLUMN reverb_item_link, DROP COLUMN reminder;
ALTER TABLE sales_flat_quote DROP COLUMN facebook_order;
ALTER TABLE  sales_flat_quote_item DROP COLUMN  reverb_item_link;
DELETE FROM eav_attribute WHERE attribute_id IN (142, 147, 181, 182, 188, 189, 203, 207, 208, 209);

SET foreign_key_checks = 0;
DROP TABLE IF EXISTS am_animation,am_css,am_revslide,am_revslider,am_seo_sitemap,amazon_login,amazon_payments_token;
SET foreign_key_checks = 1;

Using MySQL GUI editor to drop offending columns and tables, I was able to quickly copy the queries I needed for building up the fixer SQL script. In the end, this is much faster then editing migration .xml files and creating them, etc. etc.

You can also drop multiple columns:

ALTER TABLE review_detail DROP COLUMN good_detail, DROP no_good_detail, DROP response, DROP image, DROP video, DROP sizing, DROP body_type, DROP location, DROP age, DROP height,DROP remote_addr,DROP pros,DROP cons,DROP recommend_to, DROP customer_email;

Other issues

After migration you might find error :

Unable to load theme by specified key: ‘themename’

This means that Magento 1 theme is not installed/present in Magento 2. You are likely using a new theme, so remove reference to the old with:

UPDATE core_config_data SET `value` = NULL WHERE path = 'design/theme/theme_id';
TRUNCATE TABLE design_change;

MySQL configuration

If you happen to receive this error during migration:

[PDOException] SQLSTATE[08S01]: Communication link failure: 1153 Got a packet bigger than ‘max_allowed_packet’ bytes

You need to adjust your MySQL configuration with:

[mysqld]
max_allowed_packet=16M

And then restart the MySQL server.

Reindex!

Problem now after running data migration, you might see:

We can’t find products matching the selection.

Fix with:

php bin/magento indexer:reindex
n98-magerun2 cache:flush

Fix cron jobs

DELETE FROM core_config_data WHERE path LIKE 'crontab/jobs%';

Develop your migration script

The best approach to any migration would be building a bash script that will do all the above steps for you. This will also allow you to easily re-test your migration and improve it until it’s perfect to go live.

We are going to create migrate.sh which we will restore Magento 2 database to its initial “fresh” state (when all Magento 2 plugins and the theme were just installed and configured), then run migration using “fixed” (M2 compatible) live database copy.

Migration script migrate.sh goes down to all the steps we made earlier, with few additions:

• Import latest livecopy database from live Magento 1.x server
• Import (sync files) from live Magento 1.x server

First, you’ll need:

• install n98-magerun2 on the Magento 2.x server
• write a import.sh script to import latest livecopy database

Now the migration script migrate.sh will look like this:

echo "Restoring good db for M2."  
mysql m2db < m2fresh.sql
echo "Done restoring good M2 db"

echo "Start importing M1 data to its own db"
# Run import of latest M1 data into "livecopy" database (this goes in another custom script)
./import.sh
# On Citrus stack servers, you'd use the following instead of custom import.sh:
# copy-magento.sh 1.2.3.4 root 22 /path/to/m1/on/origin/server /srv/www/livecopy
echo "Done importing M1 data to its own db"

echo "import data integrity fixes to M1 copy:"
mysql livecopy < prepare-m1.sql
echo "done import integrity fixes"

cd /var/www/html # or wherever your Magento 2.x is installed

echo "Running settings migration"
sudo -u username php bin/magento migrate:settings --reset vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.2.2/config.xml

# sync media
rsync -avvz --delete --exclude=.htaccess /ssh/or/local/path/to/m1/media/folder/ /var/www/html/pub/media/
# On Citrus stack servers, you'd use the following instead (since the copy-magento.sh has already synced the media as well):
# rsync -avvz --delete --exclude=.htaccess /srv/www/livecopy/media/ /srv/www/example.com/pub/media/

chown -R username:username /var/www/html/pub/media

echo "Runnign data migration"
sudo -u username php bin/magento migrate:data --reset vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.2.2/config.xml
sudo -u username php bin/magento indexer:reindex
sudo -u username /usr/local/bin/n98-magerun2 cache:flush

echo "Migration complete"

Run the script with ./migrate.sh and see whether anything fails. Rinse, adjust the script, repeat.

Depending on your requirements, the migration can be a good time for documenting your Magento 2.x setup. So you might use a different approach than the outlined above. Write script to run migration in a different order:

• import “virgin” Magento 2 MySQL database (you had to do run MySQLdump right after M2 installation in this case).
• install all the plugins and theme (scripted)
• fetch latest livecopy database and fix it with prepare-m1.sql script
• run data migration.

It takes more time, since you have to write all the commands to install plugins and theme, but allows for a more documented setup from bare Magento installation to a solid live website.