Running Magento 2 Tests Via Bitbucket Pipelines

We continuously try to improve the overall quality of our Magento projects. When we started with Magento 2, we decided that it is finally time to set up automatic test execution. Since we currently use Bitbucket for our private projects, we implemented that via Bitbucket Pipelines. I think the process for GitHub Actions, GitLab CI, you name it, is quite similar.

I will not go into the details of the Bitbucket Pipelines syntax. Check out the official documentation if you want to know more about the basics. I will also not go into the details of everything, just a few interesting notes and recommendations:

• We run three steps:
  1. Linting & Static Checks & Unit Tests: Checks the syntax of all PHP files, checks the Magento coding standard, executes a PHPStan analysis and runs all (client-specific!) unit tests.
  2. Integration Tests: Runs all (client-specific!) integration tests.
  3. MFTF Tests: Runs all (client-specific!) MFTF tests.
• Run multiple steps in parallel. This will not lower the costs / number of build minutes you spend, but it will speed up the execution.
• Make sure to activate the cache for Composer to speed up the setup and to save money / build minutes.
• When you use docker-php-ext-install, make sure to pass -j$(nproc) as a parameter. This will parallelise the installation, which again saves money / build minutes.
• We started with php:7.2-apache as a base Docker image, which does not come with too many unnecessary dependencies. However, this also means that we need to install many required dependencies ourselves.
• Bitbucket Pipelines currently does not nicely support multiple databases by default. The workaround is to define a root SQL password and then create the necessary databases yourself via CLI.
• For the MFTF tests, we need a (stripped) client database. We currently generate that stripped database dump via a cronjob each night and then get that database dump from a server via scp.
• MFTF currently does not have an option to run all tests from a specific directory. Our workaround is to use a specific group for all client-specific tests. I created a GitHub issue for that.

Here is the complete Bitbucket Pipelines configuration file, which you can probably use with a few customisations in your projects:

Bitbucket Pipelines is nice, but a hell to debug. It took me quite a few evenings to get that to work, so I hope that I can save your time with this post 🙂 If you have any questions or want to say thank you, feel free to leave a comment or contact me!

When I come across interesting ideas how to make debugging easier, I will add them here:

• I stumbled upon an approach to run Bitbucket Pipelines locally. It does not support every feature and I did not try it yet, but the approach is awesome 🙂
• MFTF saves the results to dev/tests/acceptance/tests/_output/allure-results by default. I did not find a way to change this. However, you can add a after-script section to your configuration, which is executed no matter if the pipeline fails or succeeds. You can then copy the directory to e.g. test-results, where Bitbucket will pick up any XML file with test results. Unfortunately, Bitbucket currently cannot correctly parse the Allure output, so this currently does not help. I hope this will change in the future. I created an issue for that. Please vote for it! I also created an MFTF issue to support xUnit-compatible output.
• It is often quite handy to have a look at the HTML output of failing MFTF tests. You can have a look at it by adding cat dev/tests/acceptance/tests/_output/allure-results/*.html to the after-script section of your pipeline.