Automation with Cypress

With the launch of new vertical brands by Dotdash came a new design for Atlas, our in-house content management system (CMS), tailored to each brand. This also required architectural changes to allow each brand to customize Atlas to suite their needs, which also meant a rethinking of our automation framework.

Having dealt with inconsistencies in test results while using Protractor, we explored some alternatives that could address the inherent flakiness of Protractor tests. We landed on Cypress, which had just recently launched to the public, and promptly proceed to set up a proof of concept (POC) . After some consideration of the framework and the results we saw in terms of a significant reduction in flakiness, we decided to write our new automation tests with Cypress. This is a write up on our journey in setting up Cypress framework for our Atlas web app.

Background :

The testing team for our CMS was set up around 3 years back, and back then we had no testing and no automation of any kind in place. We were excited to set up automation for testing Atlas using Protractor. After set up of entire suite we constantly ran into the flakiness in Protractor and over a period of time,the results were unreliable due to the inherent flakiness. It eventually led us to reconsider the framework that we wanted to continue using for CMS AFT’s.

Why we chose Cypress

• Cypress is written completely in JavaScript with React on the front end and Node.js on the backend. The desktop app runs on Electron.
• Runs in the browser, but also supports Chromium and Electron. It shows a very clear description of which test is running alongside the app it’s running on making the debugging of the test steps much easier.

• Does not require the user to specify the element selector with id or class. It’s generic element getter accepts any of the above and more. Hence it is easier to find the elements to write tests.
• Supports a wide range of frameworks and libraries like Mocha, Chai, Chai-jQuery, Sinon, Sinon-Chai, and Sinon-as-Promised.
• The Cypress team is currently working on a feature to allow running tests headlessly without spawning a browser.
• Supports a wide range of reporters, including mochasome, which gives a clear picture of number of tests run, pass/fail, etc. It also provides a snapshot of the code that caused failures.

• Supports asynchronous calls with a feature called aliasing, which is a DSL that solves referencing work done in previous commands.
• Supports storing a fixed set of data in a file (.json, .gif , etc.) that can be used as a baseline for running tests.
• Provides user direct access to the XHR objects, enabling user to make assertions about its properties. Additionally users can even stub and mock a request’s response.

In addition to these basic features, there are additional useful add-ons that make automation easier:

• Automatic re-run on the desktop application triggered by code changes.
• Record feature allows to record the entire execution of the tests.
• A dashboard that provides user insight into what happened during the test run by giving access to tests that were recorded while running in headless mode.

• A screenshot of each failure is captured.

• A video for each run is created.

What drawbacks did we find?

Since Cypress is still evolving, there are some key features yet to be supported.

The features that we needed but were not supported by Cypress are:

• Hover functionality;
• Running specific set of tests using a glob; and
• The ability to check if elements exists.

What challenges did we face?

The main challenge we faced was using Cypress through our CD framework, Jenkins. Since it is not open source and runs on more recent versions of Node.js, it could not yet be run via our Jenkins instance.

Structure of Automation

We thought very carefully about making our automation suites more reusable and extensible to support custom tests for our various brands. We also decided to write them in ES6, which made it easier to write reusable code. We landed on a setup that’s highly configurable, allowing the use of any user to run automation tests against any of our brands on any of test environment.

The Verdict

We are convinced that choosing Cypress as our automation framework future proofs us for the next few years and allows us to focus on improving test coverage than focusing on on fixing the test execution issues with Protractor . Since then automated tests have become more stable and reliable, and they’re a lot of fun to write. We plan on using Cypress to automate testing for additional internal tools in the near future.