Automated Visual Testing using aShot and Selenium Web driver

The objective of this post is to explain how to add Automation visual testing for existing selenium, Java based automation test framework. Let’s talk about the experience, challenges we faced while setting up automated visual testing at dotDash(https://www.dotdash.com/)

Software engineering teams who are used to frequent deployments of their features tend to build tools that help them mitigate the test efforts for such a high turn around in deployments.At Dotdash we use a combination of functional tests,HTML scraping and Visual regression tests to reduce the amount of time spent on regression testing.Functional tests focus on the features that are being developed while the HTML scraping focus on configuration settings etc and Visual tests help us detect design issues before we deploy code changes to production.

BENEFITS OF VISUAL TESTS?

Image Courtesy :

We shipped lot of design bugs to production which prompted us to search for a solution that will help us detect them in our QA pipeline.These are examples of visual design changes that we tend to leak to production prior to the addition of visual tests to our tool set :

• Minute pixel changes to logos, font size,SVG and design elements which are hard to find during manual testing phase.
• It is manually intensive to cover all the test matrix regularly for frequent releases.
• Testing on various web browsers, Devices, Operating systems , Screen resolutions and Responsive design.

Here is the list of tools/technologies we used for visual testing(it forms the foundation for functional testing at dotdash as well):Selenium WebDriver, Selenium Grid,Java 8, Junit 4, Jenkins, aShot.

Venus(our in house web application test framework) is designed and developed using page object model. All our functional tests are executed on remote grid nodes. We trigger different categories of functional tests(smoke , regression, visual tests) by leveraging parametrized Jenkins jobs.

CHALLENGES

• Handling multiple screenshots : We need to scroll till end of the page and take screenshots to make sure all of the page elements are captured.Not all of the browsers supported full page automated screenshots.
• Data changes and ad sizes: We started running visual tests in QA environment by baselining against Prod environment.Soon we found out the many of our data driven components were producing false positives, similarly Ads rendered on QA environments were different than Prod as well. So we end up with more tests failures.
• Scrollbar and Mouse : While taking screen shots of the web pages we should hide Scrollbar and Mouse to make sure they don’t interfere with automated comparisons.

How It Works?

About aShot (https://github.com/pazone/ashot)

WebDriver Screenshot utility

• Takes a screenshot of a WebElement on different platforms (i.e. desktop browsers, iOS Simulator Mobile Safari, Android Emulator Browser)
• Decorate screenshots
• Provides flexible screenshot comparison

Basic Framework code

We need to have the below setup before running tests to import the ashot relevant classes to your existing framework.

Test code

The actual tests looks like below, we support multiple devices(PC, mobile_chromeEmulator, iPad) and we run tests on Chrome browser.We can hide or delete some web elements/components(css selectors) from the page.

Test Methods

Note: All these methods are part of Runner class in the Interface commonTestMethods.

REPORT DIFFERENCES

Here are some examples of screen shot differences after running tests.

Notes: we run tests on selenium Grid and we trigger jobs through jenkins and upload Junit tests reports/screen shots to AWS s3 bucket, so that we can see test reports anytime.

Feature Enhancements

• Run on multiple qa environments
• Support multiple browsers.
• Add element level validation.

About Me:

This is Mahesh Oruganti, working as Automation QA Engineer at Dotdash. Happy Learning. Thanks.