Automated test with Puppeteer

Software testing is a serious and required task to reach a certain quality. This blog post focuses on how it can be automated with Puppeteer. Before going into the details of Puppeteer, we should go over software testing and clarify the concepts related to it.

Software Testing

"Production-ready software requires testing before it goes into production" [1]. Here, one approach is to follow manual testing and the other one is to automate the whole testing process. "It's obvious that testing all changes manually is time-consuming, repetitive [, not scalable] and tedious. Repetitive is boring, boring leads to mistakes" [1] that we don't want. Therefore, automation is a great alternative for those repetitive tasks. Further, automation of tests can deliver requested development pace and reliability of the software product, especially on a massive scale. In addition to these, ensuring the reliability and stability of the software is also very important for the developers in order not to lose time on bugs created by changing code blocks. Specifically, it becomes very curial when team size increases.

Test pyramid [2] is a key concept to follow when you want to write automated tests in your software. This concept defines how much tests you should add to your software for each level. There are three layers of this pyramid:

  • Unit Tests
    • Tests that cover isolated pieces of code, e.g. functions, etc.
  • Service Tests
    • Cohn's [2] naming convention about the second level is not self-explanatory. This level generally is accepted as Integration Tests by the community. Integration Tests cover connected pieces of the application, e.g. database, filesystem, etc.
  • User Interface Tests
    • With the modern front-end frameworks such as React.js, Vue.js, Angular.js user interface tests can be accomplished in the level of unit tests. Therefore, it becomes harder to set the right level for User Interface Testing, it actually spreads to different levels. However, we can put End-to-End Tests to a higher level. It is different than to User Interface Testing since it covers the whole journey of functionality from user interaction to services, e.g. login flow, purchase flow, etc.
Test Piramide

The key points of this pyramid:

  • "Write tests with different granularity" [1]
  • "The more high-level you get the fewer tests you should have" [1]

However, this approach may not suit for each case. In your software, for instance, you may have less business logic and more integration. In this case, you may want to add more integration tests to your software. Therefore, it is best to think of how many tests you should write for each granularity. (You can check this post for more discussion: https://kentcdodds.com/blog/write-tests)

twitter post

As this blog post is related to automated tests with Puppeteer, it can be used for both end-to-end and user interface tests. Even if end-to-end tests are triggered by the user interface, end-to-end tests cover whole flow and it doesn't have to check all possible cases on the UI.

When do you really need end-to-end testing?

End-to-end tests are useful to identify problems in user journeys. User journeys are the flows that a user can follow in your application. Therefore, the idea of end-to-end tests is to imitate a user's behaviour in certain flows from start to end to ensure that everything works as expected. Running end-to-end tests are slower than to unit tests since they can touch many components of your application or third party services. As a result, we can think of them as expensive in terms of resources that are reserved for them. For instance, assume that you want to run end-to-end tests of a web application, to accomplish this you need to run a browser and this browser consumes a significant amount of memory. Furthermore, the initialization of the browser and the other components takes a lot of time. Hence, only relying on end-to-end tests is not preferable in terms of development efficiency. Therefore, you may want to write end-to-end tests for high-value interactions of your application such as checkout, login, etc.

What is Puppeteer?

Puppeteer provides a high-level API to control Chrome or Chromium programmatically. It is an open-source Node.js library. Puppeteer runs headless (i.e. a browser that doesn't have a user interface) by default, but it can be configured to run fully (non-headless) Chrome or Chromium.

With this tool, you can run your web application on a browser and imitate user actions programmatically.

Use cases:

  • Generate screenshots and PDFs of pages.
  • Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)).
  • Automate form submission, UI testing, keyboard input, etc.
  • Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features. Run regression and end-to-end tests.
  • Capture a timeline trace of your site to help diagnose performance issues.
  • Test Chrome Extensions.
  • Check for console logs and exceptions.
  • Replicate user activity.

Jest and Puppeteer

Puppeteer API is not designed for testing and it doesn't provide you with the whole functionality of a testing framework. Therefore, it can be used with Jest JavaScript testing framework. The samples on this blog post use jest-puppeteer Nodejs library. It provides all required configuration for writing integration tests using Puppeteer with Jest.

Samples

This section provides a couple of examples to give you better insights into Puppeteer's usage. They don't cover all the features of Puppeteer and this blog post doesn't aim to give you the detailed information of the Puppeteer API. You can build upon the given examples and explained concepts.

The shown examples are also available at this GitHub repository.

Prerequisites and Installation

You need:

  • The recent stable version of node.js
  • The recent stable version of yarn or npm

Examples have these dependencies:

"devDependencies": {   
 "jest": "^25.1.0",   
 "jest-puppeteer": "^4.4.0",
 "pixelmatch": "^5.1.0",
 "puppeteer": "^2.1.1" 
}

Puppeteer library downloads lastest Chrome executable. You can use puppeteer-core instead of puppeteer if you want to use existing Chrome executable in your system and pass the path of executable as a configuration option.

You can run the tests with yarn test command.

Taking Screenshot

You can take a screenshot of your website with different options such as setting viewport or emulated device.

const puppeteer = require('puppeteer');

(async () => {   
  const browser = await puppeteer.launch();   
  const page = await browser.newPage();   
  await page.goto('https://designisdead.com/');   
  await page.screenshot({path: 'homepage.png'});    

  await browser.close(); 
})();

Comparing Screenshots

You can detect changes visually by taking screenshots of different versions of your application. You can take screenshots with Puppeteer but you need another tool to compare them. This sample uses pixelmatch library.

const fs = require('fs'); 
const PNG = require('pngjs').PNG;
const pixelmatch = require('pixelmatch');  

const img1 = PNG.sync.read(fs.readFileSync('version1.png')); 
const img2 = PNG.sync.read(fs.readFileSync('version2.png')); 
const {width, height} = img1; 
const diff = new PNG({width, height});  

pixelmatch(img1.data, img2.data, diff.data, width, height, {threshold: 0.9});  

fs.writeFileSync('diff.png', PNG.sync.write(diff));
Image 2 blog Mustafa

Be aware that some image comparison tools find differences by checking the pixel difference, therefore, if the text is changed, they will show it as a change. In other words, if you change your content, you will see it as a difference.

Checking integration of third party applications

You may use third party services, scripts, etc. in your application. Therefore, it will be a good idea to check that their integration with your application works as expected.

describe('Analytics', () => {
 beforeAll(async () => {     
   await page.goto('https://designisdead.com/')   
 })    

 it('should return google tag manager', async () => {     
   const tagManager = await page.evaluate(() => google_tag_manager)     
   expect(tagManager).toBeDefined()   
 }) 
})

Mobile and Desktop Layout

Since there is a significant variance in screen sizes, there are a lot of cases that need to be tested here.

const devices = require('puppeteer/DeviceDescriptors'); 
const iPhonex = devices['iPhone X'];  

describe('Mobile', () => {
  beforeAll(async () => {     
    await page.emulate(iPhonex)
    await page.goto('https://designisdead.com/')   
  })    

  it('should render hamburger menu', async () => {     
    await page.waitForSelector('.Page-hamburger', {       
      visible: true     
    })   
  }) 
})
describe('Desktop', () => {
  beforeAll(async () => {    
    await page.setViewport({ width: 1280, height: 768 })   
    await page.goto('https://designisdead.com/')   
  })    


  it('should not render hamburger menu', async () => {     
    await page.waitForSelector('.Page-hamburger', {       
      hidden: true     
    })   
  }) 
})

SEO checks

Since search engines crawl your production website, it may be a good idea to check your pages' SEO performance. However, even if the below example handles this issue on test cases, you may want to generate a score and corresponding report.

describe('SEO', () => {
 beforeAll(async () => {     
   await page.goto('https://designisdead.com/')   
 })    

 it('should display "Design is Dead" text on title', async () => { 
   await expect(page.title()).resolves.toMatch('Design is Dead')   
 })    


 it('should have description meta-tag', async () => {     
   const descriptionContent = await page.$eval("head > meta[name='description']", element => element.content);

   expect(descriptionContent).toBeDefined();   
 })    

 it('should have a headline', async () => {     
   const headlines = await page.$$('h1')      

   expect(headlines.length).toBe(1)   
 }) 
})

Login

One of the most important features of web applications is logging in to the user's account. It can be count as an example of End-to-End tests since back-end services involved in the process.

// put GITHUB_USER and GITHUB_PWD values to .env file require('dotenv').config()  

describe('Github - Login', () => {
  beforeAll(async () => {     
    await page.goto('https://github.com/login')   
  })    

  it('should log in and redirect', async () => {     
    await page.type('#login_field', process.env.GITHUB_USER)     
    await page.type('#password', process.env.GITHUB_PWD)     
    await page.click('[name="commit"]', {waitUntil: 'domcontentloaded'})
    const uname = await page.$eval('#account-switcher-left > summary > span.css-truncate.css-truncate-target.ml-1', e => e.innerText)          

    await expect(uname).toMatch(process.env.GITHUB_USER)   
  }) 
})

Selenium and other tools to automate browsers

Puppeteer is not the only tool that provides high-level API to manage and automate browsers. Playwright is an alternative Node.js library that supports Chromium, Firefox and WebKit. It is developed by the same team built Puppeteer and its API is very similar to Puppeteer. Another option is Selenium. It supports all the major browsers. Further, you can use Selenium with Java, Python, Ruby, C#, JavaScript, Perl and PHP.

Conclusion

In this blog, we went over the levels of software testing and discussed the use-cases of UI and End-to-End tests. Later, we looked into the details of Puppeteer as a UI and End-to-End test tool by giving some examples. Even if we cannot look at the whole capabilities of Puppeteer API, it looks promising in terms of both ease of use and provided control over the browser to automate tests. NPM download statistics also prove that its acceptance as a tool by the community.

If you want to know more about automated testing with Puppeteer and how it could improve your business, contact us!