przemuh.dev [EN]

Short story about optimisation

Sun, 08 Nov 2020 00:00:00 GMT

Some time ago, I posted a screenshot on Twitter showing a flame-chart from the Profiler tool. At that time, I was working on improving the performance of an application we were developing at Egnyte. A certain functionality, for a large amount of data, took an incredibly long time - 3.5 minutes! During this time, the application displayed a "spinner," and the user didn't know if something was happening or if it had frozen. After a few days of working with the Profiler, I managed to implement improvements that reduced the calculation time from 3.5 minutes to 35 seconds. In this post, I would like to describe how I achieved this.

Description of the functionality

Let's start with a description of the functionality that, to put it briefly, was lacking in terms of performance. One of the main views shows a list of folders with files containing sensitive data. These can be credit card numbers, medical data, personal data, and more. This data can fall under one of several built-in policies, such as HIPAA, GDPR, but we also allow users to define their own policy, for example, based on a previously created dictionary. Initially, the "Sensitive Content" view showed only a flat list of folders. Last year, our Product Owner, along with the UX team, concluded that working with a flat list of folders might be inefficient. Instead, a much better, and essentially more natural way of representing data would be a folder tree.

Sensitive Content List View

Tree building algorithm

There have been several approaches to folder trees in our project. They mainly relied on the unique folderId property. Unfortunately, in the case of "Sensitive Content," we couldn't use this because not all folders could contain sensitive data, and only for such folders did we receive a folderId.

/Shared/A/B/

In this case, we have 3 folders (Shared, A, B), of which only B has a folderId

There can be a multitude of such SC (Sensitive Content) locations. The performance issue appeared already at 250K locations. And it was not an exception, as confirmed by a client where we found almost a million folders. For 1M list elements, the tree-building time was 3.5 minutes. Therefore, my task was to ensure that the tree for 1M elements builds in less than 60 seconds.

Returning to the algorithm. Very simple or so it seemed :)

Take the entire path and split it into fragments according to the separator, e.g., /
Insert each folder into two structures: "tree" and "flat"
If a folder does not have a folderId, treat it as a meta-folder

For simplicity, I omit the fact that we support different data sources, and these separators can vary greatly :) Moreover, as it later turned out, some data sources can have two folders with the same name at the same nesting level 😱. And how to distinguish them? I will also skip the fact that the resulting tree was to be presented in the form of a "sparse-tree." In short - it means that if a folder contains only one sub-folder, the parent path should be collapsed/merged.

List:

/Shared/A/B/C
/Shared/A/B/D

-->
Tree:

/Shared/A/B
    /C
    /D

First, or rather second implementation

As I mentioned earlier, this was not the first tree we had to display in the application. In a completely different view, we also had to create a sparse-tree and didn't want to have several different implementations. Therefore, we wrote a simple module for building and managing the tree. It was based on two small components:

a buildTree function that took a flat array of nodes and the place (path in the tree) from which it was to insert these nodes
a "slice" from redux-toolkit that managed the tree structure (expanding, collapsing nodes, etc.)

The entire tree, or rather these two structures "tree" and "flat," were kept in redux as follows:

{
    tree: {
        path: "", // root
        children: {
            "Shared": { // path-part or folder name as a key
                path: "/Shared",
                children: {
                    "A": {
                        path: "/Shared/A",
                        children: {}
                    }
                }
            }
        }
    },

    paths: {
        "/Shared": {
            meta: true,
            ...nodeProps
        }
        "/Shared/A": {
            meta: false,
            folderId: "some-unique-id",
            ...nodeProps
        }
    }
}

This structure is obtained from the helper function buildTree.

Thanks to the use of redux-toolkit, and consequently the immer library, we could perform operations on the tree very easily:

const initialTreeState = {
  initialized: false,
  tree: {
    path: "",
    children: {},
  },
  paths: {},
}

export const createTreeSlice = treeName =>
  createSlice({
    name: treeName,
    initialState: initialTreeState,
    reducers: {
      insertTree: ({ tree, paths }, { payload }) => {
        paths = {
          ...paths,
          ...payload.paths,
        }
        const node = getNodeByPath(payload.parentPath || "", tree)
        node.children = payload.tree.children
      },

      toggleNode: ({ tree }, { payload: path }) => {
        const node = getNodeByPath(path, tree)
        node.expanded = !node.expanded
      },
    },
  })

The helper function getNodeByPath is used to search for a node by path. It can also search for a node in a sparse-tree.

First attempts and first mistakes

And everything was going smoothly, but then a client with 1 million folders came, and boom... The Product Owner sets up an Epic in Jira titled "Support 1M folders on SC tree view." A quick brainstorming session and a list full of ideas right away:

maybe build the tree on the fly, as we parse JSON?
maybe build the tree in a web-worker, at least we won't block the main thread for 3.5 minutes?
or maybe just drop everything and becoma a farmer? ⛰ 🐑

The first mistake - no one even started the Profiler to see what was taking so long. Everyone assumed that the current tree implementation was top-notch and couldn't be better. The Profiler itself, at first glance, is not a simple tool, and maybe that was the reason we jumped to ideas like building the tree "on the fly" or moving it to a web-worker.

You're probably wondering - but how on the fly? After all, when a request is made, only when the response comes does the browser parse the JSON and provide the response. Yes, but... in this case, our backend developers also had to work a bit on optimization, and instead of returning full data, they started returning our SC list as a stream. Thanks to this, we could, for example, use the oboe.js library to parse JSON on the fly.

Of course, I tried this approach because, after all, someone wrote it into the Jira task, so it had to be checked, right? 😜 Cool, the JSON was parsing "on the fly," but the stream lasted 30s instead of 10s, and I hadn't even started building the tree yet. So I gave up and decided to look elsewhere.

Web-worker

I also tested the approach with a web-worker. But I encountered a completely different problem. Okay - I can download 1M elements and build a tree based on them, but I have to send it later from the web-worker to the main thread. The tree structure is quite extensive, along with the data we saved in this flat paths structure. If we want to send such large data from one thread to another, the browser has to serialize the data, send it, and then parse it again. This also caused the browser to "freeze" during the transfer from one memory location to another. Of course, there are ways to send "directly" (without copying) through so-called Transferable Objects, e.g., ArrayBuffer, but I decided that for now, it might not be worth the effort and decided to check if our tree implementation was as great as we thought 😜

Profiler

I sat in front of the computer screen, launched the dev-tools, and pressed the "record" button in the Profiler. After a while, I got a colorful graph that reminded me of the defragmenter times of Windows 98 🤣

Flame graph

The first thing that caught my eye was this purple color, which dived very, very deep. Upon closer inspection, it turned out that a lot of these purple elements were the work of immer.js. A quick glance at the documentation and boom! A bullseye. It turns out that when "inserting" a large amount of data through immer, we can speed up this process through Object.freeze more info here. This procedure allowed me to go from 12.54s to 11.24s for 54K elements. For 1M, the jump was, of course, proportionally larger. But it still wasn't it...

From profiler to sources

Did you know that if you click on a block in the Profiler and then move to the file, you get times for individual code blocks? No!? 😎 Now you know ;)

Times before optimization for buildTree

What stands out is 229ms for building a simple string 🤯 which is the current path. It turned out that this simple oversight could be replaced with a shorter piece of code, which ultimately takes 1.7ms.

Times after optimization for buildTree

You might think - (ironically) wow... 227ms... "bravo" 👏. What is 227ms? If we look at it as a single value - indeed... micro-optimization. But remember, the goal was to handle 1M elements, and the path concatenation operation concerned each sub-folder.

Spread operator AKA Object.assign

How to make a shallow copy of an object or extend another object - nothing simpler - spread operator .... If you have to support browsers like IE11, you probably use babel.js - just like us... and such a spread operator, in the end, is translated to Object.assign (*big simplification).

Object.assign is relatively slow and can cause problems at a larger scale. In this case, I opted for simple key-by-key copying. Thanks to this simple procedure, I reduced 154ms to 44ms. And again, for individual elements, it doesn't matter at all, but when iterating over a large data set, such optimizations can work wonders.

Dan Abramov on Object.assign

Value aggregation

After "tuning" immer and removing a few Object.assign calls, or rewriting them into a simple loop, I ran out of ideas for "simple" optimizations. It was necessary to tweak the way the tree was built.

The previous implementation divided SC locations by source and built a sub-tree for each of them. For each sub-tree, aggregated values were calculated (e.g., if a folder itself contained 10 SC, but also had 50 sub-folders, we wanted to show the summed values). For each such sub-tree, summations were performed, and then the source node was updated according to the summed values for all folders.

Each such operation put something into the redux state. I thought - who needs it? Who needs it? After all, we don't show the tree until everything is calculated and updated. Therefore, I changed the code so that the entire tree, along with calculated aggregated values, is first built in memory, and then with one operation, the built tree is inserted into redux.

Moreover - in the tree requirements, it was written that certain nodes were to be expanded by default, e.g., the first level + potentially previously selected item on the list (you can go from the list to the tree with a simple button). Previously, expansion operations were triggered by the toggleNode action. I changed that too - instead of triggering a redux action, I simply change the expanded value to true directly in the node object.

You might say - numbers please! :)

For 54K elements, I went from 12.25s to 2.4s 🚀

Product Owner is over the moon.

Tests for 1M

I asked the backend developers to prepare an environment for testing 1M elements. I wanted to see if my optimizations for 54K would be justified. And the smile didn't leave my face :)

Before optimization, the tree-building time was ~3.5 minutes. After applying the above-mentioned changes, it was reduced to 59s.

Approximately ~70% savings. In total, one could say - job done - it was supposed to build in less than 60s... 59 is less than 60 😅 It's all good...

I was a bit tired of digging but a teammate rightly pointed out:

Well, nice, nice, but for me, it's still slow.

He also added later that it doesn't take anything away from me, and in his opinion, I did a great job... but it was hard not to agree with him. From the moment of clicking on the navigation element to the time the view was displayed, the user had to wait a total of 90s:

25s data retrieval (streaming)
6s browser parsing JSON
59s tree building

As a user, if I saw only a spinner ("spinner") for 90s, I would be furious :) I don't want to think about what our users felt when they had to wait 3.5 minutes... probably none of them lasted 😅

ID generation

I dived into the Profiler again. For meta-folders, a folderId was generated. This was because another place in the code needed this id (never mind). In the end, the generated id meant nothing (it was never sent to the backend). However, someone came up with the idea that this meta-folder-id should be a hash of the path...

export const createUniqueIdForLocation = path => btoa(encodeURIComponent(path))

The btoa function encodes a string as base64. It takes an average of 0.25ms... which is a fraction of a millisecond. But when you think about it more - who needs this hash? who needs it?

Exactly! If the meta-folder-id is just base64 of the path, which in fact also contained the source id, so it was unique concerning the entire list, then why even bother with this whole hash?

-            id: createUniqueIdForLocation(path),
+            id: path,
             name: getLocationName(path),

This one diff made me go from 59s to 35s for 1M elements, which gave ~40% gain 🤯

So now the client no longer waited 90s but 66s - including data retrieval and parsing! Considering that the requirements stated that the tree should be built in less than 60s, the Product Owner and clients should be satisfied 😅

Next steps

Of course, we don't rest on our laurels. Blocking the user for 60s is still a bad idea, so we continue to think about improving the implementation. Maybe we'll finally throw it into a web-worker. Who knows? Maybe I'll manage to gather material for the next post 😉.

Summary

Lesson one - instead of guessing it's better to measure.

Lesson two - if you operate on a large scale, iterate over a large data set, optimizations at the ms level for one iteration can work wonders 🚀

Lesson three - put into redux only when you're ready 💪

Lesson four - if there's no need, don't complicate the situation 😉 (see ID & btoa).

I hope that thanks to this story, you'll reach for the Profiler earlier and manage to improve the performance of more than one application.

Watch out for fixtures in cypress.io

Fri, 26 Jun 2020 00:00:00 GMT

Today I would like to tell you a story about a bug that cost me two days of searching and debugging sessions. It turned out a trivial thing, and with a better error message, it could have taken seconds instead of days. Let's go!

Hey Przemek! Could you help me?

A few days ago, I noticed that our VRT (Visual Regression Tests) suite started to fail for one case. I've asked my colleague, Monica, to check it. She accepted the challenge. After a long day of searching the root cause, she told me that she doesn't have any idea why the test is failing. On the local machine, it has been passing all the time, but on our GitlabCI, we got an error. Weird thing, isn't it? Monica was resigned and asked me for help. After two days of trying, committing, pushing, waiting, we've finally found it.

Fake server

We use a lot of tools in our tests. For unit testing, we use jest. In E2E, we use py.test with webDriver bindings. We also have UI tests that check our app on a higher level (interactions between components, pages, or views). Recently we introduced another test suite - VRT (Visual Regression Tests). The last two (UI and VRT) are based on cypress.io. It is an excellent tool for writing tests - from unit to full E2E.

Backend in our app is very complicated, and it is tough to setup a local environment. Because of that, for UI and VRT tests, we use a killer feature from cypress.io - network stubbing. Cypress can plug in between our app and network request giving us a possibility to decide about the response from API endpoint.

it("test with network stubbing", () => {
  // First, we need to start fake server
  cy.server()
  // Next, declare the route that we want to stub
  cy.route("/api/endpoint", { value: 1 })
})

More info about stub responses can be found in official Cypress documentation.

Fixtures

Fixtures are another feature from cypress.io that we use a lot, especially in our VRT suite. A fixture is a simple file that holds the data. We can reuse this file in many places. It helps us in organizing tests and managing the common responses from stubbed network requests. To load a fixture, we use a cy.fixture command. It expects a path to the file that we want to load. The path should be relative to a folder specified to hold fixtures (cypress/fixtures by default). Let's assume that we have the following file structure:

- fixtures
    - myFixture.json
    - someSubFolder
          - mySecondFixture.json

And now let's look at code which loads fixtures:

it("test with fixtures", () => {
  // We don't need to specify the file extension
  // Cypress will try to figure it out
  cy.fixture("myFixture").then(data => {
    // Here we can read the data
  })

  // We can save the fixture as an alias ...
  cy.fixture("someSubFolder/mySecondFixture").as("myAlias")

  // ...and then use the alias in stub of response
  cy.route("/api/endpoint", "@myAlias")
})

Authors of Cypress took care of reducing a boilerplate needed to use a fixture in stubbing network requests 🔥🔥🔥. The cy.route command can take a shortcut to fixture as a response argument:

cy.route("/api/path", "fixture:myFixture")
cy.route("/api/endpoint", "fx:someSubFolder/mySecondFixture")

In this way, we stubbed a network request with data kept in reusable fixture files. Great job!

Where is the hero of the story?

Ok, but where did our bug go?

I've created a simple app to visualize the issue. In the beginning, the app displays the Loading… message, then makes a request and replaces the text with a downloaded response.

Fetching the data in old, good XHR way 😎

<body>
  <div id="main">Loading...</div>
  <script>
    const mainEl = document.querySelector("#main")

    const req = new XMLHttpRequest()
    req.open("GET", "/api/endpoint", true)
    req.onreadystatechange = function() {
      if (req.readyState == 4) {
        const msg = req.status == 200 ? req.responseText : "Error"
        mainEl.innerHTML = msg
      }
    }
    req.send(null)
  </script>
</body>

I've also written a test:

describe("Simple fixture test", () => {
  it("displays response", function() {
    cy.server()
    cy.route("/api/endpoint", "fixture:examplefixture")

    cy.visit("/")

    cy.get("#main").should("have.text", "Hello")
  })
})

And created a fixture file fixtures/exampleFixture.json:

Hello

Have you noticed a bug yet?

In my case, the screenshot from the failed test was very helpful. Cypress takes them by default for failing tests, which is neat 🔥!

Screenshot from failed test

And now...Have you noticed a bug yet?

A message about the status from the stubbed request caught my attention. It was 400 instead of 200. That was a clue.

The typo and file systems

Our bug, which we've been trying to solve with Monica, was a simple typo. The name of the fixture file was in camelCase, and we tried to load it via shortcut without the same naming convention.

exampleFixture.json vs cy.route("/api", "fixture:examplefixture")

Ok, but why does it work on the local machine and doesn't on CI?

99% of our frontend team works on MacBooks. Our CI runs the tests in the docker container (Linux). You can think - "so what?". The default file system on Linux is case sensitive. On the other hand, the default file systems on Mac or Windows are not. What does it mean in practice?

On Linux you can create two files with the "same" name (different letter case):

myAwesomeFile.js
myawesomefile.js

Linux treats them as separate files. Try to do the same on Mac or Windows - you can't do it. It has also impact on the way how you load the files, for example in nodejs. On Mac, there is no difference in load file by "myFixture" or "mYFiXtURe" names - the file will be loaded. On Linux, we will get an error - file not found.

Let's check it

If we modify the code of our test in this way:

cy.route("/api/endpoint", "fixture:ExAmPlEFiXTuRe")

The test is always green on Mac. On Linux we get a 400 status for stubbed network request and an error message in console.

Screenshot with 400 for stubbed request

CypressError: The following error originated from your application code, not from Cypress.

When Cypress detects uncaught errors originating from your application it will automatically fail the current test.

This behavior is configurable, and you can choose to turn this off by listening to the `uncaught:exception` event.

https://on.cypress.io/uncaught-exception-from-application

Wait, wait, wait...WAT? The following error originated from your application code, not from Cypress. Are you sure Cypress? 🤔

Let's try to load the fixture without a shortcut:

// We made a mistake in fixture name
cy.fixture("examplEFixture").as("response")
cy.route("/api/endpoint", "@response")

// With storing fixture in an alias we can use it in our assertions
// We don't need to hardcode the "Hello" string
cy.get("@response").then(data => {
  cy.get("#main").should("have.text", data)
})

The error message for this code is quite different:

Error: A fixture file could not be found at any of the following paths:

> cypress/fixtures/examplEFixture
> cypress/fixtures/examplEFixture{{extension}}

Cypress looked for these file extensions at the provided path:
.json, .js, .coffee, .html, .txt, .csv, .png, .jpg, .jpeg, .gif, .tif, .tiff, .zip

Provide a path to an existing fixture file.

And this is the error message that I've been counting on 👏 . We know right the way where we should start looking 😎.

Summary

There are two takeaways from this story:

small typo could make you cry for two days of debugging session
you are as good as the error message from your test runner ;)

I think that Cypress could return the better message about missing fixtures than CypressError. That's why I've created an issue in cypress GitHub repository - here you can check the status.

Thank you for your attention. I am going to try to solve the issue that I've created 😉. Maybe I will be able to add something to the OpenSource community to make cypress.io even better 😁

Stop the time with cy.clock

Wed, 22 Apr 2020 17:00:00 GMT

Today I’m going to show you how to stop the time with one command. Unfortunately only in cypress.io tests. If you know how to do it in real life please DM me. It would be a very helpful tip 🙂. Ok, let’s stop joking and get our hands dirty!

App description

First, we need to have something to test. Our app will be deadly-simple. We want to display the enter time and a counter shows how many seconds we spent in the app.

Enter time:

Time on page: 0

import React from "react"

export default () => {
  const [enterDate, setEnterDate] = React.useState()
  const [counter, setCounter] = React.useState(0)

  React.useEffect(() => {
    setEnterDate(Date.now())

    const intervalId = setInterval(() => {
      setCounter(prev => prev + 1)
    }, 1000)

    return () => clearInterval(intervalId)
  }, [])

  return (
    <div>
      <p>
        Enter time: <span data-testid="enter-time">{enterDate}</span>
      </p>
      <p>
        Time on page: <span data-testid="counter">{counter}</span>
      </p>
    </div>
  )
}

Ok, we have our app. Now it is time to write some cypress tests.

We are testing!

In our test scenario we would like to check:

if the enter time is displayed properly,
if the counter increases its value after a one-second tick.

Let's try this way:

cy.visit("/")
  .get("[data-testid=enter-time]")
  .should("have.text", Date.now().toString())

The test looks decent but it doesn’t pass 😢

Assertion error

If we would like to show some formatted date (eg. 22-04-2020) instead of a number of milliseconds, then it would not be a problem. But our client wants to display milliseconds and we need to live with this requirement 😉

The cy.clock command comes with a rescue. It overrides native global functions related to time allowing them to be controlled synchronously via cy.tick() or the yielded clock object. This includes controlling:

setTimeout
clearTimeout
setInterval
clearInterval
Date

You can find more info about cy.clock in the cypress.io official documentation.

Now, let’s try to add cy.clock to our test:

cy.clock()
  .visit("/")
  .get("[data-testid=enter-time]")
  .should("have.text", Date.now().toString())

We still get an error. But this time the error message is different.

expected <span> to have text '1587547901669', but the text was '0'

What is going on with that 0? Because the time represented in timestamp value is a number of seconds passed from the start of Unix epoch (1st January 1970). We could ask what will happen after the 19th of January 2038 but this is a topic for another blog post 🙂.

Calling cy.clock without any arguments sets the date in our app to 1st January 1970. We could change it by passing an argument to the cy.clock:

const now = Date.now()
cy.clock(now)

Right now, with a little luck, our test will pass. It depends on how fast our computer is 😄. To fix this issue we need to remember that cy.clock overrides the time in our app, not in our tests (command chain). That’s why we need to change Date.now() in our assertion to now value that we’ve created at the beginning of the test.

const now = Date.now()
cy.clock(now)
  .visit("/")
  .get("[data-testid=enter-time]")
  .should("have.text", now.toString())

The test is green - always! - success! But there is one little difference in how our app works now. Before using cy.clock our timer has been running. Right now it stops on 0. Fortunately, it is expected behavior in our test-case scenario. We've set and stopped the time.

In order to move the time with some value, we need to call cy.tick command:

const now = Date.now()
cy.clock(now)
  .visit("/")
  .get("[data-testid=enter-time]")
  .should("have.text", now.toString())
  .get("[data-testid=counter]")
  .should("have.text", "0")
  .tick(1000)
  .get("[data-testid=counter]")
  .should("have.text", "1")

Tada 🎉! We've just wrote the test checking the enter date and the value of the counter.

What if we would like to set the date only - without stopping the time? 🤔

That’s a great question. Sometimes we would like to override the Date object only, leaving the rest untouched (setTimeout, etc.). In this case, we need to pass a second argument to the cy.clock - an array of timing functions that we want to override.

cy.clock(Date.UTC(2020, 3, 22), ["Date"])

In this example we set the date/time to 22th April 2020 00:00 UTC (yeap - months in Date starts from 0 that's why April = 3 🙂). In the same time we don't override the setTimeout and the rest time functions.

That's all for today. I hope that with this knowledge you can go now and stop the time in your tests 😉

Good luck!

3 Steps to Awesome Test Reports with Cypress

Wed, 18 Dec 2019 00:00:00 GMT

In this article, you will learn how to generate informative test reports with Cypress and how to enrich them with some screenshot context. This will help you to fix your potential bugs way faster 😄 All you need is three simple steps.

At Egnyte we ❤️ to test

Keeping the highest possible quality of the product is one of our top priorities at Egnyte. That's why we love to test. But our applications are rather large and relying on manual testing would be exhausting for us. That's why test automation and Continuous Integration techniques are our best friends. We write a lot of tests: unit, integration, end-to-end, module, etc. The most important part is that, at the end of the day, if our Jenkins pipeline is green, we are sure that we didn't break any parts of the system.

So where is the problem? Didn't you know that tests not always pass? And that's fine :) We don't need to panic right away. First, let's calm down, get into the test report on Jenkins, check what is broken, and fix it. That's it. The problem is that the test report is very often just a plain error message plus a stack trace. And it's enough for unit tests or integration tests for our React components, redux connections, and so on. On the other hand, this is not always helpful for tests that are run in the browser. Let's imagine the following result from a test:

Jenkins console output

I took this failed test report from our data governance product (Egnyte Protect), which is one of our core products. For writing integration-UI tests, we use an awesome tool called cypress.io. I must admit that Cypress and cypress-testing-library are doing an excellent job in terms of error messages. Judging by the test report shown above, it is clear that we cannot find an element with matching text. Of course. But what is the visual state of the app? As a developer of Egnyte Protect, I know that this message should appear in a dialog. Has this dialog been opened? Or maybe it is only a typo? So many questions and no answers. If we wanted to check it, we would need to run the test locally once again and see what the visual state of the app is. Only then we would know (spoiler alert) that we have a typo :).

What if we displayed the visual state of the app right in the Jenkins report?

Screenshot from the app

Wow! Now we know that the dialog is opened, and that the subheader text is incorrect! We have some valuable context just from reading the test report enriched with a single screenshot.

So how we could add screenshots to our test reports? Let's find out!

HTML reports to the rescue!

Cypress is based on mocha.js. And this is great because mocha.js is a very mature project with many custom extensions. Test results can be generated within elements called reporters. We can write our custom reporter or use an existing one, for example, mochawesome. As the name suggests, it generates AWESOME reports! Badum tsss.

And now, I would like to show you how to integrate mochawesome with cypress to generate HTML reports with a screenshot context for failed tests. For the sake of this blog post, I've used an example repo cypress-example-kitchensink. We will do it in 3 simple steps. Let's get our hands dirty!

Step 1 - set up the reporter

First, we need to install proper reporters. Yes, that's right - plural - reporters. We still want to see test results in the console. Maybe you also want to have a JUnit XML report. We need to have one reporter per expected outcome (console, HTML, XML). In order to set up many reporters, we will use the cypress-multi-reporters package. On top of that, we also need mocha and, of course, mochawesome.

npm install --save-dev mocha cypress-multi-reporters mochawesome

Or if you use yarn:

yarn add -D mocha cypress-multi-reporters mochawesome

Then, in the cypress.config file, we need to specify which reporter we want to use:

{
  "reporter": "cypress-multi-reporters",
    "reporterOptions": {
      "configFile": "reporter-config.json"
    }
}

The configFile field points to the reporters configuration file. We need to add this file to our repository. For each of the reporters we can specify some options. Let's do that for the mochawesome reporter:

{
    "reporterEnabled": "mochawesome",
    "mochawesomeReporterOptions": {
        "reportDir": "cypress/results/json",
        "overwrite": false,
        "html": false,
        "json": true
    }
}

In this fragment of config, we specify an output directory for the results file. We want to collect only the JSON files for each spec file. That's why the html flag has been set to false. Because cypress is able to run tests in parallel, we need to set the overwrite flag to false. It means that for each spec file, we will generate a separate file. In our case, these will be JSON files.

Let's try to run our tests via npm run local:run command.

Running:  examples/location.spec.js                                                      (9 of 19)
Location
    ✓ cy.hash() - get the current URL hash (169ms)
    ✓ cy.location() - get window.location (101ms)
    ✓ cy.url() - get the current URL (78ms)
3 passing (1s)
[mochawesome] Report JSON saved to /Users/przemuh/dev/cypress-example-kitchensink/cypress/results/json/mochawesome_008.json

As you can see, after the spec reporter results, we received information that the mochawesome_008.json file has been created. Each of the spec files generated a JSON with results.

List of generated results

We are ready to go to the next step.

Step 2 - generate the report

We've collected the test results. Now, we need to merge them into one file and generate an HTML report based on it. We will use the mochawesome-merge tool to merge result files. Let's install it.

npm i --save-dev mochawesome-merge
yarn add -D mochawesome-merge

Now, let's add an npm script which will be responsible for running the merge tool.

"report:merge": "mochawesome-merge --reportDir cypress/results/json > cypress/results/mochawesome-bundle.json"

The reportDir flag specifies where we keep results files. The output of the command is passed from stdout to the mochawesome-bundle.json. One caveat here: the result of the merge needs to be put in a different folder than where the single results file is.

After merging we are ready to generate a final HTML report. In this case, we will use mochawesome-report-generator.

npm i --save-dev mochawesome-report-generator
yarn add -D  mochawesome-report-generator

Let's create an npm script for that action:

"report:generate": "marge cypress/results/mochawesome-bundle.json -o cypress/reports/html"

Marge is a short form of MochawesomeReportGEnerator in case you've been wondering :)

Once the script has run, our awesome HTML report should appear in the cypress/results/html folder.

HTML report view

There is one more thing to do. Add a screenshot to tests that have failed.

Step 3 - add screenshot context

Cypress automatically generates screenshots for failed tests in the cypress/screenshots folder. You can disable this behavior if you want. Screenshots are collected within the following folder structure:

path-to-the-specfile/spec.file.js/context - describe - describe - testTitle (failed).png

For example, the following test placed in examples/actions.spec.js:

context('Actions', () => {
  context("nested context", () => {
      it('.type() - type into a DOM element', () => {})
   })
})

will generate something like this on fail:

Folder with screenshot from failed test

Ok, so how we can connect these two elements: a screenshot generated by Cypress and a test result generated by a mochawesome reporter?

First, let's copy our generated screenshots to the folder where we keep the HTML reports. In order to do this, we will use an npm script:

"report:copyScreenshots": "cp -r cypress/screenshots cypress/results/html/screenshots"

Next, we will use the cypress/support/index.js file and write some code that will be listening on the test:after:run event.

Cypress.on("test:after:run", (test, runnable) => {
    if (test.state === "failed") {
        // do something
    }
});

For adding the screenshot to the test result, we need to use the addContext method from the mochawesome/addContext package. This method takes two arguments: an object with the test, and the context. If the context is a valid URL (could be a local path) to the image, then that image will be displayed. To see more details, visit the documentation page.

import addContext from 'mochawesome/addContext'

Cypress.on("test:after:run", (test, runnable) => {
    if (test.state === "failed") {
        const imageUrl = "?";
        addContext({ test }, imageUrl);
    }
});

Ok - but how to define the imageUrl? This is a time for magic to happen.

Just kidding :) we will use the runnable object. As we saw earlier, Cypress generates the name of the screenshot based on the test suite structure. We need to re-create that.

Cypress.on('test:after:run', (test, runnable) => {
  if (test.state === 'failed') {
    let item = runnable
    const nameParts = [runnable.title]

    // Iterate through all parents and grab the titles
    while (item.parent) {
      nameParts.unshift(item.parent.title)
      item = item.parent
    }

    const fullTestName = nameParts
            .filter(Boolean)
            .join(' -- ')           // this is how cypress joins the test title fragments

    const imageUrl = `screenshots/${
      Cypress.spec.name
    }/${fullTestName} (failed).png`

    addContext({ test }, imageUrl)
  }
})

From now on, if our test fails, a context field with a local URL to the image will appear in the JSON results file:

{
  "title": ".type() - type into a DOM element",
  "fullTitle": "Actions .type() - type into a DOM element",
  "timedOut": null,
  "duration": 10395,
  "state": "failed",
  "speed": null,
  "pass": false,
  "fail": true,
  "pending": false,
  "context": "screenshots/examples/actions.spec.js/Actions -- .type() - type into a DOM element (failed).png",
}

What is more, the image itself will be attached to the HTML report.

HTML report with screenshot context

TADA 🎉 We got it!

You can check all necessary code changes that we have done in the following pull request: https://github.com/przemuh/cypress-example-kitchensink/pull/1/files

Optional steps

You might want to add cypress/results and cypress/reports folders to your .gitignore.

It would be good to remove screenshots, results, and reports before the next test run. It can be done by a simple npm script. In our example repo, I've added:

"precy:run": "rm -rf cypress/screenshots cypress/results cypress/reports"

"Pre" means that this script will be run before every cy:run. See npm docs for more details.

In the example repo, there is a npm-run-all package installed. We could use it to run in sequence: merge, generate report and copy screenshots scripts in one command:

"report": "run-s report:*",
"report:merge": "mochawesome-merge --reportDir cypress/results/json > cypress/results/mochawesome-bundle.json",
"report:generate": "marge cypress/results/mochawesome-bundle.json -o cypress/reports/html",
"report:copyScreenshots": "cp -r cypress/screenshots cypress/reports/html/screenshots"

There is also one caveat. The file name in most systems is limited to 255 characters. So what will happen when we have a very nested structure of a test suite with long descriptions? It's simple - our file name will be truncated. Cypress truncates the full test name to 220 characters. So we could also do the same in our code:

const MAX_SPEC_NAME_LENGTH = 220;
const fullTestName = nameParts
    .filter(Boolean)
    .join(" -- ")
    .slice(0, MAX_SPEC_NAME_LENGTH);

But this is an implementation detail. We don't know whether Cypress devs are about to change that number. So, a better option would be to read an article from Kent C Dodds about avoiding nesting when you are testing.

Wrap-up time

I hope that this article will help you to set up awesome HTML reports in your project. It helps us a lot when it comes to quick investigations of why a given test is failing. Let's recap what we did here:

Install and set up the mochawesome reporter.
Collect test results and generate an HTML report based on the merged JSON file
Add screenshot context with an addContext function.

You can check all code changes here in this pull request. And of course, after you generate the HTML report you need to connect it somehow to your Continuous Integration tool. But this is a story for a separate post. :)

Now…are you ready to create your own Cypress HTML reports?