We are a Melbourne-based nearshore software outsourcing company providing world-class dedicated tech talent and development expertise to companies listed in the Australian Share Market to innovative startups who are shaping the future.

Contacts Us

Melbourne
Suite 37/11, Wilson St,
South Yarra,
VIC 3141.

Colombo
19, Katukurunduwatta Road, Ratmalana,

info@unibench.com.au

Melbourne
1300 320 487

Colombo
+94 769 360 433

Uncategorized

Bring human intelligence to automated testing using approval tests

1_kZ1-qbaIvG-8L-bvYeK02A Bring human intelligence to automated testing using approval tests

How many times did you check the output of what you are testing and then add that value as the expected value in your assertions? How many times did you write many assertions in your test code that make you feel bad, but you have no option? Next time when you are in such a situation Approval testing might make your life easier.

Table of Content

· What is Approval Testing?
· Why Use Approval Tests
· The Difference Between Traditional Asserts And Approval Tests
· More About Approved and Received Files
· Example Use of ApprovalTests Library
· Other Important Concepts And Features of The Library
∘ Reporters
∘ Scrubbers
∘ Configuration

What is Approval Testing?

Approval testing embraces human verification instead of simple assertions in a context where verification is not trivial. Think about a piece of code that you are trying to use to produce an image or audio, how can you do the assertion in your test? probably you might need to write additional code for the assertion, so do you need to test that code also?😁. Although computers can’t verify it as easily as humans. We can verify it very easily, by just listening to the audio or seeing the image. Approval tests embrace this factor.

In approval tests, we run the tests and get the output in a verifiable way like an image an audio a text file, or any kind of file that can be verified the correctness just by looking at it. We saved this verified output as an approved file. If future tests produce anything different than the one we approved the test will fail and the important part is now you can see the difference between the previously approved file and the currently generated file using an applicable diffing tool.

Now you have the idea so you might like to implement it, luckily there is a library called ApprovatTests which has its implementation in many popular programming languages. Before seeing an example let’s understand more about approval testing.

Why Use Approval Tests

  • Assert on complex test output with fewer lines of code
  • Utilizing human intelligence for verification in an automated way
  • Start testing legacy code more easily
  • Facilitate customer/stakeholder collaboration in testing because they can engage in the approval process

The Difference Between Traditional Asserts And Approval Tests

1_-esLwm2KG_NjYGlE2dE2WA Bring human intelligence to automated testing using approval tests

More About Approved and Received Files

When we run our test the first time it should fail because we haven’t approved anything yet. The output of the test will be created with file name suffixes with .received.* (* means file extension). To make it pass the test we have to check the content and rename this file to *.approved.* From next time onwards when a change occurs we can compare this received file with the approved file using an applicable diffing tool. So it’s clear that we should not commit the received files and we should commit the approved files. (The exact file names will be discussed in the example section). You don’t need to delete received files manually, when the received file content matches with the approved file it will be deleted by the test.

So a test will fail when

  • There is no approved file
  • The content of approved and received files is different
  • When an exception is raised during the test

Example Use of ApprovalTests Library

I will use the Java library to show the power of this technique. All the examples will be available in this GitHub repository. I will not discuss how to set up a Java project and run tests on it. If you are not familiar with Java don’t worry, the concepts will be the same and you will most probably find a library implementation for your preferred language on the ApprovatTests site.

Let’s consider a traditional unit test that validates a model object using junit5.

    @Test
    public void traditionTesting() {
        Customer customer = customerService.createCustomer();

        assertEquals(1L, customer.getId());
        assertEquals(30, customer.getAge());
        assertEquals("vinod", customer.getFirstName());
        assertEquals("madubashana", customer.getLastName());
        assertEquals("test@gmail.com", customer.getEmail());
        assertEquals("Sri Lanka", customer.getCountry());
        assertEquals("address", customer.getAddress());
        assertEquals("vinod", customer.getUserName());
        assertTrue(customer.isActive());
    }

Let’s use the approval tests library. (Assume there is a toString method implemented in the Customer class which is responsible for producing the printable format of objects in Java)

@Test
public void approvalTestingUsingTextRepresentation() {
    Customer customer = customerService.createCustomer();
    Approvals.verify(customer);
}          

Run this test (use your IDE, maven, gradle, or any method). At the first run, it will fail because there is nothing that is approved yet, but it creates a file named ..received.txt. The content of this file will be the string representation of the customer class (The output of the toString method in this case). If your operating system or IDE has any text diffing tool it will open automatically and show the diff this file with another file named ..approved.txt. Check the content of the received file and move it to the approved file if the output is as expected. Now you have approved the output as it is correct. As mentioned when you run the test again it will pass. The received file also will be deleted. If in the future this test fails you can compare the difference between the two files visualize what goes wrong and update the test easily.

To summarize you pass verifiable content into the library and the library will produce a visually verifiable file (received file) and you verify the content and save the verified result (approved file). When a test runs and produces different content than the approved result it fails and you can easily see the difference between the approved and the current version. (Using a diffing tool, if available for the content type)

Tip: There are many verify methods available for different scenarios. You can use your IDE’s code suggestions to select the correct verify method

You can find more examples in this GitHub repository.

Note: Sometimes your IDE might format your files automatically for example remove trailing whitespaces in lines, which can cause tests to fail each time if the received file contains trailing whitespaces. So be careful about changes done to your files automatically by the settings of your IDE.

Other Important Concepts And Features of The Library

For the Java library, you can learn many concepts and examples from https://github.com/approvals/ApprovalTests.Java/tree/master/approvaltests. I will briefly go through some important concepts and provide references for more details.

Reporters

Reporters are used to give feedback on failures. They are run when the test is failed. For example, these are the ones that will open a diffing tool and show the diff when a test fails in our development setup. You can find many details about how to configure them, use different diffing tools, and many more at https://github.com/approvals/ApprovalTests.Java/blob/master/approvaltests/docs/reference/Reporters.md

Some of the useful examples of using reports in the development phase

  • If you are lazy to rename the file from receive to approved in the first test run there is a clipboard reporter that will copy a command to the clipboard which will do that. Just past it in a terminal and execute.
  • Verify images using BeyondCompareReporter or P4MergeReporter (These reporters use external tools and install them before using the reporter. Make sure that the reporter works with your operating system too)

But what about in your CI environment? Do you need to open a diffing tool while you are running your production builds?😊No, you don’t. You can configure Front Loaded Reporters which will block the normal reporters and trigger environment-aware reporters. You also can commit received files in your CI environment

Tip: Changing your test names might end up having obsolete approved files in your repository. To prevent this you can explicitly give names to your files

 Approvals.verify("verifying content",
                new Options()
                        .forFile()
                        .withBaseName("custom_base_name")
        );

Tip: Sometimes you might need multiple reporters. For example, HTML files can be verified using a browser with the help of FileLauncherReporter, although the output visualized in the browser as expected, semantic HTML text might not be correct, hence DiffingReporter might also need to compare the text value of the HTML

Scrubbers

When you have some dynamic content in your tests and you need to replace them with stable values in the files you can use scrubbers. For example, random UUIDs and dates can be replaced using srubbers. More details available at

https://github.com/approvals/ApprovalTests.Java/blob/master/approvaltests/docs/Scrubbers.md

Configuration

There are many customizable options documented in

https://github.com/approvals/ApprovalTests.Java/blob/master/approvaltests/docs/Configuration.md

Leave a comment

Your email address will not be published. Required fields are marked *