Property Based Testing

A few weeks ago, a colleague of mine by the name Ikechukwu Eze, bombarded me and the guys with evidence of his new discovery on writing tests more efficiently. Given my experiences with various test frameworks which claim a lot of things only to introduce more complexity, I had my reservations. However, after a quick dive into it. I've decided to share my findings. This would be part of a series of test implementations using PBT, as I'd explore its use case in other languages in subsequent articles.

What is it?

Property based testing is a refined form of Unit testing, is simply a test approach where the properties or variants of a system are defined or identified, then test cases are automatically generated to vary these properties and verify how the system handles these test cases against a predefined outcome or expectation.

Why?

The current way in which we write unit tests (Example based testing), generates false truths due to confirmation bias. This is because the developer tests and modifies the system and its properties using very limited examples.

This blinds the dev to other non-obvious loop holes and unchecked edge cases.

Example

describe('Example based tests', () => {
  it('should return -1 if text does not contain the given pattern', () => {
    expect(indexOf('abc123', 'zzz')).toBe(-1);
  });

  it('should return 0 if text contains the given pattern', () => {
    expect(indexOf('123abc', '123')).toBe(0);
  });

  it('should return 0 if empty strings are compared', () => {
    expect(indexOf('', '')).toBe(0);
  });
});

Advantages of Property-based Testing

Enhanced Test Coverage: Property-based testing generates a wide range of test cases, including edge cases, which increases the chances of finding bugs and improving test coverage.
Edge case discovery: Property-based testing encourages developers to explore the boundaries and assumptions (known or not) of their code, leading to a deeper understanding of the problem domain and their solution.
Reproducible and Independent: Property-based tests are concise and reusable, they are strictly dependent on the test suite or framework and the immediate properties of the tested system making it easier to maintain and evolve test suites as the codebase changes.
Shrinkage Feature: After a failing test, the framework tries to reduce the input to a smaller input. An example: If your test fails due to a certain character in a string the framework will run the test again with a string that only contains this certain character.

Writing Property-based tests with Typescript

As we have already said, in order to write property based tests, we need a test suite or framework. For typescript, my most preferred is Fast-Check, for the following reasons.

It is written and very actively maintained in Typescript
It is strongly typed and up-to-date with built-in types.

Writing our first tests with Fast Check

Requirements:

VsCode or any preferred code editor (I’d be using VsCode)
Deno (Javascript & Typescript runtime environment on steroids), you can use the normal NPM but you’d have to set it up with package.json and its requirements. With Deno, I just go straight to my code.
```
  brew install deno
```
Also setup Deno plugin for your code editor if you choose it with me.

Setup your project

Open a new directory on VsCode and in the terminal enter the following command to generate three files main_test.ts, main.ts, main_bench.ts etc.
```
 deno init
```

On VsCode a pop-up asking you to activate Deno extension on the workspace would come up, click enable. If it doesn’t pop up.

Enter the Key combination, CMD + shift + P
Type Deno in the dialog that pops up and hit the workspace initialization command.
You’d observe a new directory in your file explorer named .vscode with a file settings.json contained within.

For the purpose of this example, we’d be writing two functions for encoding and decoding titles of articles, then we’d test them against two conditions.
1. Encoded titles should have no spaces.
2. The result of decoding encoded titles should be exactly the same as the original title.

export function decode(input: string): string {
  return input
    .replaceAll("--", "%20")
    .replaceAll("-", " ")
    .replaceAll("%20", "-");
}

export function encode(input: string): string {
  return input.replaceAll("-", "--").replaceAll(" ", "-");
}

Above is how our main.ts is currently looking, now let’s write tests for it in main_test.ts below.

import { 
  assertEquals, 
  assert as assertTrue 
} from "https://deno.land/std@0.189.0/testing/asserts.ts"; 

import { decode, encode } from "./main.ts"; 

Deno.test(
  "decoding an encoded string should return the original string",
  () => {
    const input = "abc";
    assertEquals(decode(encode(input)), input);
  }
); 

Deno.test(
  "An encoded string should not have spaces",
  () => {
    const input = "abc ";
    assertTrue(!encode(input).includes(" "));
  }
)

What’s happening above?

We have imported assertEquals and assertTrue from deno’s test script URL, that’s one of the perks of deno, the ability to import and cache dependencies without actually installing them, unlike with npm.
We have imported our decode and encode functions from main.ts

Now on your terminal just run deno test

As expected all tests should pass, but does that really mean that our implementation is airtight? Time to find out.

import { 
  assertEquals, 
  assert as assertTrue 
} from "https://deno.land/std@0.189.0/testing/asserts.ts"; 
import { decode, encode } from "./main.ts"; 
import { string, assert, property } from "npm:fast-check";

Deno.test(
  "decoding an encoded string should return the original string",
  () => {
    assert(property(string(), (input) => {
      assertEquals(decode(encode(input)), input);
    }))
  }
)

Deno.test(
  "An encoded string should not have spaces",
  () => {
    assert(property(string(), (input) => {
      assertTrue(!encode(input).includes(" "));
    }))
  }
)

What’s happening here?

We just did a pseudo import of fast-check from npm, without running npm install, this is another perk of deno.
We have imported string, assert, and property from fast-check and have used them to rewrite the previous tests.

I’d guide you through the syntax, firstly we define the assert keyword,
```
 assert(

 )
```
Then we add the property keyword, this is what is responsible for generating the test variables.
```
 property(Type, callbackFunction(randomData: Type)=> {})
```
Type here is a function from any of the arbitraries in our case string() then we pass the random string generated into the callback function which returns our initial test. For more information check out the fast-check documentation.

Now let's run deno test again, now I’d advice we run it at least 10 times. Due to the fact that property testing involves generating random values to test, it’s important to ensure it does that sufficient amount of time to check all edge cases. By the time you’re done with that, you’d discover that the test must have failed. Here’s my result below.

If you zoom into the picture above you’d see that it ran over a billion test variables through our test case, which is way more than any developer could ever conjure. It also found out that our first test case would fail if the user entered double spaces for encoding, while it might be a very rare occurrence, this goes to demonstrate how property-based testing pushes the limits of our system, catching as many bugs as possible, thereby, drastically reducing the incidence of runtime failure.

In order to save us the numerous iterations, I’ve taken the time to rewrite the functions to suit all use cases.

 export function decode(input: string): string {
  return decodeURI(input)
    .replaceAll("--", "%20")
    .replaceAll("-", " ")
    .replaceAll("%20", "-");
}

export function encode(input: string): string {
  if (input.match(/\s{2,}|--{2,}/)) return encodeURI(input);
    const final = input.replaceAll('-', '--').replace(/\s/g, '-');
    return encodeURI(final)
}

What have I done?

We introduced decodeURI function to decode the URI-encoded characters in the input string in the decode function.
We’re using regex to check for two or more consecutive spaces or hyphens, which if present we encode the input using encodeURI.
The rest of the code base is what we’re used to already, we had to introduce the Uri functions to help us handle extreme character and spacing cases, making our general solution less verbose.

Finally, on running the tests again, you’d find out that for a greater amount of time, the test passes.

Summarily, I believe we can all appreciate the utility of property based testing well enough to study its documentation for adaptation.

Tech Nomad