Let’s get the formality out of the way – what is property based testing

In the very narrow definition, a property based test is comparing and verifying that a program’s output complies with a rule (property), rather than a specific value.
That being said, add this to the mix (Taken from scalatest.org):
“…a property is a high-level specification of behavior that should hold for a range of data points”.

Translation

Property based testing involves testing your code’s output against one (or more) constraints, for one (or more) input values.

In real life

Property based testing is usually used by running multiple, randomized (or otherwise entered by the test developer) input values, and matching the output values to a set of business rules.

Why do I need this in my life?

•  All odd numbers are prime – Random data is (sometimes) smarter than you. Each test run gives you multiple values, that to some degree, were never used before. Some of them are exactly the secret key needed to break the code.
• Real world data is your enemy in production, but your friend in testing – Dumping a load of production derived input values tests your code against the threats that it is expected to meet in real life (see here).
• Unknown unknowns – What you think is wrong with the code (known) should be fixed on the spot. Areas that may hold issues (unknowns) should be tested specifically. Dangerous areas that you don’t know that you don’t know about (unknown unknowns) will not be encountered unless you cast a very wide net, spanning more than you can see. Combining a multitude of input values with many constraints will help you stumble upon them.
• “The only doctor that could have saved me was me” – Sometimes, producing the specific expected value is too resource costly, or only possible using the very same code you are trying to test (nullifying the effort).

Getting our hands dirty (with kinetic sand) – a trivial example

Usually, this is the place you will see the use of addition as the code, and “associative”, “commutative”, “additive” and “additive inverse” as testable properties.
Or in other words:
(x + y) + z = x + (y + z)
x + y = y + x
x + 0 = x
x – x = 0

There is exactly an infinity of versions of code that will work great for 1,2,3 and break somewhere else. You can even pick any X,Y,Z you wish, some code versions will do addition just great, and fail elsewhere. With a simple randomizer, you can now run the addition test within seconds, covering thousands / millions / gagillions of values (depending on computer power). Caveat: read the last section about not randomizing everything, it is just as important as the rest of the stuff here.

Here you go. Trivial example done. Take a look at this presentation by Scott Wlaschin, going over this very scenario.

Getting our hands dirty (with krazy glue and glitter) – a real life example

OK, training wheels off. Time to see how all this relates to our real life.
This example tests a well known algorithm, that has a several business rule constraints added to it.

Once upon a time, we discovered that certain objects, consisted of a list of arrays and several other simple properties, are sent to a 3rd party web service, via a very slow SOAP call. So slow, in fact, that grouping small objects to bigger mega objects (which means combining the array lists to one list) and then sending them out, saves a lot of time. The array lists, however, are limited in size. So the challenge was to unite the objects in a way that will resolve in as few as possible mega objects. Yes, before unleashing the business rules that will soon appear, this is a classic “knapsack problem“, and as such, has a well known solution.

Enter the complications, and analogies. Remember the “other simple properties” mentioned above? Well, here is where they shine. Let’s pretend we run a hotel, catering for families of one to X members. Currently, as soon as a family (of 1 to X people) arrives, they take a painfully slow elevator to their assigned floor. A family in an elevator represents the object, families in an elevator the optimized object, and person being a single array. We want to cram as many people as possible into each elevator, minimizing the amount of slow rides. So far, we are still at the knapsack problem domain, so let’s sprinkle some business rules.

• Rule A – We don’t break up families. All family members ride together, they will always go to the same floor.
• Rule B – All passengers in the elevator goes to the same floor (the elevator has just one stop before going back, empty, to the lobby).
• Rule C – An elevator can only carry up to X people (yes, same X from the maximal family size).

Since the elevator ride is so slow, and the elevator is now (hopefully) packed, we need to do what we can to negate the invasion of personal space. For this reason, we choose to avoid placing in the same elevator more than one person who tends to engage in endless conversations relating a to something they are very passionate about. So let’s add a rule for that.

• Rule D – Only 0 or 1 persons on each ride can practice a specific branded fitness OR diet regimen. The same person can have both properties, which is still OK, but for the sake of everybody’s ears, there should never be two or more people that have any of the two properties.

Before even thinking about automating anything, we manually constructed a few combinations to prove the optimization algorithm work, and since I could not help myself, some edge cases. Since there was no manual way to actually test these scenarios, I hard coded them into the unit tests that will later run the property based testing as well.

Now for the a meat of it. I used a unit test platform to interact directly with a DLL provided by the feature developer.

The tests consisted of two sections:

1. Hard coded scenarios, as mentioned before. Since in this case I knew exactly how the result should be, I could assert the exact optimization (go over each object of the result, or the content of each elevator ride, if you will).
2. Multiple test runs of random and semi random data. In this case, I don’t exactly know how the result is going to look like, so I’m not testing the quality of the optimization, but rather the validness of the result.

Let slip the dogs of war

At the heart of it, there are two questions we should ask ourselves:
What generates our input, and how to assert the output.

The first is story specific, so I will run over it quickly. I used random data, and, more interestingly, data from production.
As for assertions, the answer might be long and complicated, but you already know it by now. It is the set of the business rules mentioned before, plus “implicit” business rules. So we assert rules A, B, C, D on each optimized object, and add some assertions that are bound to be true if all works well, but are too downright silly for a business user to specify. For example – no object should be empty of arrays (the elevator never rides up empty), the amount of arrays in the output is the same as the input (the feature did not break up, remove, copy or just invent an array. In other words, no person was left behind or replicated by the elevator. We leave this behavior to the transporters, when brave people choose to be beamed to their rooms). The overall amount of objects should be the same or smaller than the input version. We can always take it a step further, and checksum the sorted content of the concatenated arrays, before and after the optimization. It must be the same.

No man is an island

So, are we done now?
Not yet, since there is one caveat that we should always remember.  Randomizing input data tends to take our mind away from the very important basic methods we do manually, mainly edge cases, boundary value analysis and equivalence partitioning.

If you test a simple arithmetic function that does something with three input integers, and randomize the values between 0 and one million, you will indeed cover a lot of ground. However, in most test runs (and with a slight round down, ALL test runs) you will not get three zeroes (or even a single one), three single digits, three same numbers, three max number etc.

Always remember that while automation is fun and useful, it is in no case intelligent. Automation is magical brooms that can flood the house with well water, a Golem that knows nothing but its literal instruction set. It is a small minded entity that only raises the alarm for the most specific of issues. You are the intelligent free thinking person that everybody are counting on to save the company from production disasters. Don’t let that part of you to grow sleepy.