I didn’t see that one coming

The world works best when QA professionals do not get surprised a lot.
Actually never would be just fine. After some time you really do see it all, right?

Wrong.

There are some laws of nature that we tend to forget, or just push aside, but always comes back, like a karmic
boomerang of truth. If your system is more than trivial, and you run a database profiler, you will learn something new.
If you run a profiler on your code and stress it, you will learn something new.

If you run your system on real world (“production”) data – you will learn something new.

Mainly, that you have some unexpected bugs.

Keeping it real

Real world data have three things going for it:

  1. There is a lot of it.
  2. It is diverse, and in it’s way, devious.
  3. It mirrors issues with parts of the system that are not currently under test, including 3rd party issues.
  4. If your system has elements open to the world, you get calls that are invalid in just any permutation you can come up with.

1. Bigger is always bigger

All my tests ran like magic. Everything ran fast, every assertion met with.

UI tests ran without as much as a hiccup every time.

SLA? A joke. All APIs are 10 times faster than needed.

All batch processes completed their tasks (even though they were all configured to run at midnight).

No database deadlocks.

No timeouts on clients that consume our services.

Let’s publish……?

Enter database imported from production* (see important remark below),
that for good measures I bombarded with pseudo users and activity, until in became 5 times larger.

My UI tests failed because they were poorly synchronized. When all APIs returned a result in 10 msec, it went fine.
When the response time stretched, my UI tests asserted on controls that had no data.

We might still be, in a very positive thinking way, within the SLA limits for the current user base, but in no way can support it if it doubled (a long way from the x 10  factor that would let me sleep better).

Batch processes blocked each other, and took forever to complete.

Database had several deadlocks, since procedures took so long to run they could not stay out of each other’s way.
(We needed a better approach for our stored procedures, which required profiling the database activity and creating some new indexes).

Our service consuming applications just gave up from time to time. Their default time out was way to optimistic, which was a bug on it’s own.

 

2. People have messed up names

We all know to expect a few special characters in people’s names. That’s not new. But sometimes they are so special that your code, collation or encoding fails you. User input also tend to be longer than expected, especially when someone manages to cram an entire form worth of text into one field.

Do you think you can come up with all the input fields validations? If you missed one, some user will found it for you.

But even if we skip the human element of data generation, there is still a great benefit to using real world data – it’s not fake, and does not carry our biases we hold when generating fake data.

Long ago, when responsive web sites were at their infancy, I tested a mechanism that custom tailored landing pages to be viewed on old time mobile phones, image resize and everything. It relied on effective caching of user agent headers and their matched set of properties.
Testing it against fake data always worked, since the repetitive nature of the fake calls created a very small cache. I only approved it after importing tens of thousands of production calls, and stressing the system using data driven tests.

 

3. Fault vs Responsibility

If something is wrong with the SUT, and it goes into production, it is definitely your fault and responsibility.
If a bug, not related to the SUT reaches production, it might not be your fault, but since it affects your system,  it is still your responsibility.

Do you remember that people have messed up (sorry, original and diverse) names? Well, we adjusted our code and learned to live with it. But guess what – we also exported these names to a 3rd party service that went gaga over those special characters. Not my fault, still my bacon on the line.

 

4. A thousand monkeys typing on assault rifles

If your system is in any way opened to the world (a web site is the most trivial example) you can expect to get all the valid format calls, which is cool, but also all the permutations of invalid calls, which seems to be out to get you.

I was once heading the QA for an affiliate network. We had thousands of ad banners floating around the internet, all leading to our servers, with structured, consistently formatted links. All machine generated, no pesky humans to interfere. Over the years, and the millions of beautifully formatted HTTP requests, we also had just about any character combination.
We had Google search terms in Chinese that found their way to the HTTP call. Parameters cut in half, doubled, doubled with different values, missing, concatenated,. Characters where numbers were expected, missing headers, extremely long headers, and just about anything you think of, and most of what you cannot.

Each of these “gems”, when processed and eventually stored in a database, managed to find a fault in the system. A naive piece of code, that like a human cell, was fooled by a pathogen’s receptor, and tried to process the invalid data. On the good cases, the call was discarded. On the bad cases, things crashed, failed huge batches of data, or stored messed up data to the database. This example, so far, has nothing to do with testing. But all products we released after each failure, tested for that scenario.

* An important remark regarding production databases

In some cases, especially in highly regulated domains (like financial data), you can’t just move the production database from its well protected home. Usually development teams can receive, upon request, a copy of the data base with all sensitive data hashed beyond all criminal use. This is unavoidable. However, try to make sure the database is not stripped out of its interesting bits. There is no real interest in a database that looks like FirstName1,Address1,SecurityQuestion1 etc……