The Mother-in-Law’s Guide to Software Testing

This is part of my Mother-in-Law’s Guide to Technology. My Mother-in-Law is a very smart woman even if she isn’t a “computer person.” The goal of this series is to take some big and treacherous sounding ideas and bring them down to earth.

Dearest Mother-in-Law,

Remember when you had kids and you told them to do stuff. And remember how they used to do what you told them but that wasn’t always what you intended them to do? Well, that’s the way computer programs work.

Just like kids, computer programs will do what you tell them, but beyond that, all bets are off. They don’t do anything that directly contradicts what you said but that doesn’t mean they’ll do what you want them to do.

That’s why you need test kids (or computer programs). Once you give them instructions, you need to make sure that these instructions are understood and that the kids will carry out the spirit of the instructions, not just the letter of them. You also need to make sure that the kids will know what to do in ambiguous situations and that the kids are positioned well to defend against any bad actors. You do this by testing the kids by watching them perform the task once or twice in a variety of circumstances. It’s much better to find out any problems or ambiguity in your instructions before the kids are sent out into the wild.

Let’s look at two kinds of kids that are representative of common software errors. There are many others, but these are two of the big ones. Testing allows you to find these errors early in the process, preventing these issues from causing big problems later on.

The Happy Path Kid

This kind of kid is naively optimistic. He also may not be very bright. When you tell him something he always says, “Of course. Sure. I can do that!” But he almost certainly doesn’t understand all the nuances of what you want him to do. If the instructions don’t exactly match the situation he’s in, trouble quickly ensues. For example, imagine this situation:

You yell at him, “That was so dangerous. I told you never to cross the street when the light is red. What were you thinking! You almost killed yourself!”

Only to hear him say back, “But there was no red light. In fact, there was no light at all.  So I crossed the street.”

When thinking about the Happy Path kid, make sure you examine all of the possible ambiguous situations so that he (or the program) doesn’t run into any confusing situations.

The Adversary

Then there’s the kid who intentionally tries to skirt the rules. He’s far more clever than you are. He doesn’t break any rules but you’re continually surprised how he can turn a little imprecision into a HUGE opportunity for misadventure. He’s the kind of kid who always wins at Scrabble because he’s looking up all the words on his phone. You assumed he wouldn’t do that but you never SAID he couldn’t.

While the Happy Path Kid may encounter ambiguous situations and be led astray, the Adversary will look for weaknesses in your system and figure out how to exploit them.

Here are two stories from my time at Yale where I ran into real-life examples of these “problem children.”

The Happy Path Problem: Heads

This situation came up during my freshman year. It was a beautiful day and we were sitting outside in a seminar called “Perspectives on Science.” As we were sitting on the lawn under a tree, one of my classmates was performing a demonstration of how probability works. If you flip a coin a large number of times, half of the time will be heads and the other half tails.

“Can someone give me a quarter?” she asked the group. My friend Christine excitedly reached into her pocket and grabbed a quarter. The first person flipped the coin.

“Heads,” they said.

“Heads,” the second said.

“Heads.”

“Heads.”

“Heads.”

“Heads.”

“Heads.”

“Heads.”

Flipping a Coin from Rosencrantz and Guildenstern are Dead

By this time the quarter came back to the leader who examined the coin. “Who walks around with a two-headed quarter?!” she blurted out in surprise. As it turns out, Christine did. At the time, we were obsessed about Tom Stoppard’s Rosencrantz and Guildenstern are  Dead which includes a scene where the laws of probability are broken with a coin that continuously lands on heads. So Christine got one.

The Adversary Problem: How to Cheat at Hangman

When I was at Yale, I took our most famous computer science class, CS223, with Professor Stanley Eisenstat. This is where I first learned about the adversary.

Professor Eisenstat Teaching How to Cheat at Hangman (pic via Twitter)

Professor Eisenstat Teaching How to Cheat at Hangman (pic via Twitter)

Professor Eisenstat wrote out a game of hangman on the board with 3 letters filled out:

_ill

We had 8 guesses to get this right. The class started shouting out different possibilities. We started very confidently with

bill

dill

fill

gill

hill

This went on for quite a while as we gradually lost that confidence. Then Professor Eisenstat told us that there were many different words that this could be — far more than 8. Bill, Dill, Fill, Gill, Hill, Jill, Kill, Mill, Pill, Sill, Till, and Will. That’s 12 words if you’re counting. Here’s where the adversary comes in. Because Professor Eisenstat hadn’t committed to an answer beforehand, there was no way that we could win the game. When we chose a letter, he removed that word from the set of possible winners. He always had an option that we hadn’t chosen.

Summing Up

We are in an age when software is part of everything we do. We don’t have finance anymore but finance + computers. We don’t have cars anymore but cars + computers. With software being such an integral part of everything we do, it’s even more important to ensure that software does what we intend it to do.

Let’s take cars for example. We are on the cusp of self-driving cars. The happy path for self-driving cars is pretty easy — driving on a highway on a sunny day. However, there are many unhappy surprises that the car will encounter like people walking across a street carrying a giant piece of plywood or human drivers falling asleep at the wheel. The adversary in the self-driving car example is even more interesting. You have people trying to take advantage of the polite self-driving car and street signs can be modified so that cars will incorrectly identify them

Given how much we rely on computer code, let’s make sure that our software does exactly what we want it to do!

Note: I previous version of this post was titled Be Careful with Your Assumptions OR Who Would Have Thought that Would Happen?

How I (Re-)Built My Favorite T-Shirt

Read in the voice of the Mission Impossible announcer: This t-shirt was originally created as a protest against US Export laws. Until 2000, US export law considered the computer code on the shirt as a “munition” that should not be exported from the United States or shown to a foreign national. Your mission, if you choose to accept it, is to re-create this shirt.

The T-Shirt

When I was at the GEL conference in 2016, I met a woman who worked for the website Design a Shirt. We started talking about the most creative t-shirts we’d ever seen. This brought me back to the late 1990s when I discovered one of the most innovative shirts ever created — a t-shirt that the US government classified as a weapon.

Oddly Necessary Background on US Encryption Export Policy

This t-shirt was created as a protest against the way the US government was treating the export of encryption (i.e., secret codes).  Until 2000, US government considered encryption as a munition that should not be exported from the United States or shown to a foreign national.

It seems a little odd that software that’s embedded in everyone’s iPhone today used to be illegal to export. Put in historical context it makes more sense. For centuries, encryption was used to allow military organizations to pass messages. The most famous of these devices was the German Enigma machine from World War II. The capturing of an Enigma machine allowed the allies to break the German codes and win the war.

The German Enigma Encryption Machine

In the 1990s, US government policy still hadn’t veered from this idea. Any secret codes that were used by foreign governments should be breakable by the US government without too much effort. At the same time, encryption was becoming a critical part of internet communications. The issue here was that the exact same technology that was powering the internet was also used to send secret government and criminal communications.  This led to internet browser companies like American Online to create two different browser versions. They distributed a “strong encryption” that was only available in the US and a “weak encryption” that could be exported everywhere else.

Some internet activists were upset about weak encryption. You see something similar in the fight today between the US government and Apple on the right to be able to break into criminals’ iPhones. The government was claiming that it was dangerous for people to have secrets that the government couldn’t see if they needed too. The protesters were saying weak encryption creates a weak internet.

Making The T-Shirt

This is where t-shirts make their appearance. Some encryption advocates had the idea to create a very small but strong encryption program whose entire code could be put on a t-shirt. Therefore, anyone wearing this t-shirt to a foreign country or even seen by a foreign national would be exporting a munition would be breaking a law.

That’s a creative t-shirt! I really wanted one. However, I ran into two problems. First, most of the t-shirts are pretty ugly. Secondly,  since the law changed in 1999, the demand for this t-shirt has plummetted and it’s no longer sold.

With no one making these t-shirts anymore, I needed to do it myself. First, I needed to find a design of the shirt that I really liked. Second, I needed to find a way to print it.

For the design, Vipul Ved Prakash made a wonderful version of the computer code in the shape of a dolphin.  The company ThinkGeek printed the shirt back in the 1990s.

So then I needed to make the t-shirt. Most t-shirt printers want you to buy in bulk but Design a Shirt works well for one offs.  Making the shirt is easy. I just uploaded the image and choose a shirt type. The benefit to making your own shirt is you can pay the $5 more for a super premium quality shirt. I also chose to pay another dollar to print the dolphin in blue. If you want to print the shirt, it’s still available on the Design a Shirt site.

Here’s the Final T-Shirt
And Here’s the Image I Used in the Shirt

I really love this t-shirt and proud I re-created it. I thought someone would recognize it when I wore it; however, even at technology events, I haven’t met anybody who recognized it. I even added the last paragraph to the shirt to explain it to people. I’m glad that I made it because the world needs more of these shirts — clever, interesting and pretty designs that tried to change the world. If you want to print one for yourself, it’s still available online.

Note: In the unlikely event you’d like to learn more, there’s a good Wikipedia article on US encryption policy that even mentions the t-shirt.