A leader’s job is to inspire their team. This is easy when things are going well. A good leader can keep morale up even when there’s bad news. But a great leader can find a way to use the bad news itself as inspiration.
I remember two years ago working at a financial services firm in an innovation group. John was the leader of the group. He started out the weekly team meeting by saying, “We got some news over the weekend. We redid our projections for the project launch and instead of the $50 million that Tony projected, we are only going to make $5 million in the first year.”
This is part of my Mother-in-Law’s Guide to Technology. My Mother-in-Law is a very smart woman even if she isn’t a “computer person.” The goal of this series is to take some big and treacherous sounding ideas and bring them down to earth.
Remember when you had kids and you told them to do stuff. And remember how they used to do what you told them but that wasn’t always what you intended them to do? Well, that’s the way computer programs work.
Just like kids, computer programs will do what you tell them, but beyond that, all bets are off. They don’t do anything that directly contradicts what you said but that doesn’t mean they’ll do what you want them to do.
That’s why you need test kids (or computer programs). Once you give them instructions, you need to make sure that these instructions are understood and that the kids will carry out the spirit of the instructions, not just the letter of them. You also need to make sure that the kids will know what to do in ambiguous situations and that the kids are positioned well to defend against any bad actors. You do this by testing the kids by watching them perform the task once or twice in a variety of circumstances. It’s much better to find out any problems or ambiguity in your instructions before the kids are sent out into the wild.
Let’s look at two kinds of kids that are representative of common software errors. There are many others, but these are two of the big ones. Testing allows you to find these errors early in the process, preventing these issues from causing big problems later on.
The Happy Path Kid
This kind of kid is naively optimistic. He also may not be very bright. When you tell him something he always says, “Of course. Sure. I can do that!” But he almost certainly doesn’t understand all the nuances of what you want him to do. If the instructions don’t exactly match the situation he’s in, trouble quickly ensues. For example, imagine this situation:
You yell at him, “That was so dangerous. I told you never to cross the street when the light is red. What were you thinking! You almost killed yourself!”
Only to hear him say back, “But there was no red light. In fact, there was no light at all. So I crossed the street.”
When thinking about the Happy Path kid, make sure you examine all of the possible ambiguous situations so that he (or the program) doesn’t run into any confusing situations.
Then there’s the kid who intentionally tries to skirt the rules. He’s far more clever than you are. He doesn’t break any rules but you’re continually surprised how he can turn a little imprecision into a HUGE opportunity for misadventure. He’s the kind of kid who always wins at Scrabble because he’s looking up all the words on his phone. You assumed he wouldn’t do that but you never SAID he couldn’t.
While the Happy Path Kid may encounter ambiguous situations and be led astray, the Adversary will look for weaknesses in your system and figure out how to exploit them.
Here are two stories from my time at Yale where I ran into real-life examples of these “problem children.”
The Happy Path Problem: Heads
This situation came up during my freshman year. It was a beautiful day and we were sitting outside in a seminar called “Perspectives on Science.” As we were sitting on the lawn under a tree, one of my classmates was performing a demonstration of how probability works. If you flip a coin a large number of times, half of the time will be heads and the other half tails.
“Can someone give me a quarter?” she asked the group. My friend Christine excitedly reached into her pocket and grabbed a quarter. The first person flipped the coin.
“Heads,” they said.
“Heads,” the second said.
By this time the quarter came back to the leader who examined the coin. “Who walks around with a two-headed quarter?!” she blurted out in surprise. As it turns out, Christine did. At the time, we were obsessed about Tom Stoppard’s Rosencrantz and Guildenstern are Dead which includes a scene where the laws of probability are broken with a coin that continuously lands on heads. So Christine got one.
The Adversary Problem: How to Cheat at Hangman
When I was at Yale, I took our most famous computer science class, CS223, with Professor Stanley Eisenstat. This is where I first learned about the adversary.
Professor Eisenstat wrote out a game of hangman on the board with 3 letters filled out:
We had 8 guesses to get this right. The class started shouting out different possibilities. We started very confidently with
This went on for quite a while as we gradually lost that confidence. Then Professor Eisenstat told us that there were many different words that this could be — far more than 8. Bill, Dill, Fill, Gill, Hill, Jill, Kill, Mill, Pill, Sill, Till, and Will. That’s 12 words if you’re counting. Here’s where the adversary comes in. Because Professor Eisenstat hadn’t committed to an answer beforehand, there was no way that we could win the game. When we chose a letter, he removed that word from the set of possible winners. He always had an option that we hadn’t chosen.
We are in an age when software is part of everything we do. We don’t have finance anymore but finance + computers. We don’t have cars anymore but cars + computers. With software being such an integral part of everything we do, it’s even more important to ensure that software does what we intend it to do.
Read in the voice of the Mission Impossible announcer: This t-shirt was originally created as a protest against US Export laws. Until 2000, US export law considered the computer code on the shirt as a “munition” that should not be exported from the United States or shown to a foreign national. Your mission, if you choose to accept it, is to re-create this shirt.
When I was at the GEL conference in 2016, I met a woman who worked for the website Design a Shirt. We started talking about the most creative t-shirts we’d ever seen. This brought me back to the late 1990s when I discovered one of the most innovative shirts ever created — a t-shirt that the US government classified as a weapon.
Oddly Necessary Background on US Encryption Export Policy
This t-shirt was created as a protest against the way the US government was treating the export of encryption (i.e., secret codes). Until 2000, US government considered encryption as a munition that should not be exported from the United States or shown to a foreign national.
It seems a little odd that software that’s embedded in everyone’s iPhone today used to be illegal to export. Put in historical context it makes more sense. For centuries, encryption was used to allow military organizations to pass messages. The most famous of these devices was the German Enigma machine from World War II. The capturing of an Enigma machine allowed the allies to break the German codes and win the war.
In the 1990s, US government policy still hadn’t veered from this idea. Any secret codes that were used by foreign governments should be breakable by the US government without too much effort. At the same time, encryption was becoming a critical part of internet communications. The issue here was that the exact same technology that was powering the internet was also used to send secret government and criminal communications. This led to internet browser companies like American Online to create two different browser versions. They distributed a “strong encryption” that was only available in the US and a “weak encryption” that could be exported everywhere else.
Some internet activists were upset about weak encryption. You see something similar in the fight today between the US government and Apple on the right to be able to break into criminals’ iPhones. The government was claiming that it was dangerous for people to have secrets that the government couldn’t see if they needed too. The protesters were saying weak encryption creates a weak internet.
Making The T-Shirt
This is where t-shirts make their appearance. Some encryption advocates had the idea to create a very small but strong encryption program whose entire code could be put on a t-shirt. Therefore, anyone wearing this t-shirt to a foreign country or even seen by a foreign national would be exporting a munition would be breaking a law.
That’s a creative t-shirt! I really wanted one. However, I ran into two problems. First, most of the t-shirts are pretty ugly. Secondly, since the law changed in 1999, the demand for this t-shirt has plummetted and it’s no longer sold.
With no one making these t-shirts anymore, I needed to do it myself. First, I needed to find a design of the shirt that I really liked. Second, I needed to find a way to print it.
So then I needed to make the t-shirt. Most t-shirt printers want you to buy in bulk but Design a Shirt works well for one offs. Making the shirt is easy. I just uploaded the image and choose a shirt type. The benefit to making your own shirt is you can pay the $5 more for a super premium quality shirt. I also chose to pay another dollar to print the dolphin in blue. If you want to print the shirt, it’s still available on the Design a Shirt site.
I really love this t-shirt and proud I re-created it. I thought someone would recognize it when I wore it; however, even at technology events, I haven’t met anybody who recognized it. I even added the last paragraph to the shirt to explain it to people. I’m glad that I made it because the world needs more of these shirts — clever, interesting and pretty designs that tried to change the world. If you want to print one for yourself, it’s still available online.
I’m a devoted husband and father to an awesome family who works at AIG as Head of Commercial Digital Product. For more information about what I do at work, please visit my LinkedIn profile.
What is a Digital Raconteur?
Throughout the 1920s, some friends would meet daily for lunch at the Algonquin Hotel in New York. They included the founding editor of the New Yorker Harold Ross, the playwright George S. Kaufmann and the writer Dorothy Parker. This group, called The Algonquin Round Table, would meet to tell stories and share quips in a bustling city that was finding its place on the world stage. They were the original raconteurs of New York, getting together to share stories that would enlighten and entertain. In an age when we no longer have two martini lunches, I wanted humbly bring that sensibility online.
I like building Alexa Skills. Skills are Amazon’s name of the apps that run inside Amazon’s Echo and other Alexa products. Building skills is a good way for me to understand how Alexa works and it’s a pretty neat party trick to get Alexa to pretend that she knows me.
Building skills used to be difficult. You needed to know how to program a voice user interface. Alexa soon came out with templates which made things easier but still required you to know how to program and so some basic system administration. I persisted in building a simple trivia game and Amazon sent me a T-Shirt for having published one of the first 1000 skills.
I remember thinking, “Where can Alexa be useful to me?” At the time, my boys were just starting to learn peoples birthdays. So I started building a skill for family birthdays. This was a great way to drill them on an important topic until they memorized it.
Alexa would ask, “Who’s birthday is on June 20th? One … Daddy … Two … Mommy … ”
And the kids would respond, “Daddy!” (Actually, they probably said “One” because the technology wasn’t very good at picking up names at the time.)
The problem with this skill is that you needed to include all the birthdays into the skill itself. So I could build a birthday family trivia skill but it would have the birthdays for my family. That’s not a particularly useful skill for other people.
My wife Abigail had a good idea for a skill as well. She said, “It would be really convenient to have a skill for the babysitter to know everything about the house — like bedtimes, WiFi passwords and emergency phone numbers.” But we couldn’t do this either because the program needed to have OUR information inside of it.
Amazon solved a lot of this problem last month with Alexa Blueprints. Alexa Blueprints are a simple and convenient way to create a customized skill that’s just for you. Instead of programming to create the skill, you just type into the web forms and Alexa does the rest.
Update (10/1/18): I’ve also created another couple of skills for the boys. I’ve used the Flashcards Blueprint to create a vocabulary quiz for my son’s fourth grade class. It’s a great way for him to come up with the definitions for us to put in. Also, we use Whose Turn to decide who’s turn it is to do different activities.
This is part of my “Mother-in-Law’s Guide to Technology.” My Mother-in-Law is a very smart woman even if she isn’t a “computer person.” The goal of this post is to take a very big and treacherous sounding idea and bring it down to earth. I tried this before in a post which I’ve now renamed The Mother-In-Law’s Guide to Chaos Engineering.
How many checkout lanes should Target build? At first, I thought about how many customers Target has on an average day and that they built that number of cash registers. If they have 300 customers in a day they would need enough cashiers to serve 300 people. The problem is that the flow of people into the store isn’t constant. For example, if the peak time of day is at 4PM and there are 10 people in line, Target can’t tell those people to come back at a less busy time. So on a daily basis, they need to plan for this by making sure they have enough checkout counters (and cashiers) available to keep the lines down to a reasonable level even when it gets busy. The way most retailers do this is to have only a few full-time dedicated as cashiers for the slow times and some other part-time cashiers that mainly do another job but can jump in when the store gets busy.
But that doesn’t answer the question of how many checkout lanes they need to build. Target needs to have enough checkout lanes so that even on the busiest days, they can hire enough part-time cashiers to keep lines relatively short. This means that Target needs to build the number of checkout lanes that they need for the busy time of the year, not for the peak time of day. At Target, this is the Christmas shopping season starting with Black Friday. On Black Friday the store is filled with shoppers struggling to check out. This is the day that Target opens up all their checkout lanes. So even though they’re not used a good portion of the year, Target still needs to build the number of checkout lanes they need for Black Friday.
So what does this have to do with cloud computing? Cloud computing is like Target having these checkout lanes only where they’re needed, like on Black Friday. They wouldn’t have to pay for the cost of having these checkout lanes at less busy times of the year. They would be able to create new ones during the Christmas season and get rid of them at other times of the year. How does this work? Instead of buying checkout lanes (or in the case of cloud computing, computers), they just rent what you need. This means that Target can increase or decrease their capacity based on the actual need from your customers.
Now let’s make the jump from Target to Cloud Computing by defining a few Let’s define a few things:
Servers: These are computers that “serve up” the information you need. Just like the cashiers at Target, if a server isn’t available you’re going to have to wait in line.
Server Capacity: This is the total number of servers that can be available to provide information. Just like the number of checkout lanes at Target, once you’re out of checkout lanes, you can’t have any more cashiers.
Peak Request times: This is your Black Friday time when you the most requests.
Now you can understand one of the key benefits of cloud computing:
Cloud computing provides flexible server capacity to meet demand during peak request times and release that capacity at during other periods.
So there you have it. In the real world, Target needs to build enough capacity (checkout lanes) to meet demand during peak request times (Black Friday). But the cloud computing model allows companies to greatly reduce their capacity during non-peak times because they can easily turn on or turn off this capacity.
Note: You can actually see a checkout model like this (sans the physical checkout lanes) at Apple stores. They can easily increase or decrease capacity because they don’t have any physical checkout aisles. This allows for flexibility by just adding or removing salespeople to the store with their mobile checkout devices.
I recently attended a session on Autonomous Cars at the law firm Herbert Smith Freehills. It was an insightful session where the lawyers gave great presentations on legal issues they advise on, like M&A, regulatory and product liability. However, one non-legal item they talked about was the ability for car manufacturers to “monetize data.” The idea of monetizing data comes up often but it’s a lot harder than it sounds.
A decade ago, I was working for a large credit card company looking at new growth opportunities. We were convinced that we could become the most valuable company in the world. Our reasoning went like this. Google was worth billions of dollars. But Google’s value was based on what web links people clicked. We, as a credit card company, had data on what people actually bought. Because our data was more relevant to advertisers than Google’s data, we should clearly have been worth more than Google.
There was just one problem. While we had this data, so did Bank of America, Capital One, JP Morgan and every other bank. And everyone was looking to monetize their data.
Did I say one problem? It wasn’t just financial services companies looking to out-Google Google. The phone companies were in this game too. They were saying, “Hey, we should be the most valuable companies in the world. Google has data on where people go on the web, but we have data on where people actually are in the real world.” Suffice it to say, there was a lot of data around.
This reminded me of an article written about undersea cable capacity in the days of the telegraph. Andy Kessler shared the following cautionary tale:
After undersea telegraph messages were first sent between Newfoundland and Ireland in 1886, a half-dozen companies sprang up to relay messages between London and Paris and New York. Half the traffic was for stock trading. These companies charged up to $5 per word and could transmit 15 to 17 words per minute. Each thought it could generate revenues of $5 million dollars or more per year. It was easy to raise the $2 million it took to lay undersea cable and investors, who constantly dashed off telegrams themselves, were all too happy to lend money.
Each of these companies assumed that they’d have a monopoly on the market. But when many companies entered the market based on that same assumption, all of the excess capacity created a race to the bottom for telegraph message pricing, forcing many of the companies into bankruptcy.
So what makes Google different? I remember a discussion with stock analysts around that time. I had written a paper on Mobile Payments along with Citi’s Equity Analysts. The topic of data was very hot and various analysts asked me, “Who’s going to win the data game? Who has the best data?” I explained that the real differentiator, and what people will pay for, isn’t the data itself but what you can do with the data.
As the famous Harvard Marketing professor, Theodore Levitt said, “People don’t want to buy a quarter-inch drill. They want a quarter-inch hole!” In the data space, this would be, “People don’t want to buy data, they want to buy results!”
How Google Uses Big Data
The goal of a search engine is to find the most relevant documents. In the early days of search engines, things were relatively easy. You could:
Examine Web Pages: Early search engines like Lycos and Altavista would look at web pages and determine which ones were the most relevant. They would do this by looking at factors like the number of times a word was repeated or whether the search term was in the title of the document.
Curated Directory: Yahoo, on the other hand, had humans hand-curating the web into a giant directory. This was relatively easy when the web only had a few thousand pages.
However, as the web grew, it became more and more difficult to manage search with these methods. Lycos and Altavista were overwhelmed. Not only was it difficult to distinguish between multiple similar pages based on the text in the page but there was also web spam that was trying to fool the search engines into promoting their pages. Yahoo had a problem hiring enough people to keep up with the quickly growing web. Both had doomed strategies.
Google went down a different path. By using an algorithm called PageRank (after Larry Page), formerly called BackRub (oh those Googlers and their funny names), Google was able to make use of data that everyone else was overlooking. The links between pages were just as valuable as the data in the pages themselves. For example, any page can claim to be the authoritative page of IBM. But if 100 people point to IBM.com as the right answer, it’s easy to lift that one to the top.
There are a few things to realize about Google’s use of data:
1. Google didn’t have the “best” data. Yahoo had a more accurate method for categorizing the web. Having humans look at content gave better results for each individual page. Unfortunately for Yahoo, that method was too slow and expensive to sustain.
2. The data didn’t cost Google anything. At the time, everyone was concentrating on the web pages themselves — not the linkages between the pages. This kind of information is often called “information exhaust” — information that’s a by-product of what you’re really looking for. It was already out there, free for anyone to use.
3. It’s the capability that made the difference. While the data was free, it was up to Google to organize the data and make it useful. Going back to the jobs to be done metaphor, Google put this data to work solving a problem for users.
4. More data is better. While other search engines were getting overwhelmed by the torrent of data from an explosion of web content, Google’s product actually benefited: The more links that can point to a quality web page, the better search results Google produces.
Google has been using this template for various other projects since they were founded. They can leverage data in some very creative and useful ways. Take location data for example. If you have an Android phone or Google Maps on your phone, Google is keeping track of your location data. You can take a look at your data here. The data is useful to me but it’s a bit odd seeing that Google holds a record of everywhere I’ve been.
So how can Google use your location, along with that of others, to create value? Well, one way is to aggregate this data to show where there’s road traffic. If you have a lot of phones not moving, then you can flag that road as congested. But where else could Google use this data? Google added a feature to Google Maps that let you see how crowded a restaurant was at different times of day based on how many cell phones they found at the restaurant.
It’s important to remember that Google did not have the best data to determine busy times at restaurants. Telephone companies and restaurant sites (e.g., Yelp, OpenTable) likely had better data. For example, OpenTable manages the reservations systems for many restaurants and actually knows how busy they are. But yet again, Google was the best at putting the data to work at solving this problem.
So let’s sum up. People still talk about monetizing data but their data isn’t as valuable as they think it is. There’s a lot of data out there that can solve problems and generate value. The tricky part is extracting the value from the data. Google did this in search and continues to do so in lots of other ways.
The Fitbit Alta HR is one of the few technology products that gives me almost everything I need and very little that I don’t. It’s a tour de force of good design. When I look at what I need on my wrist, it’s not really a smartwatch or even a fitness tracker but something else (maybe a “smart wristband”). Let’s take a look at the 3 features of the Fitbit Alta HR that are most important to me.
3 Features I Love About My Fitbit
EASY Sleep and Exercise Tracking. I need to track how much I sleep and how often I go to the gym. I used to have a Fitbit Flex, which while being a good product, made me manually track my exercise and sleep. For example, to track sleep I was supposed to tap on the band when I went to sleep and when I woke up. This meant that I forgot many days and didn’t have good data on my sleep patterns. The Fitbit Alta HR makes use of its heart rate tracker to guess at when I go to sleep (my heart rate drops by a lot) and when I’m exercising (heart rate goes up). It can even figure out the type of exercise I’m doing (i.e., bicycling, walking, sports).
Vibrating Notifications. For about 20 years, since I got my first flip phone, my friend Seth Gilbert and I talked about how difficult it is to make sure that we got our phone calls. We would put our phones on vibrate in our pocket and occasionally miss calls. Women had it worse because their phones were in their purses. With the Fitbit Alta HR, the wristband connects to my phone and will vibrate when there are calls or text messages. But notifications are limited to ONLY calls and text messages so I’m not bothered or even tempted by anything else (e.g., app notifications).
Tactile Alarm Clock. A wise man once said, “The hardest thing about being married is not having your own alarm clock.” Not really. I like to wake up earlier than my wife so I can relax and let my mind settle. But if I set off an alarm clock, then everyone wakes up. Having a vibrating alarm clock allows me to wake up with a vibration that’s much nicer than the buzzing of the alarm and wakes me up without waking up the rest of the house.
How I Might Improve These Features
As a product person, it’s fun for me to think about how to make these features even better.
Exercise Guidance (building on the feature EASY Sleep and Exercise Tracking). It would be great if my Fitbit could offer me sleep and exercise guidance in addition to tracking. Fitbit’s certainly going in this direction after having bought Fitstar and renaming it Fitbit Coach. Trying to put all of Fitbit Coach into something without a screen is difficult so I don’t think it’ll be done in the near term. However, I can see something simple like the Seven Minute Workout coming relatively soon.
I Need You NOW (building on the feature Vibrating Notifications). In an emergency, you really want to get someone’s attention. For example, parents want to know where their kids are but they’re not checking their phones. One way that I’ve seen parents get their kids attention is to set off the “Find My Phone” alarm on their kids iPhone. The “Find My Phone” alarm is normally used when the phone is missing in a large house so it puts out a piercing screech which cannot be ignored. One way of bringing this to the Fitbit would be priority vibrations. For example, if someone calls multiple times in a row, the vibrations would increase in intensity so that the wearer could not ignore them. If we wanted to bring this to the next level we could add a small shock to the wristband though I’m not sure how well that would sell 🙂
Wake Up and Relax (building on the feature Tactile Alarm Clock). The first thing I do when I wake up in the morning is turn on my meditation. It would be great if my Fitbit would wake me up and then go right into a 5-minute breathing exercise. The Fitbit Charge 2 HR already has a clever meditation routine for a small screen called breathing sessions. I imagine that my experience would go something like this:
The alarm rings
I wake up and tap it to tell the watch I’m awake
The guided breathing exercise starts. In … Out … In … Out
If I’m not breathing at the guided breathing rate for 30 seconds, the Fitbit assumes I’ve gone back to sleep and tries to wake me up again.
Reasons I Like My Fitbit More Than an Apple Watch
The obvious comparison here is to the Apple Watch. “Why not just get a smartwatch?” you might say. Then you’d only have to have one thing on your wrist. Here are the reasons that my Fitbit is better than an Apple Watch — at least for me.
It’s Not a Watch. I really like my watch. It’s beautiful and far nicer than an Apple watch. Wearing both watches looks a bit silly because hey, how many watches does a person need? It’s also a bit annoying because it has the feeling of “Go Apple or Go Home” —monopolizing my wrist. It’s one thing for my technology items in my life to be Apple. I mean I love my iPhone and I love my MacBook Air but I don’t need everything in my life to be Apple.
Long Battery Life. Because it’s not trying to do too much, the battery can last a week without charging. This lets me wear it to sleep which gives me half of the features that I want in my wristband (sleep tracking and alarm clock). The Apple watch needs to be charged every night so misses these features.
Small, Light and Fashionable. Because it doesn’t have these Apple Watch features (especially as it doesn’t need a full screen) the Fitbit Alta HR can be small, light and fashionable. I can wear it to the sleep, to the gym and even do vigorous exercise I don’t notice it.
Limited Notifications and Features. While there’s another feature or two that I’d love to have, I like the fact that I’m not distracted or even tempted by them. I’m sure if I had an Apple Watch I’d be tempted to read the newspaper or check my appointments on my watch — which I really don’t need to do.
Overall the Fitbit Alta HR is an awesome device. It gets me 90% of the things I need and little else. When I started writing this post I thought about how weird it was that I was talking about a fitness tracker that I don’t really use for fitness very much. Then I came across Wearable’s Top Fitness Tackers of 2017 and saw that they rated the Fitbit Alta HR as their top choice. One of the top features for the favorite “fitness tracker” was sleep tracking!
“Big Data” is a technology buzzword. The idea is that we have so much data about people and the way they interact with a company, we should be able to generate new and interesting insights from this data that will solve business problems.
But there’s a catch. Big Data is just another form of analytics. In most companies, each additional piece of data provides less value than the previous piece of data. In economics, this is called the diminishing marginal utility. So if the first piece of data (e.g., has this customer bought this product before) may be worth $1, the 50th piece of data (e.g., how old are the customer’s children) may worth less than a penny.
Unfortunately, many people are unduly optimistic about the value that big data can provide. They have this idea that they have so much data, if only they could search all this data, there’s bound to be something useful in there. It reminds me of one of Ronald Regan’s favorite jokes:
The joke concerns twin boys of five or six. Worried that the boys had developed extreme personalities – one was a total pessimist, the other a total optimist – their parents took them to a psychiatrist.
First the psychiatrist treated the pessimist. Trying to brighten his outlook, the psychiatrist took him to a room piled to the ceiling with brand-new toys. But instead of yelping with delight, the little boy burst into tears. “What’s the matter?” the psychiatrist asked, baffled. “Don’t you want to play with any of the toys?” “Yes,” the little boy bawled, “but if I did I’d only break them.”
Next the psychiatrist treated the optimist. Trying to dampen his outlook, the psychiatrist took him to a room piled to the ceiling with horse manure. But instead of wrinkling his nose in disgust, the optimist emitted just the yelp of delight the psychiatrist had been hoping to hear from his brother, the pessimist. Then he clambered to the top of the pile, dropped to his knees, and began gleefully digging out scoop after scoop with his bare hands. “What do you think you’re doing?” the psychiatrist asked, just as baffled by the optimist as he had been by the pessimist.
“With all this manure,” the little boy replied, beaming, “there must be a pony in here somewhere!”
So how should you use Big Data? Start with small actionable analytics − analytics that matter. That’s what I did when I joined a new group at a large company. Though the business unit had a chief data officer, our team didn’t have the analytics that we needed. So I implemented a simple plan for understanding the users, making sure we had good data and building iteratively over time.
Understand the Users
The first thing we needed to do was understand who was going to use the system. These users were going to define success for the project. We had two user groups:
Senior Management: This was the easy one to identify. Senior management wanted some basic information to run the business.
Power Users: However, we wanted to have a robust view of the data, not just high level reports. So we needed to look for our power users. And that’s when we found him. Let’s just call him Power User. He was based in London. Each month, Power User asked for a download of all the raw data and ran a 10 year old program in Excel to get the output he needed. The output looked great but the program was very fragile, hard to read and not extensible. But the output of this program gave us a great starting point.
Make Sure We Have Good Data
Analytics, first and foremost, is about the quality of the data. If you don’t have quality data none of your analytics will be right, whether it’s big data or anything else. When I started working on this project there was a report that showed the volume of payments being processed. Every few months there was a massive spike in volume. This wasn’t something hard to see. It was a spike of a quadrillion dollars going through the system. By comparison, all the value of everything in the world is only $241 trillion. This report was a couple of years old and people reviewed it each month but no one mentioned anything. As it turns out, testing data was being included which is what messed up the numbers. Cleaning all of the data and ensuring its quality took a lot of time but was an integral part of the project.
Build Iteratively Over Time
We spent a lot of time with our users understanding what they wanted. We did most of this on the cheap. Instead of building out real applications, we created PowerPoint and Excel mock-ups to understand their needs. We went through many iterations to pin down what they needed. The real test was with Power User. He was this really smart data geek and had been doing the same thing for 10 years. I remember holding my breath when I asked him for feedback. I was happily surprised with the reply, “Fantastic!!! This is really really good, guys, thank you so much!!!”
Don’t Over Build
Once we’d finished the first phase we’d expected to move our work out of Excel to a “real” business intelligence platform like Microstrategy or Cognos. However, we realized that the user group was so small and the flexibility they needed was so great that there was little value in moving off of Excel.
Big Data is just another form of analytics. If you don’t do analytics well, you’re unlikely to find something magical under your pile of Big Data. It’s best to start small and really understand your basic analytics by understanding your users, ensuring your data quality and building iteratively.
For decades the US military was trying to create self-driving cars with little success. Once the private sector got into the game, these cars improved at a breakneck speed. In 2004, when the first DARPA Grand Challenge took place, no car in the world was able to complete the 150 mile course through the Mojave desert. By 2016, self driving cars had driven over a million miles in regular traffic. The secret was not better computers or better cameras. The secret was better maps.
Peter Norvig, Google’s head of research, told the New York Times that Google Street View is the secret sauce behind Google’s self driving cars. He said:
It’s a hard problem for computer vision and artificial intelligence to pick a traffic light out of a scene and determine if it is red, yellow or green. But it is trivially easy to recognize the color of a traffic light that you already know is there.
I remember hearing that and thinking how convenient it was that Google happened to have Street View and that they could apply it to self driving cars. This would have been a classic case of “unlocking the power of data.” But then I learned the rest of the story.
Sebastian Thrun is the creator of Google’s self driving car and the founder of Google’s “X” lab. Google didn’t just “happen” to have Street View data lying around. Street View was created by Thrun after he met Larry Page at the DARPA Grand Challenge — the self driving car race. Thrun tells the story on CNBC’s The Brave Ones:
Larry came to the race itself and … came disguised with, like, a hat and sunglasses so he wouldn’t be bothered by everybody. But … he had a keen interest in this. Larry has been a believer in this technology for much longer than I even knew. And so was Sergey (Brin). And they really want to understand what’s going on,” Thrun said.
A later iteration of the car had cameras attached to its roof, so the team could review its progress each day, leading almost by accident to the development of Google Street View.
“We realized the video’s actually amazing. And we went to Google and said ‘we’d love to help you build Street View.’ And we kind of ended up – felt like an acquisition of a little start-up company, kind of Stanford transitioning into Google where me and four of my grad students then became Street View enthusiasts.”
“And we built up Street View and with a singular vision to photograph every street in the world.”
Street View became the first project within the secret Google X. “We had a separate building that no one knew about. At least for a year and a half, no one in Google had a clue we existed,” Thrun said.
So what did we learn? Data was the secret sauce for getting self driving cars to progress as well as they have. But it wasn’t a matter of finding a data set and applying it. It was about creating the data set for that specific purpose. Street View wasn’t a useful data set that was applied to self driving cars. It was the output of the mapping exercise that made self driving cars work so well.
One final addendum: When talking about Google Street View I have to add a link to an early version of Street View from 1979 that was created at MIT. The Aspen Movie Map (movie) used laserdiscs to simulate driving through the town of Aspen.