Book: LP #2: Ownership

Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say “that’s not my job.”

In January 2004, Amazon started its digital media business. Jeff didn’t have a specific product or business in mind. His first decision was to appoint an owner to the project. He promoted Steve Kessel to the job. Previously, he was Amazon’s head of books, video, and music. Steve became a “single-threaded leader,” an Amazonian term for a leader who owns a single major initiative and can relentlessly focus on it.

By making Steve a single-threaded leader, he was able to drive the business unencumbered by the rest of Amazon’s business. It also lets the rest of the book business thrive without worrying about eBooks. By splitting the responsibilities between today and the future, Amazon was able to focus on being the world’s best physical book e-commerce site while eBooks eventually eclipsed physical.

The idea of the single-threaded leader comes from Amazon’s technical architecture. In the early 2000s, Amazon.com ran as a single massive program named Obidos. It was named after a village on the fastest part of the Amazon river. Obidos was built to run fast. Amazon also had a single database that ran the entire company, an Oracle database called acb (short for amazon.com books). As Amazon got bigger, everyone wanted to make changes to Obidos and acb, creating two huge bottlenecks. The speed of innovation slowed down and developer frustration skyrocketed. Also, having one single application meant that a junior developer could crash the entire website (which happened more than once).

Building large monolithic systems is fairly common. Most businesses start with one big application because it’s the fastest, easiest, and most efficient system to develop. In the early days of enterprise computing, computers were giant, room-sized machines. They spoke an arcane language and ate specially formulated punchcards. Computing power was the most scarce resource in the company. A mistake in a punchcard could cause the business to waste thousands of dollars in lost processing time. In order to run these machines at peak efficiency, a cadre of high priests of computing grew up to tend to their every need. Much like ancient gods, these priests’ main goal is to make sure that the machines were kept happy with their daily supply of punchcards.

Over time, things changed. Computers got cheaper and these large expensive computers became smaller and ubiquitous, embedding themselves into our personal and work lives, but the software continued to be built at large applications in most places. As software got bigger, the speed of software creation started to slow down.

In theory, you should be able to just add more people to the project, but this doesn’t work. Fred Brooks wrote about this problem in his book, The Mythical Man-Month. Brooks wrote that while adding another person to the project gives you an additional amount of work, it also increases the complexity of the communication.⁽¹⁾Note that Amazon likes to keep these teams below 10 people, just as the communication lines start to get dense. See below for how communication scales as the number of people grow.

So how does Amazon Web Services manage to move so quickly. The platform has hundreds of thousands of updates a year, yet the platform is stable enough to get 99.999% (five 9’s) uptime, that’s 5 minutes of downtime a year, good enough to meet the needs of emergency services. It’s storage system has 11 9’s of durability which means that you could have a billion items for a century and likely wouldn’t lose any of them.

Microservices

So how did Amazon do this? They split out the different pieces of the system and separated them. The different pieces were focused only on their individual job. This is a bit counterintuitive because you’d think that things would get more simple and more efficient with more communication. However, as we saw above, it’s the communication that’s killing us. To fix this problem, you need to implement “loose coupling.”

The original Obidos system had too many cooks in the kitchen. Imagine that we’re starting a food service called Amazon meals. in order to be most efficient, we have one kitchen and we ship orders to the whole country. When we started up, we had a chef, maybe two chefs. But as the company gets bigger we might have a hundred chefs. They’re all trying to fiddle with the same ingredients. So when the new chef has a new idea for a Spanish tortilla soup he orders a different type of tomato. We now have fantastic tortilla soup, but it breaks the recipe for the chili. Well, we can’t have that happen! So you could implement a big process prioritizes all of these changes to ensure that no one breaks anyone else’s recipes. This process works but it’s very slow and frustrating for everyone in the company.

So we separate out all of these pieces. So you’d have a sauces department, a produce department, a bakery, etc. Each of these departments publishes their offerings as a set of offerings to the other groups (which in technical terms would be called an API). So if you want to make a tortilla soup, you need to use one of the available tomatoes from the produce department. If you want a different type of tomato you talk to the produce department and ask them to add something new to their offering. No one needs to know anything about the other departments other than what they offer. This means that any group can move fast and create new things without breaking anyone else’s recipes!

Each of these separate single-threaded groups is a microservice. In the early 2000s, Jeff mandated that Amazon move to this model internally, which became the basis for AWS. Jeff said something like:⁽²⁾There’s a famous internet post by Steve Yegge that summarizes Amazon’s strategy.

1. All teams will expose their data and functionality through interfaces.
2. Teams must communicate with each other through interfaces.
3. There will be no way to communicate with another team except through APIs. Teams may not read or write to each other’s data store, share memory models or use any other non-public communication.
4. How you build these interfaces doesn’t matter. It can be HTTP, Corba, Pubsub, or even a custom protocol.
5. These need to be built to be used by developers outside of Amazon. Though there was no plan to do it then, the software needed to be architected that way.

While microservices profile enormous benefits in flexibility and resistance, they also create a number of challenges:

• Every service now needs to be rock solid. This means that they all need to have the same security, resiliency, and uptime as the larger application. The failure of any component can affect the entire system so fault tolerance needs to be built into the system design.
• Every service has the ability to take down any other service and should be seen as a possible denial of service attack. For example, if I create a new service for employees to post thoughts about the company holiday party, I want everyone to sign up. I’d be using a firmwide account management service to manage my accounts. If everyone signs up at the same time, it could take down the account management service for the whole firm. This means that serious quotes and throttling should be in place for every service.

As a side note, smaller projects work better than larger projects everywhere. Bent Flyvbjerg has found that larger projects are more likely to have cost overruns. However, it’s not necessarily about the size of the project itself, it’s much more related to the size of the segment of the project. Public works projects like building a bridge or a dam, which can only be built in one large chunk, are more likely to have cost overruns than a road, which can be built in small segments.

Single-Threaded Thinking

Single-threading (focusing on one thing) works well for people as well. While we think we’re good at multi-tasking, scientists have found that performance is a lot better when we focus our attention. Without that, you’re always thinking about what came before and what could happen next. You’re never in the moment and you have a person who:

1. When he’s at work thinks about playing golf
2. When he’s playing golf thinks about being with his family
3. When he’s with his family thinks about work

It’s tempting to try to combine tasks. If doing one task is good, doing three should be better! But it’s important to give each task its own space. It makes me think about drinking tea. Some days I like iced tea because makes helps cool me down. Other days I like hot tea because it makes me feel warm inside and wakes me up in the morning. But compromise is not useful here. There’s never been a day when I woke up in the morning and said, “I’d really like a big cup of lukewarm coffee.”

It’s easy to mix things during the day. If you look at an 8 hour day as one chunk of time, things tend to merge. I’ve found that segmenting the day into chunks of time, it’s far easier to make progress. My favorite tool for this is the Pomodoro.⁽³⁾The New York Times also had a nice article on Pomodoros called This Time-Management Trick Changed My Whole Relationship With Time. The Pomodoro was created by an Italian who used a tomato-shaped timer.⁽⁴⁾Pomodoro is the Italian word for tomato. It’s simple but surprisingly effective. You split up a large chunk of time into 25 minutes of work with five minutes of rest in between. 25 minutes doesn’t seem that long and helps you stay focused on your work. The five-minute breaks give you a rest and help break up the monotony.

The best example of focus is from my friend Marc. Marc and I have been friends since summer camp and we meet up pretty regularly. The last few times we went to lunch, Marc sat down and he set a timer on his phone. He said to me, “I hope you don’t mind that I’m doing this but I want to really pay attention to you. I don’t want to have to check my watch and be worried about how much time I’m spending here. If I set a timer I can give you my full attention and then I don’t have to worry about checking my watch.” This free’s him up to focus on one thing, ME! And that makes both of us happy.

Questions/Exercises:

• How can you carve out your time to focus on one thing at a time?
• Try the Pomodoro method for a day, it’s a shockingly good way of overcoming procrastination.
• How can you define what you’re going to do and what you’re not going to do better.
• Can you take something you’re doing today and buy that service from someone else?