Overview
Transcript (Lightly Edited)
Below is a lightly-edited transcript of the video.
Introduction
- Facilitator
-
…an [SP] author. He works for the API Group. Is that right? Yeah. So, he’s a director of architecture at the API Group. And he’s written several books, maybe many of you have read them. There is a book, an O’Reilly book on microservices another book on RESTful web services. So, he’s an expert in REST and APIs and he has graciously come and decided to join us, so we’re really happy to have him. Thanks, Mike.
- Mike
-
All right. Thanks. Can you hear me okay? Cool. All right. It’s an honor to be here. I had a great time. I got here late yesterday and got a chance to mingle a bit and then spend time at dinner. This is an amazing group. I do a lot of talks. I travel a lot. It was 40 weeks last year. I’m cutting back. I’m hoping for 35 maybe, but this is a chance to speak to a group that I don’t always get.
So, I’m really, really excited and just slightly nervous. I’m also a little bit annoyed, dumb [SP], had a handful of my slides already. So, I had to take some out. So, hopefully, we’ve still got some interesting things to talk about. I wanna ask one favor, and then I’ll move on. Because I travel a lot, I forget a lot. And I need evidence that I was actually here. So, if you don’t mind, if you would just wave and say, "Hello." Thank you very much. Okay, that’s done. All right. I’ll look at that later and I’ll say, "Oh, yeah. That’s where it was."
- Facilitator
-
It’s out of focus.
- Mike
-
It’s out of focus. It’s always out of focus in my life. Okay. All right. So, I have this idea about a talk. This is a talk that I’ve never done before. This is something that I put together because I had this opportunity to speak to you. And I kept the idea of scaling in the title because this is what you’re really talking about, and futures are a big deal. So, there are a handful of things that I’d like to touch on. First of all, this is me, this is how you find me on GitHub and LinkedIn and Twitter. And I would love to connect with you. I’d love to learn what you’re working on, what you’re doing, what’s a challenge for you, and what you see in the future.
As Mike already mentioned, I work in a group called API Academy. It’s an amazing group of people I’m honored to be with. We traveled around the world and all do similar things. We talk to customers, we talk to people, we talk at public events, we talk to developers, and see a lot of people, all sorts and we gather that information together and share it in lots of ways. The Microservice Architecture book was the last way that we did that. And I’m gonna share some of those things with you today.
As I already mentioned, I have been stuck on a particular theme in my writing over the last several years, and that theme has to do a lot with the web. What’s it like to really build something? What’s it like to build something for somebody you’ve never met? What’s it like to build something for a machine that you’ve never met, that’s never met you, and how could they work together? How can we start to build these web of things? Now, we did… Everybody does a Microservice Architecture book. So, I was participating in one.
I’m also just agreed to do a developer book on just building and designing APIs. So, I’m very excited about that with pragmatic publishers. That will come out maybe later in the year. I’ll just mention, you can get a free copy of this microservice book. I brought some print copies that I’ll share with the team and they can figure out how they’re gonna get rid of them. These slides will be available, so you’ll be able to see it.
Scaling
Okay. Preliminary is out of the way. I wanna talk about the idea of scale, and I wanna talk about it in possibly a different way. What I’m hoping to do today is sort of provoke you and prod you into thinking about things a little bit differently, and taking that difference back to where you go. So, scale is a funny thing. We always have some sense of scale. It really blows my mind. I just learned this not too long ago. As of about the 1920s, astronomers were convinced that those things that they saw really far away were just merely dust, right?
They thought the universe was our galaxy up until the '20s. And it wasn’t until we had a little bit better sensors and a little bit more creative thinking that we realized, that’s not dust. Those are stars and that the universe was infinitely more vast than suspected. It’s amazing what a change of perspective can do. Often we think we’re building an app, or we’re maybe working on some functionality or some product. But instead, we’re actually contributing to a universe of things. And when you think about it that way, your team, your product, your company is just one star in the universe, and they’re all connected in some kind of way. It might change the way you think about scale.
I love that the word scale, the root of scale actually has to do with to climb over, to get beyond. Because often that’s really what scaling is. It’s our ability to get beyond where we are today. And I heard lots of talk yesterday about what got us here, won’t get us there, and old habits and so on and so forth. Often, getting beyond where we’re going means thinking very differently. And I wanna talk about a couple of ways that might make some sense.
Networks
So, as I mentioned, I travel a lot. This is my usual commuting vehicle. People get used to it, but I also traveled the network a lot. We all travel the network. The notion that Sydney is really just a few clicks away, the notion that you can get on a call that you can get on a video call, the notion that an application exists in all sorts of places to come together is actually pretty challenging. Your job is to program the network, not a machine. That’s the biggest thing, right? You program the network. And the rules for programming the network are totally different, totally different. They’re not like programming a single machine.
A single machine where I have clicks, and a clock, and transactions, and predictability, and miniscule time slices, that’s easy. But when I have to talk to lots of machines, machines that I don’t control, machines that are somewhere far away, all of a sudden, it’s a lot tougher because the network doesn’t behave the way we want. We know these kinds of lists, right? We know the righteousness of fallacies. I have this posted up above my desk because although I know them, I rarely remember to behave this way.
Almost every tool, almost every IDE, almost everything that I have lies to me about every one of these things. It makes me think that I’m working on a machine. When I’m really trying to work on a network. We still don’t have an IDE, we still don’t have developer tools that remind us every second that this thing could be across the globe, this could be turned off, this could be unavailable, this could be slow, could be unreliable. All those things. So, we have to keep filling in that blank every time. Someday we need the right tools. We don’t have them yet, but we need them someday.
I love this line from Pat Helland, "There is no simultaneity at a distance." Right? Time is an architectural element in what we build. I look at one of those stars, I see that’s beautiful, but I don’t realize these billions of years of die down [SP]. The light just died [SP]. Right? I read a little app, I say, "How many do you have in stock?" it tells me, "Five." I don’t realize that by the time the message got to me, there’s not five in stock. Thinking and designing and building where we realize that there is no simultaneity is an incredibly powerful way to make sure that you build great apps. And that isn’t just about code, it’s the same for teams, right? There’s no simultaneity.
Communities
There’s distance, and distance is a powerful element in the way we design code and the way we build. And that comes out in the notion of our communities, our connections, right? We’ve heard this guy before, right? Mel? "How do committees invent?" This is a paper he writes in 1967, and he confesses he didn’t do a good job of writing it, but nonetheless, he wrote it. By the way, this is what Mel looks like today. Yep, that’s right. That sucker is still around. He’s an incredibly creative person. He is so amused that something he wrote half a century ago is actually interesting to us, right? When I was interviewing him and talking to him, he said, "You know…" after we talked about this whole Conway’s Law stuff he said, "You know, I have done other things." He’s a former Apple fellow.
He developed some of the key medical software system called MUMPS. The reason that we have line by line compilers is because of him. The reason we have the notion of co-routines is because of him. He’s an incredible thinker. And today, he’s working on the notion of how to write applications differently. He has a sense that the way we write systems is entirely different than the way we write applications. Not only different in the sense of the actions but the code of them in all the gestures and everything. He’s an incredibly creative person.
Now, that paper is famous for that thing about the communication dictates design, right? But actually, there are several other laws, several other real good lessons in that paper that people often miss. We know this one really well. There’s always enough time to do it right. There’s never enough time to do it right, but there’s always enough time to do it over, right? That’s called version two. That’s called we’ll release it next week. We do this all the time. That’s not a bug. That’s a feature. That’s what we do. All right? That’s why continuous delivery is so important because we can do it again and again and again and again, right?
This one is just kind of a restatement of the same idea. There’s a homomorphism between the graph of the group and the graph of what you produce. They look the same. This is why it’s really important to think about teams. If I want four different components to speak to each other, four teams is a smart way to do it. One team is not a smart way to do it because there’s all sorts of hidden pieces that we get confused about, right? So, we need to pay real clear attention to what we’re doing. Also, if certain people have to make decisions, if certain people have to be involved, that becomes the boundary, and if you try to make that boundary disappear, if we pretend that two people in Sydney and three people in Gadang’s [SP] are a team, you’re in big trouble because there is no simultaneity at a distance, right?
So, you have to think carefully about that. And this one, the one I love so much, the one that we like to kind of ignore, large systems disintegrate qualitatively more than small systems. The bigger you are, the harder you fall. And that’s not just in, like, how much code you write, that’s not just in how big your team is, it’s in how big your vision is. You go all in. If you bet all in on something, the likelihood of you failing is astronomically high. Simple as that. We’re all gonna use X product, we’re all gonna use X framework, we’re all gonna start acting like this. Astronomically high. That’s not a bug. In nature, we don’t go all in. We make tiny, tiny changes. Systems stay relevant, stay alive, stay fundamentally stable as they adapt and change because they go step by step by step.
If you haven’t had a chance to read the paper, I encourage you to read it. It’s a little bit of historical fiction like because it’s a long time ago, but it’s still really, really relevant. And if you visit Mel’s site, you’ll find lots of other really interesting things there.
Individuals
I wanna talk just one more thing about individuals, right? We’ve heard of this Dunbar’s number, right? Dunbar’s number is this notion that I can only keep 150 friends in my head at one time, what he calls the casual friend list. It’s not actually the interesting number to me. The interesting number to me is in the bottom of the end, 5, 15. These are what calls the intimates and trusted groups. And these are the groups that make up our startups.
Nobody creates a startup with 150 people day one. All right? They create a start with a small group of people. Why? Because I trust everyone because I know what they’re capable of and because it’s easy for me to see when they’re not doing it, right, because we’re all contributing together, we’re all together. And at that lower end of the scale, I don’t have to explain much of what I’m talking about. We all know. Get up to 15. Okay, maybe we have to have a little touch point. We’re good? We’re good? Okay, fine. Get up to that 35, 40, 50 range, maybe we have to have a stand up in the beginning, and make sure we know. I might have to explain a few things. Get up around 150, somebody’s gonna write an email, right? You get past that, all of a sudden we hired HR, we got legal… All these crazy things, right? Again, that’s not a bug, that’s a feature, right?
So, knowing what’s going on at these different stages, at these different levels is incredibly powerful. Now, you know the company Gore-Tex Strength, W. L. Gore? He had this intuition in the '60s and '70s, he said, "When we build manufacturing plants for what we’re building, you get more than 150 people in a building, go build a building next door and start putting people there." He sensed even then, before Robin Dunbar had done his research, that there’s this sort of group element to this thing, and there’s a sort of a scope and size. Understanding how scope affects what you do is incredibly powerful.
So, again, this intimate, trusted, close and casual gives us a sense of what tribes can be like, what groups can be like. Sure, Spotify has a sense of this. They’ve created a great model for that. Everyone needs to figure out their own way of dealing with this reality, this truth element. Now, Dunbar’s research is a little bit controversial. Dunbar actually thinks that this is a function of brain size and the ability for language that actually language comes out of this, some people think it’s reversed. We don’t need to get too excited about that, but we see this time and time again when we visit companies. Some companies are incredibly explicit about understanding Dunbar’s, some stumble upon it through just common sense. But it seems to be something that’s inevitable. When you think about who your groups are, how you connect, it seems inevitable.
I want to mention just a couple other things from nature, which I find really fascinating. Max Kleiber’s law in 1930, he postulates this notion that metabolic rate, the burning of calories has a relationship to size of a creature. The size of this thing burns up so much energy, takes so much energy to keep started, to keep going. It takes a lot of energy to keep you guys going, a lot of effort, right? So, it’s really important to keep that in mind. And at some point, animals get so large that they spend most of their energy, most of their calories, just getting through the day, not really getting anything done, right?
There’s a point at which you spend so much time feeding the beast that the beast doesn’t get to do any work. I think many of us feel this, right? We have this notion that, "I used to be a really vital member of the team. I used to be able to contribute. Now all I’m doing is pushing papers, going to meetings, on phone calls, video calls." Again, that’s not a bug. Dunbar’s notion is that somewhere at around 150, 40% of the group’s time is in grooming, he calls it. And that’s emails, phone calls, coordination, all these other things. 40% of your time is suddenly taken up just feeding the beast. Down at the group of five, almost 0% of your time is feeding the beast. So, when we talk about effectiveness and efficiency, Kleiber’s law may be very interesting.
And finally, this is the one that I find really interesting. Allometry. It turns out in nature when something is just born, it may have a very different shape than when it finally grows up. In other words, scaling isn’t just this notion of being the same shape and doing the same things, but just doing it more. It turns out that as you scale, as you grow, proportions change. Sure, we’ve got HR and legal and central and all these other things because they’re necessary parts of making sure the system is healthy and effective and efficient.
So, it’s important to be aware of the possibility that at somewhere along the line, you may be asked, your team may be asked to change its shape. Your product may have to change shape in order to meet users' needs as the size of users grow. There’s all sorts of these things that we kind of know internally. And in fact, Nassim Taleb, who had done the book on antifragility, Nassim Taleb thinks allometry is a really important element of looking at where the fragile parts are of a system. And I’ve got a link at the bottom where he talks about this specifically, but then you can look in more and more about what Taleb talks about in the sense of how an organization change its shape as it scales.
Symptoms
Okay. So, we’ve talked about this idea of scaling. I wanna talk about this in a little bit, continue this idea of this sort of medical metaphor, this idea of symptoms and infections and long-term cures. And I wanna start with these symptoms. Hopefully, these all look familiar. So, I love the sort of medical definition of a symptom is. I apologize it’s a little bit graphic here. Blood is involved. But symptom is a subjective evidence. Symptoms are subjective, right, point of view, right? I find that really fascinating. What are some symptoms that we do with all the time? This stuff is too slow. It takes too long to get things done. It’s disorganized. It’s a mess. This is a common symptom.
Some of this is due to lack of attention, but some of this is actually due to built-in complexity, right? We have this lovely idea that we should be able to move anything, anywhere, anytime. We should be able to find anything anywhere. And, in fact, that’s not really true. There’s time involved, right? You think about how the web works and how the web system itself works, it’s taken incredibly long time, more than 20 years for it to get to where it is today. It’s not necessarily in a great place. And there is no single directory of everything. You’re looking for things that…it takes a hard time. That’s a feature. When somebody says, "I wanna know where everybody is at any one moment in time," of course, the answer is no, right? We know from physics, like, I can get really accurate about a certain moment in time, but then I can’t really predict anything. I can move back and see general patterns, but then I don’t know where any one thing is. That’s what we build. That’s our work. That’s what we do.
A lot of times, what we find are things that are stuck somewhere. You know what? We’ve got great functionality, but it’s buried somewhere. We’ve got great functionality somewhere. Didn’t we build one of those? I can’t understand why we don’t have that available. It’s really important to be aware of how we can mitigate that element. But noticing that things are slow is really important. Often we also have this idea that it’s too costly in time and resources and money too. It takes too much energy or too much effort. It goes back to what we talked about on scale.
Often, we fall into this trap, we have companies that fall on this trap where they’re just gonna go over to an empty field somewhere and build a new one. And that’s hardly ever a good idea. In fact, the better idea is to work with what you have. Often the companies that seem to do well at adapting and scaling over a long period of time are able to, as we say, build the airplane while flying. That’s a very different set of skills than going out and clearing a field and digging a new hole, right? Actually being able to incrementally change without breaking backward compatibility, without hurting people, without actually disrupting the daily business, that’s what we’re supposed to do. It’s a lot harder. It’s a different way of thinking.
And finally, we touched on this a little bit as well. Often we say things are too brittle or unreliable. I don’t know if I can change that. I don’t know if I can fix that. Often what we do is we create all these dangerous dependencies in things, so that we think we have to know about all of them before we change something. Does that make sense? I actually…I had a customer that spent all sorts of time. They did a fantastic job of building this great detailed dependencies tracking system. Every time they released something, it sort of automatically updated their dependency system, then they had a dependency manager, right, and then they had dependency supervisors there. Their whole life became dependencies.
And that’s entertaining, but, of course, the person who is managing dependencies and writing the code that handled the dependencies, and doing all the reports that talked about all the dependencies, was not contributing, right, because they’re just managing the beast. Often the best way to handle things like dependencies is to stop them. What’s the thing on networks? The most efficient network request is one that doesn’t involve the network, right, it’s called caching, right? There are all sorts of things we can do. The most effective deployment for a new version is no deployment at all, right? The new version simply appears that adapts, right? That’s what the web is supposed to be about, right? We don’t deploy a new website, we just change it.
So, thinking about things that are brittle and unreliable, often it’s not just because they’re disorganized, it’s because we want to reorganize them. There’s a sort of a Zen of, like, if I spend enough energy, I get this all organized, but you create entropy somewhere else. So, thinking about that can be really powerful. John Allspaw of Etsy has a great line, "You can lower the risk of change through both tools and culture," tools that allow you to make changes safely, tools that prevent you from doing harm, and culture that allows you to make changes without permission, right? So, where I can make some changes and I don’t have to coordinate with everyone else. We don’t hold hands and step off, right? We have this ability to move different parts of the system at different times.
The thing that I think is most interesting about symptoms is their subjective. Like, I just described this whole dependency thing. For some people, that’s not a symptom of anything, right? That’s subclinical. That’s not even noticeable. They think that’s the way it works. For other people, I talk about they may have a huge Q&A team of hundreds of people who manually test everything. That’s not a symptom of anything. That’s the way their system operates. So, it’s all subjective. What one team’s trouble, could be another team’s dependency, right? I’m depending on that other team doing all that detailed testing, I don’t need to do that, right? I’m depending on the build tools to make sure that I never put anything in that’s gonna break something, right? We become dependent on those and sometimes we forget.
Infections
Now, often what we see companies do is when they recognize these things that they call subjective symptoms, they wanna come up with a solution. And sometimes those solutions actually turn into what we call infections. Infections are things that exist inside the system that don’t normally exist there. They might not even be noticeable. You might be a carrier and you infect others, but it doesn’t affect you, all sorts of things. All these metaphors kind of makes sense. An invasion and multiplication not normally present may cause no symptoms. Nobody may notice. It seems fine to me. It may or may not spread. It may be localized. I don’t know what that team is doing over there. They are nuts, right? It may be localized.
There are all sorts of things that happen. People coming with all sorts of great ideas, all sorts of new things to do. Stupid people from outside come and tell you crazy things, and then you take them all back and you infect the rest of the company. I’m typhoid Mike, right? But you need to be aware of these kinds of things and the way these infections work in culture groups, the way these infections work in companies is memes, right? Memes are incredibly powerful. RESTful, it’s a good meme. Web-scale, event-driven.
All these things you can think of, you can just go over and over and over. There’s all sorts of memes. Memes are incredibly powerful for lots of reasons. They’re easy to self-replicate, they’re compelling, they are likable, they are particularly vague. Are you RESTful? Yes, I’m RESful. Are you RESTful? I’m totally RESTful. We know nothing, Right? They are adaptive, right? What was RESTful 10 years ago is not RESTful now, but we’re both RESTful, right? These things happen all the time. And sometimes they’re incredibly valuable, but other times they become your boat anchor, right? There are lots of memes. There are memes for libraries, memes for frameworks. Which way work are you using? "Oh, we’re all in on Vue.js now," or whatever the case may be.
There are memes for languages. There are people who track all these things. We’re gonna use this language or that one. That one’s not popular anymore. We shouldn’t use this one. There are memes for just all sorts of things. They are not bad, but they’re still memes, right? We still need to pay attention. We still need to figure out what we’re doing. We still need to make sure we’re aware of them. Occasionally, you may need inoculation from them. The real trouble is when memes become the cure, memes become the way that we’re fixing symptoms, right? Because in the long term, all of these memes keep changing all the time, right? One thing is not popular anymore, another thing is popular.
Canonical system control memes are really, really popular. Some have been around a long time. They can let [SP] a long time. They can sort of hide in the weeds from everything from ontologies to TOGAF to all these a canonical data, canonical object. They’re all memes. What is more powerful than a meme? Is your overall health your principles? One of the most powerful things that we see companies do is they define the ways in which we interact, the principles on which we measure, and that pretty much inoculates you from any meme that passes through.
I have a principle about customer first, about how we first do no harm, about some basic principles, about the way we operate. When somebody says, "Let’s use library X," I can say, "How does that help our principle? How does that help us in our mission?" And it’s easy. If it helps us, that’s fine. If it doesn’t, that’s fine. That’s okay. So, memes are very, very powerful. They can give us this sense that we’ve solved the problem, but often they’re self-serving, right? Often, I’m gonna construct a library, I’m gonna write about a style or something, and my job is to convince you. I don’t know anything about you. I don’t know anything about your company, your organization. Memes evolve over time.
The idea of what data-driven memes now means, it was totally different in 1980, right? That’s fine. But you need to be aware of what you’re talking about. And you could be talking to somebody that may think something else. They’re often very beneficial. They help solve a problem. But as I kind of alluded to it earlier, one of these things you learn in human counseling is what is a life raft today, becomes a boat anchor tomorrow. The thing that saved you may be the thing that kills you. So, being ready to give something up, being ready to say, "You know what? I’ve spent so long working on this library. I really love this, but I need to move on. This is not actually solving the problems I have today."
I remember talking… And this happens even in standards groups. I remember talking to Roy Fielding about some of the decisions in HTTP protocol, about timing and scheduling and caching and things like that that had to change radically from one version to the next. And I said to him, "What does that mean that in the beginning when you first started this, you really got these elements wrong, you didn’t really design this correctly?" And he said, "No, no, no. They were right in 1998. They were wrong for 2005, and we had to change in order to make the system functional." So, often we need to change. We need to move on. Things will outlive their usefulness. They often do. That’s okay. It’s time to get rid of my woobie, it’s time to get rid of that thing that I love so dearly. That’s okay. They may never go away.
The RESTful thing is not gonna go away like data [SP]. It’s always gonna be on. Somebody’s gonna come up with some story. It doesn’t really matter. We have lots of things that never went away that were helpful to us at one time, but can be rather dangerous, right? Tonsils, appendix, little toe, lizard brain, that’s a mess, but it’s better to leave them in, right? It’s too dangerous trying to take them out. Surgery, you die on the table. It’s bad. So, that’s another thing that’s really important for us. There may be parts of the system that you’re dealing with and it’s old.
I was working with, I can say it’s PayPal. I was working with PayPal and they have certain sort of C codes buried somewhere in the organization. Nobody remembers what it is that handles some critical payment thing. Everybody’s afraid to touch it. It’s their lizard brain. It’s like… I don’t know. They can’t… They’re not gonna get rid of it. It’s gonna be really, really hard. That’s engineering a new creature. That’s a Frankenstein right there. But you need to get used to the idea just leave it in. A lot of us who are software architects, we hate this notion, right?
Whenever we come up with a great idea we wanna actually clear out, we don’t wanna go Soviet. We wanna just…we erase the memory of that thing. It was not what I did 10 years ago when I was here. There’s no evidence of that at all. You shouldn’t be talking like that. I’m gonna erase him out of the pictures, right? No, it’s still there. And the new people come in and she says, "What the heck is this?" And you say, "Never mind, just live him alone." That’s not a bug. That’s a feature. When you deal in a complex ecosystem, there’s stuff like that all the time.
Long Term Cures
Okay. So, now I wanna talk a little about… Fine. Okay. So, we got this notion. I understand about symptoms and I understand about memes. What can we do long term? What are some of the patterns we’re seeing companies do? And long-term cures have a few basic features. I’ll sort of touch on some basic categories. But they have this notion of actually acknowledging the cyclical nature of things. I love that when we talk about cures, it really comes from the root word of care, of good health, of maintenance.
Sometimes it’s this notion of taking the cure. Ooh, that’s kind of painful, right, but we’re gonna go take a cure. But I love this idea of care and attention because that’s really what long-term fixes are about, care and attention. Now, I’m gonna include a couple of slides here for a couple of reasons. I wanna talk about this notion of cycles, and there are lots of examples of this. This is an example of mold and auto cycle from a German school and in architecture and thinking in the Bauhaus era about this idea of the effectors. People effect things that create inputs and outputs that we receive and we change. We live in this world of a cycle, right?
One of the reasons I included this diagram is this is Diego’s work. So, Diego and I have had the opportunity of work several times. We’ve known each other for four years, right? I’m making him part of the show. I’m breaking the law. These are some illustrations from the last book project that we worked on together, and I really loved the way he did this. But cycles, as a guide for the way we go about the world, is really important. Bill Verplank, when he starts working in this notion of user experience and user interaction, has this idea of feel, do, and no. I feel like something needs to change I do it. And what do I… I know something needs to change I do it, and what does it feel like? I get this cycle back and forth.
And often what we work on is very much a cycle in everything we do. We make a request, we wait for the response, and then we sit around until somebody makes another request. That’s a feature, right? We don’t often think of things in circles, we often think of them in lines. Bad idea. It’s always a circle. It’s always a circle. If anybody has ever built a game, an interactive game, you know exactly what I’m talking about. It’s always a circle and it’s always going. Complex systems. Healthy systems always operate in these circles. They always come around. We call them cycles, right, life cycles and things like that. Thinking about what you’re doing in terms of cycles is incredibly powerful because it means, just like continuous delivery, you’re gonna get another chance again. You’re gonna do it again, you’re gonna do it again, you’re gonna do it again. It’s not a straight line.
What was the thing? One-way doors, two-way doors, right? Life is not a one-way door. It’s incredibly rare. In our group, in the academy, we call it breaking the egg. It’s incredibly rare that you’re gonna break the egg, but you can’t put back together. Now, when you think you are, stand around, talk, it’s a good idea, but it’s hardly ever true. Everything we do is this big cycle. And designing like a cycle, thinking like a cycle, and acting like a cycle is incredibly powerful. I learned this from a great mentor years and years ago. It wasn’t even in IT.
I had this great idea for how we were gonna change our division and do these things, and my superior, my mentor, he said to me, "You know, Mike, I can tell you’re really passionate about this, but I gotta tell you, I admire your optimism, I don’t share it. But I don’t think I can convince you otherwise, so go ahead." Wow, that’s amazing, right? So, he understood that what goes around comes around. He understood he had a sense of the health of the system he was working with, and he understood… He trusted me that if I really started doing something stupid, I’d be stopped one way or the other.
A lot of times we have to say, "Go ahead," because it’ll come back. That’s fine. It’s tough for us to understand that it’s pretty hard to crash the entire thing. Now, there are other things that we can adapt and we can do and we can keep as a long-term pattern. And I suspect from what I’ve talked to people today that you have this idea already. One of the notions is incremental and invalidateable, right? So, small steps every single day, that’s how you change a system, not in big steps every once in a while. It’s in the tiniest, tiniest little things. And it’s every single change that accumulates over time is how a system actually changes over time. It’s incredibly rare that there’s this watershed moment.
We’re really romantic about those and we always tell those stories, but it’s really, really rare. The one I like a lot is this notion of how radical it is that Apple decided there would be no buttons on phones, like, that was mind-blowing for most of us. In fact, it was just a tiny incremental logical next step for the design teams that have been working on this in Apple. We just don’t share the same space, right? For them, it was a simple thing. Often a team will do something that seems amazing. It’s very logical to them. In fact, they are sometimes even embarrassed about saying so.
Well, I don’t know, it just seemed like a good idea, right, But for us, we think it’s amazing. Invalidateables is really important as well. Building in… I’ve heard lots of talk today about observability and making sure that you can see what’s going on. What you wanna do is get feedback as soon as possible. The sooner you get the feedback, the more you can make the incremental changes because you don’t have to wait, right? Feedback on something like the American economic system, you gotta wait, right? It’s gonna take years before we’re sure, right? But feedback and changing this one little line of color, this bug, or this layout, you can get feedback right away. You can build a system that gives you feedback right away. And when you get that feedback right away, it’s validateable, you can decide whether or not it should be sent out to more people. You can decide what to do next. So, a customer of ours, big integrator. The next big thing is incremental change.
Another big one is thinking in ecosystems. And I talked earlier about the idea that we’re in the business of actually programming networks, not machines. We’re actually in the business of programming using other people’s code, other people’s services, other people’s machines, other people’s stuff. We have to have some kind of agreed level. But I can’t agree on the database, I can’t agree on a programming language, I can’t agree on a framework, I can’t agree on hardware. That’s crazy. That’s a monoculture, right? What we can agree on is how we talk to each other. And that’s all we can agree on.
Shared understanding is what we need to start and that is the way we communicate that goes back to Conway and all these other things. Designing systems where you can use whatever tools you want, as long as we both understand each other in the end is incredibly powerful, is a key element to Amazon this notion that everything is a black box in Amazon. They don’t really care what happens inside, what they care is what comes out. What comes out has to look the same and be understandable right away. When you can adapt this pattern where you standardize on things like vocabularies and protocols in formats, suddenly, a piece that I built today works with something that was running six months ago. And a piece that somebody wants to build tomorrow will still work with the thing that I built today. Excuse me. So, ecosystem thinking is very important.
Interoperability is more important than integration. Interoperability is how the web works. We don’t control all those things on the web, we just connect them. We connect them the same way we connect all the other ones with links and forums and blog, that’s all, with nothing else. And yet it’s incredibly powerful, and we can build a massive network, and we can connect to people we’ve never met because we all share some basic understanding. In the end, what we’re really trying to do is this challenge of speed and scale. I think I heard some of this yesterday. We wanna go faster, but we can’t be less safe in doing so. Most of us were doing this notion of harmonizing speed and safety at scale, right?
And the way you do that is not just by the guardrails, but also just by the way that we see companies arrange their measure of success, right? So, they have this notion of whether or not this is gonna be less safe, this is going to… I can’t jeopardize somebody’s HIPAA regulations or somebody privacy, right? That’s okay to take that a little bit slower. But at the same time, if I can speed something up, while I have the guardrails in place, it’s incredibly powerful.
So…
Okay. So, there’s a few other things I just wanna wrap up with.
So, the thing is, where are we going? What is the possibility that you have in front of you? You work in tools that touch millions of people’s lives, right, and you’ve got some of the smartest people here in the room. When you think about these symptoms and when you see things are too costly or hard to deal with or about to get too complex too fast. When you think that somebody’s gonna introduce some new thing, are they just starting with a meme? Or are they just saying, "We’re just gonna all do Kubernetes now today just because," or is it really gonna match? Is it really gonna change our principles?
When you think about long-term challenges, when you think about incrementalism when you think about making sure we have shared understanding from the start, when you think about the difference between, "Am I just running because I need to run or am I actually making a challenge?" These things all talk about scale.
When you scale you’re gonna have to deal with committees, you’re gonna have to deal with how people work. Conway’s laws are true today as they were half a century ago. People are people, groups are groups. Make that your secret weapon. No one in the group size changed when the group’s dynamic is changing, and be prepared to start changing the way you interact, be prepared to start doing more of the things that this larger group, or this smaller group needs.
The last thing you need to do is if you take the same team, the same operational elements when you had 150 people and try to operate them at 15 people, well, we’re gonna need an HR person, we’re gonna need a person in legal, I have, like, a 5-person team know. I don’t need those anymore. And think about how scaling creates new points of fragility. There’s some limits to size, there’s a limit to the size of the creature, there’s some limits to connections. These are all powerful elements that give you ways to decide what you’re gonna do next.
So, I had this great quote at the end. But Dom used the quote, so I can’t use that. So, I’m gonna use another one that I…one that I really love. It’s a real geeky one, but hopefully, it rings true. This is from Paul Clements from Carnegie Mellon. Paul was very instrumental in the '80s and '90s in helping create software architecture. And I love what he says. "The best software architecture knows what changes often and makes that easy." Making it easy is a phrase that we use in our group. Making it safe, cheap and easy to change the things that are important that need to change, that change often. That’s powerful.
So, when you think about what’s too costly, what’s too brittle, what’s unreliable, often it’s because we haven’t designed a system that makes what’s a really important easy to change. So, that can be a great measure. If this is hard, maybe we’re not doing this right. Maybe there’s something else we can do. So, hopefully, that gives you some ideas. I would love to hear some feedback. I want to thank you very, very much for the time and, hopefully, I’ll get a chance to talk with you later on today. Thank you very, very much. Thanks.