Hear this conversation between Joe Beda, Co-creator of Kubernetes and Principal Engineer, VMware and Paul Puckett, Director of the Enterprise Cloud Management Office (ECMO), U.S. Army where they discuss:
Tune into our monthly podcast series, Mission First, where we will be focusing on the Department of Defense and National Security mission and not products. Hear from thought-leaders in the industry as they discuss complex challenges and topics buzzing within federal government IT.
Craig Bowman: Hey everyone, and welcome to mission first. I'm Craig Bowman, your host. This is the podcast that talks all about mission, not about products. That is our first episode and I'm excited to have Paul Puckett. Now he's the head of the enterprise cloud management agency under the army. And Paul is going to be having a discussion, an interview back and forth with Joe Beda, who was one of the co-founders of Kubernetes. I don't like to waste any time. So, I'm just going to pass it right over to you, Paul.
Paul Puckett: Hey, there, and welcome to Michigan first. My name is Paul Puckett. And I'm the director of the enterprise cloud management agency with the United States Army. And I've got the pleasure of getting to spend some time with Mr. Joe Beda. Some of you may know him as essentially a Principal Engineer at VMware, but really just one of the three co-founders around Kubernetes. So, Joe, welcome to Mission First.
Joe Beda: Thanks for having me on. And I'm really looking forward to our conversation today.
Paul Puckett: Yeah, absolutely. It seems like wherever we go, you just seem to be a plank owner, one of the first with Kubernetes. And now with mission first, so we're just adding to the list. Alright, so we're gonna be chatting about obviously, you know, understanding technology a little bit. But I'd really like to start to unpack in your brain, just kind of the little bit of history around, you know, Kubernetes, and kind of what problem you hope to solve, but really kind of the, the world we have today, and also the kind of where we see ourselves go. And I think there's a lot to unpack in there around people in process. So, if you're ready, we can go ahead and dive in. Sounds great. So, when it comes to Kubernetes, you know, born as project seven of nine, right? You know, when that all of a sudden hits number nine, when it comes to commits on GitHub, that what is going through your mind for what started as you know, just some internal project over at Google?
Joe Beda: Well, you know, when you start these things, and I think this is part of the fun of being in the technology industry is that, you know, ideas that could be huge are a dime a dozen. And so, you always are dreaming big of like, hey, if this happens, and this happens, and it all works out, it could be huge. And, you know, sometimes it actually does happen. And I think that was the case with Kubernetes, a lot of the stars aligned in terms of people being ready for it, and bringing in the right, you know, partners and building the right community. And it's, it's been an amazing journey. But there's definitely a lot of luck and a lot of serendipity that goes into something like Kubernetes emerging.
Paul Puckett: So, I don't think many of us or any of us can really predict kind of the future. But when it came to the problem that you're trying to solve, right there within the board team, could you ever imagine that it would become the conversation that we have today?
Joe Beda: Well, yeah, I think that's the idea is that, you know, our thinking back at the time that I wasn't on the board team, I was a board user at that time, I had built out Google Compute Engine, and I was I was the founder for that. And a lot of our thinking there was the way that Google build software, which we understood there were a lot of advantages for was very different than the way that people manage software outside of Google and Kubernetes was really about trying to sort of merge those worlds and take the best of both of those things, with the idea that, hey, this is a way that that, you know, Google at the time could essentially, you know, shake the snow globe, reset the market a little bit and create a new way for people to manage applications, manage infrastructure. And, and in doing so create a lot of opportunity. But you know, the best way to create opportunity, and I think the only way to create opportunities by creating value, people will change when it's more valuable to do so. So, a lot of our thinking there at the time was, there's got to be a better way to do this, versus dealing with the nitty gritty that a lot of folks do when they're dealing with raw VMs. And also provide the flexibility necessary to adapt to all the situations and applications that you need to find yourself running.
Paul Puckett: So when it comes to delivering that initial value, like who did you first imagine as, as your customer, like, right now we see this kind of a weird line being drawn between kind of operators and developers helped me understand kind of at its inception, like first, you know, who you're trying to deliver value for? And then obviously, some of your thoughts as to what's going to come?
Joe Beda: Yeah, so I think, you know, we throw around a lot of terms in this industry like DevOps, DevSecOps, cloud native, and these things are really, really confusing. For me, though, and I think this is my sort of definition of cloud native, is it you know, and I think this is this is, you know, as we created the Cloud Native Computing Foundation, and all of that, it's really about embracing the idea of higher level services exposed to you as a platform, having those building blocks that are super useful that you can put together in new and interesting ways. And so that idea of platform that idea of Having one party establishing a bet a set of higher level more and more useful tools. And then having the application teams really take the fullest advantage of that, to me, that platform level of thinking and raising the level of the platform over time, is something that is cloud native. And I think that was definitely in mind with us. In our minds, when we were creating Kubernetes, based on the lessons with bore because we saw like, hey, board provides a lot of higher level tools inside of Google, can we do something similar, but really take that toolbox approach that really has made infrastructure clouds, so successful, so bringing the infrastructure cloud toolkit, but moving it up the stack a little bit and making it more consumable for users. And so that, that I think, you know, we weren't as focused early on, I think it shows for good and bad with Kubernetes, on making sure that the end user experience was super polished for a certain path and a certain type of application. Instead, we really wanted to focus making sure that we had the right foundations, the right bones, so that people could do interesting things with Kubernetes. And with the cloud native platform that ended up evolving from that.
Paul Puckett: It's starting to make sense, it's wild to think you kind of had that vision of that value on the application side, but that, that kind of middle ground between, you know, what is typically like your application provider and your infrastructure provider, you know, the advent of cloud, and all of a sudden, you know, we're delivering self-service infrastructure to software developers, and there's, there's definitely immense amount of toil to make up in that ground. So, I'm hearing is how do we start to fill that gap, you know, with a platform capability like Kubernetes. But you've said in the past, that Kubernetes is a platform for creating platforms. So, help me understand kind of the extensibility that exists there, but also ways that you actually truly bridge that gap to developers that are trying to deliver value?
Joe Beda: Well, a lot of what informed Kubernetes was our experience at Google around things like App Engine, looking at Heroku, looking at things like Cloud Foundry, sort of the, the, you know, the earlier generation of platform as a service. And looking at that, and we saw this with App Engine inside of Google is that you would often see, people see a ton of success early on, until they hit some sort of wall until they hit some sort of limit. And, and it's awesome until it isn't. And then the options that you have in terms of what you do is like, well, maybe like, here's a VM and good luck in terms of managing that and connecting it in a secure way and creating something that bridges the platform and the VM. And so oftentimes, the solution to this is, well, we just need to add more features to the platform, just add more features like, oh, this problem, this problem will just sort of like, you know, solve this. And our experience watching that from the outside was really one of you're never going to add all the features that everybody needs there. So is there a way that you can, instead of trying to solve the whole problem, you can actually make a lot of progress, solve a lot of the building blocks, even if you're not solving everything for everybody, you're at least making things better, you're at least raising the abstraction over time. Now, as we did this with Kubernetes, what we found is that, you know, we amazing community, amazing set of folks coming together, that's both a blessing and a curse. And the curse was that we found ourselves in a role of being a gatekeeper, where everybody wanted to add features to Kubernetes, we wanted to make sure that Kubernetes was going to accept features that were going to add value over the long term and weren't going to become a burden to the project. And we didn't want to find ourselves in the position of judging what's valuable, what's not valuable, and really restricting the things that people were building on to where we're using Kubernetes for. And so, the answer to that was really one of instead of building more and more features into Kubernetes, how do we actually build the Extensibility Mechanisms so that people can build those things on top of Kubernetes. And this is where you start to see some of the comparisons to like, say, the Linux kernel, what happens inside of a kernel is relatively scoped, and people are very, very careful about adding new features to the kernel. But there's all sorts of extensibility models, whether it be drivers or user mode that are built on top of that, and people are much less concerned you can do absolutely, you know, nonsensical things in user mode in the kernel, people don't care because they're like, okay, you know, that's your problem. This is my problem. Creating that platform and extensibility mechanisms is, is, I think, one of the keys that made Kubernetes so successful. And what we see is that people use a lot of those services, those Extensibility Mechanisms to create more and more specialized experiences that really start to fit their use case, whether it be a traditional serving application, that starts to look a little bit like we're still figuring this out, like the traditional passes, or whether it be managing machine learning or whether it be big data type applications, whether it be stateful services, all of these things can be adapted to run on top of Kubernetes by using those Extensibility Mechanisms and building domain specific frameworks on top of that core.
Paul Puckett: So then let's get back to that line and where we derive value, right, so with all of that extensibility is that part of like how have like the crutch, if you will of Kubernetes. Because, you know, if teams are pushing all of that extensibility and optionality to developers like Wouldn't that become this like toil that's kind of accrued over time. So, like, help me understand, you know, if, if I'm looking at like a team that's trying to leverage Kubernetes is valued for their mission, where's the right place for them to start hold opinions, or as you know, enable that extensibility. And those that optionality for developers.
Joe Beda: So, this is, I mean, this is the core of it. And I think we see this with something like, like Linux also, where, with so much flexibility comes a lot of power, and comes a lot of complexity. And so, I think one of the keys for Kubernetes being successful is it's so adaptable to so many situations. But that's one of the things that makes it hard and makes it be, you know, intimidating for folks, I think folks look at, if you're not aware that Cloud Native Computing Foundation has this landscape that tries to map all the companies and projects in this space. And it's absolutely intimidating. I mean, there's just way too much there. And it's more than a full-time job to try and track everything that's going on in the community, all the projects, all the extensions, all the things built on top of it. You can blessing and curse, right, that's the sign of innovation. That's the sign of a lot of action. But from a user's point of view, from a consumers point of view, it's just it's, it's just too much. And so I think what we see is the emergence of both projects and vendors that are helping to provide a more curated view on top of this to make users more successful, whether it be projects, there's open source projects that really look to narrow this down, take opinions in terms of Well, here's the pieces that building on top of Kubernetes can deliver this type of result, sometimes these things are the whole meal deal. Sometimes they're more focused like a framework for doing machine learning, for example. And then you have vendors that will provide a more curated view of Kubernetes, and then go ahead and support that. That's one of the things that you know, that VMware does, but I think there's you know, that's one of the things that cloud providers are doing. But then there's opportunities for organizations to invest in taking the tool sets, they have either starting from an open source thing starting from a vendor solution, and then customizing that to their unique needs. And I think this is one of my learnings, as I've moved from Google, in terms of doing cloud and then working with closely with enterprises. And folks like the military, here at VMware is that, you know, there, there are patterns that you see repeating, but every organization is kind of its own little Galapagos Island that sort of evolved in its own way. And so, it's incredibly difficult to build a solution that's going to work for everybody. And so that extensibility, that sort of, you know, was that outlet for letting Kubernetes evolve, can be used, then to actually create something that can adapt to the unique needs of any particular institution.
Paul Puckett: Alright, so let's start to unpack that, because I, you know, I've lived kind of this journey of the value of containers and delivering outcomes and trying to drive automation and orchestration for delivering platforms for developers. So, I think it's just extremely hard for leaders that are kind of starting those journeys to, to understand what decisions to make, like given the extensibility, right, like, what decisions do I start to make, you know, who am I trying to support? Like, what are the metrics that like someone's starting the journey and seeing that there's perhaps value in leveraging Kubernetes? What metrics should they look to and align to, to help them kind of guide them on the journey as they're starting to make those decisions?
Joe Beda: Well, I think the first thing is recognizing that, you know, Kubernetes is a tool, it's not the solution in and of itself. And so, I think you're absolutely right, in terms of, you know, you want to be focused. And again, you know, the whole point of this podcast is you want to be focused on outcomes, you want to be focused on making sure that you enable teams to be successful. And if Kubernetes can help you deliver that outcome, then then that's what it's there for. But using a, you know, a tool, just because you have a tool is never really a good, good policy. And so, with that in mind, I think, you know, what I usually tell folks is, start simple. I think a lot of times people you know, they, I'll tell you story, like so like when I was I'm dating myself here, I had a friend in high school that got a Mac back in the day when they were new. And he sent me this letter, a snail mail letter, and it used every single font that was available on his Mac. And so, there's this tendency, you know, and it's like, now we know like, hey, that's a poor taste, right? There's a taste aspect to being able to choose the right type faces for a document, same thing, that there's a taste aspect in terms of being able to pick what are the right tools and extensibility and systems that you need to build to be able to accomplish your mission. And it takes time to build up that that intuition that idea of appropriateness, both within an organization and across our industry. So, we're still in the sort of use every font in the letter stage and a lot of places where folks start with Kubernetes and they pile on every technology because they're like a kid in the candy store, and so on. I advise people start small, start simple work with a team that is actually willing to take on some pain and learn, understand what success looks like in your organization. Because not everybody has the same mission. Not everybody has the same set of talent that they're applying to that. And then once you understand that, then you can start paving the cow paths, you can start understanding what does my organization need? Let me actually start figuring out what does tasteful look like for me?
Paul Puckett: You know, kind of what works for my mission, I think that there's, I think that there's a pivot that's happening here. Because as we start to expose kind of IT organizations who would typically be in the business of providing these platforms, we're seeing that they're starting to really realize the mission that they're in, right? You know, in the DoD, we often talk about ends Ways and Means, right, and Kubernetes being a means to an end, right? There's this kind of utilitarian approach of like, whatever gets us to the outcome. And I've seen a lot of people measure the outcome of the IT technology, and not necessarily what right what it enables. So, I think, you know, what works for the mission, I think, is also poking at it really kind of people in process here, that starting to pull people a little bit closer into realizing what their mission is. So, with that being said, can you help me unpacked kind of the application now of Kubernetes as a means into this, you know, this buzzword of DevOps and DevSecOps, can you help me kind of, like, tease it out as its roll, if you will, within that entire movement.
Joe Beda: Well, so. you know, when I look at sort of, you know, and I, you know, I mentioned cloud native before, but the being this idea of platform, this idea of providing a tool set that enables application teams to move faster, and very careful when I say that, because I say application teams, I don't say developers, and I think part of DevOps is recognizing that an application team is made up a bunch of different roles. And those you know, and traditionally, we think about, you know, ops and devs, as being completely separate functions with thrown over the wall with, you know, conflict and lack of communication. I think, you know, the core of DevOps is really, developers have to care about how their stuff runs in production. And anybody who's monitoring what's happening in production has to feedback learnings into the development to make things more operational over time. And you have to create those feedback loops between those two roles. Now, sometimes, those things get smashed together, and you have one team that's actually taking on both concerns. But you can still have a healthy relationship between ops and devs as two different things, if there's that, you know, that feedback loop and that communication that's going on there. And so, it really, these are two different dimensions. One of these is about providing a more and more capable toolkit over time, which Kubernetes can be part of that toolkit. And then the other thing is, how do you use that toolkit to make sure that you create the right alignment of motivations, so that operations and, and development are pulling in the same direction. And so, what we find is that there ends up being a lot of feedback loop between development and operations at the application level. But then you also need to make sure that the application teams and the platform teams continue to challenge each other and actually make things better when I see this work. Well, what you find is that the platform team, you know, again, this can be internal, this could be a mix of open source vendor, but whoever's providing that platform, listens to what the application teams need, and then provides more and more tools into the platform. And over time, the platform gets more and more valuable as the application teams learn how to build on top of it. And so, there's really sort of this, this sort of, you know, two-dimensional relationship, there's Dev and Ops, and then there's also app team and platform team. And I think Kubernetes is a great tool set for building that ever evolving and never more useful platform that you're providing to the application teams.
Paul Puckett: I so I so there's a lot to unpack here from like a federal government perspective. So, like this notion of just say, like an application team and a platform team is relatively foreign. In the DoD, you know, typically we follow a Waterfall, you know, system engineering methodology, capture all the requirements, you know, go by, or go build the thing and feel the thing. So, this notion of feedback loops is really different. And I believe that we're not really organized to facilitate these feedback loops. So, understand these organizations that are structured where, you know, it's an operations. That's the capability that you have, as we start to like create the need for these feedback loops in this realization of like delivering greater value. What do you think from like an organization structure perspective, what needs to shift? Like this is I think this is a x component of this culture change that we talked about, but how we organize ourselves. Can you dive into a little bit of kind of what application teams and product teams like leadership looks Light that enables this these kinds of environments.
Joe Beda: You know, for me, it's, it's a matter of trust and autonomy, and recognizing that the people that you're working with, and I think this is, I mean, we use terminology all the time. And it brings with it connotations that we may not intend. So, like, what you'll hear a lot of times, it's things like Software Factory, but a factory implies that you have essentially workers on an assembly line that are that are, you know, replaceable, that are that are totally fungible. And the reality is that anytime when you're dealing with it, and specialized skills, people matter, and they actually have a lot of expertise and a lot of Nuance knowledge that you don't get when you actually apply a more Waterfall type of thing. Another phrase that will say a lot of times here is that, you know, no plan survives first contact with the enemy. I'll say that in a business setting. But you know, here you are, you know, working at the army, and like, there is this understanding that like, Look, if you have a team that's in combat, and they're caught off from command, they have to have local autonomy, make decisions, and there's this doctrine of like, those on the ground have the most knowledge. And you know, you can't second guess them when you're sitting behind the lines, and you're not in their situation when you don't have the situational awareness that they have. And so, I think a little bit of this is recognizing that software is like being, I mean, I don't, I've never been in combat. So don't listen to me, I'm making stuff up here. But I think there's a level of like, when you're solving a problem, there are situational awareness, there's so many nuanced decisions that go into how you the choices that you make, that you have to recognize that it takes a skilled practitioner, and you have to trust them to be able to make those decisions. Now, that idea that like, hey, this is not just, you know, producing cogs, and that there is some, you know, discretion and creative work and value judgments that go in there. I think some of this really comes into modernizing how we think about software from being, you know, hey, I'm producing a house and I can actually go through and, and, you know, plan everything out like an architect beforehand, to the idea that this there is a, you know, think on your feet type of endeavor there. And so, I think there's some education, in terms of the reality of building software, that's a living, breathing, adaptable thing, not something that is made in a factory and stamped out and completely predictable. And I'm sure if you talk to people in factories, it's not predictable that they're either.
Paul Puckett: Well, I think you're I think you're poking at, you know, there's, there's a lot of stuff around this, that's, that's still hard, right? I think, like, what, what you're focusing on is very much the people process at the house. I mean, many, like technology's not, you know, anyone can understand it can be pretty complex some times, but when we talk about implementing change within organizations, again, back to what is the outcome that you're trying to achieve? It's the people process side that is, that is pretty much the difficult one, can you give you a few examples of, of organizations, companies, you know, federal government, or even over the commercial side, that have started to implement some of those changes, and where kind of from a people process and maybe even culture perspective, you're seeing like little changes that are delivering fundamental outcomes for the mission that they have?
Joe Beda: Yeah, I think, you know, it's always hard to bring up these examples, because you want to make sure that you don't talk out of turn with stuff that folks may have shared with you in confidence. But one of them that I have done public interviews with is T Mobile here in Seattle, I'm up in Seattle, for those who aren't aware, and, and they were early adopters of Cloud Foundry, and they're getting into Kubernetes. Now as they're looking to expand the types of applications that they're running. And I think that's a great example of that sort of feedback loop dynamic between platform teams and application teams. And so, they started out relatively simple. And it's gotten to the point now where, you know, an application team with an idea can go and self-service, allocate resources, get jump, started with their application and get something up and running in a matter of minutes, versus filing tickets and waiting weeks, and getting approvals to be able to do this stuff. And so, one of the aspects of cloud, like my definition of cloud from the application teams point of view is API driven self-service. And elastic API driven means that you can build on top of it, you can build automation, you can build systems that drive themselves, self-service means that no tickets, you're not waiting on people. And then elastic means that the set of resources that you get are not fixed, they can grow and shrink based on your need there. And the way that changes your thinking is pretty profound, because now you can do things like blue green deployments, where it's like, hey, I'm rolling out a new thing. I'm gonna bring up a new version of the old version at the same time test the new version, when I'm happy I switch things over that uses the elastic aspect of the cloud. But similarly, if a developer has an idea, and they want to try it out, can we use the elastic nature to provision them a little sandbox and let them go? Why do we bother wasting time putting people in processing Ron have them when we can actually let them just experiment for, you know, live. And then maybe the gate goes from first allocation to okay, how do I expose this to folks outside of my immediate team, that then becomes the gate. And at that point, you understand so much more about what you're doing, you've de risked it, you have something really you can look at. So being able to move, you know, you will still want to be responsible. I mean, like, hey, if you know, anybody who's shipping software, in any sort of real situation has, you know, responsibilities that they have to meet. But can you be more creative about shifting where you put those checks and how you evaluate those things, so that you enable much more self-service, much more autonomy with folks being able to drive and do things themselves. And that's the type of things that we've seen happen at companies like T Mobile, where they can empower their application teams to really move fast and experiment, and then make decisions as late as possible when you have the most knowledge versus trying to actually front load that stuff.
Paul Puckett: Oh, man, so three different things in that, that I want to that I want to poke at. But you talked about de risking, right? But you're talking about de risking capabilities from like a product market fit perspective, right, like de risking, like, because you talked about being responsible, which I heard a little bit more of some guardrails, some security implications. We'll see. Maybe the second that DevSecOps component, right? Well, there's this de risking thing, can you unpack what you meant when you said, you know, de risking what those application teams are trying to do?
Joe Beda: Yeah, you know, any process, any gate, any gatekeeping is almost fundamentally a matter of managing risk. Sometimes it's cost, it's a risk of runaway costs. Sometimes it's security, sometimes it's legal risk. Sometimes it's risk around effort, we don't want people spending too much time before we prove these things out. And so, I think the question is understanding like, okay, if you have a process, why is it there? What are you trying to protect against? And how can you actually use the tool set that you have to make sure that you are being responsible with respect to that risk in as lightweight away, if possible. And I think cloud and cloud native technologies allow you to shift these things around a lot. So, I'll give you an example. Security, right? There's a huge focus right now on secure software supply chain. At some point, I would love to get to a world where people can ship software in a secure way, without ever having some sort of manual review of things. We're probably a ways away from that they're still like in like highly regulated or security sensitive areas, there's probably still going to be some human signing in blood when you go live, right? That's still part of our processes here. But can we actually build a lot of automated systems to build confidence early in the early in the pipeline, that allows teams to move faster, it allows them to have more autonomy. And then by the time it's, it's, you know, somebody has to sign something in blood to say that this is secure. You have experience, you have tools, you have history that you can then build upon to inform that decision. Whereas right now, so much of what we do we default to trying to implement this process early because we don't have agility. After we say, yes, can we say yes, be optimistic about things, and then make the decision after the fact and do that in a responsible way. And I think that's some of this sort of shifting around of where you put your gates how you implement process, that the sort of the cloud nature, the elastic nature and the really enables and, and that then allows self-service, and self service allows folks to be able to move faster without waiting on, you know, a ticket to be resolved over the course of weeks.
Paul Puckett: So, so let's, let's dive into that one. So when you talk about shifting some of the gates around, right, in, you know, highly regulated industries that have built, you know, really hard, you know, foundations around some of the gates that they put in place, I think this is part of the pain and the struggle with trying to adopt new modern technologies to deliver outcomes. You've been setting your thing trying to kind of proving the value, but that would imply that there's still a question as to whether what you're doing is valuable, right. And I think so, too often in the in the government, we know something is valuable, we call it a requirement, and we spend years, you know, building it, only to find that maybe we were wrong. So, with all of those shifting gates, can you help me understand what you started about to talk about starting small, but like, now, in a highly regulated industry, that typically security is top of mind? What are the right gates to put in place? When you're starting? Like back to that leader? What are the metrics that they're trying to track to maintain that agility after you know, getting to a certain place helped me understand what kind of that looks like and that maturity over time to have the right you know, gates at first, but not all of the gates where you lose all of your agility to do to do anything?
Joe Beda: So one of the things that, you know, as we're, as we're looking at, sort of where do we as a company and as a community want to take, you know, Kubernetes in the cloud, native platform, one of the things that I talked about with my teammates is that, you know, you know, we'll put whatever word in front of ops. I think a lot of times when developers are getting started and trying out an idea, and they don't even know if this thing will fly, they don't know if there's some validity there. And you know, as part of this is recognizing that sometimes the ideas sometimes the innovation actually comes from a grassroots point of view and empowering that is, I think, a part of this story, too. That's a whole other discussion. But you know, when you're starting off, you don't want you know, you want to move fast as you're actually just sort of figuring out like, hey, where are the foundations where the footings and then as you get further along, you want to start hardening things down. But you don't want to have to go through and like retool everything as you go from, you know, what I'll call YOLO Ops, which is like, what we want to start out with this sort of like, hey, let's just throw some stuff out there and see if it works. And a lot of times people will just prototype something super quick, how do you transition from sort of yellow ops to responsible ops in a smooth way without disturbing your workflow that much, the idea being that you bring in the right gates, the right policies at the right time, as you're actually moving along. And so, I think if you can shift policy, right, and, you know, and shift tooling left, I think, you know, using some of the terminology here, where more automated systems providing more confidence early, because automated system oftentimes are lightweight from the point of view of self service. And then that allows you to actually move some of the heavier weight processes further right in the cycle, and let people experiment in more of a sandbox early on. One of the other things that you said that I want to pick up here is this idea of requirements in the Waterfall idea of requirements. And I think this is part of it, you know, and this is a challenge, because this is just the way the government works is very Waterfall, I want to do a project, I get requirements, I get budget, I find the team to do it, whether internally or externally. And then I hand it off to them. And then I figure out if they've if they're successful or not. There's like there's no feedback loops. There's no OODA going on there. Right? And I think, but the reality is, is that good engineering, is as much about solving the problem as really digging deep and understanding how you can change the requirements. Oftentimes, the best innovation comes when somebody says, you know what, I know, you said X was a requirement. But I actually think that if we tweak it a little bit, it'll still be acceptable to you. And it'll make the technical problems an order of magnitude easier. And that sort of like being able to work both sides of both the technical and the requirement, the product line is really that like the intersection of requirements, and you know, what is valuable and what is possible. If you can work both sides of that line, you can be so much more successful in terms of building, you know, functional, innovative software.
Paul Puckett: It's so I just had a conversation the other day with the leader of an office that said that one of their requirements was an actual fixed number of physical servers to ship to somebody. It was like, well, where's the intent behind that? Can I meet the intent? A lot of those conversations happen around, you know, the policy side as well. You said something that previously, you know, talking about these kind of shift in mindset about what Cloud means to you this API driven world where people are able to self-service, this extensibility, this elasticity? It definitely shifts the mindset from a service provider, right? That's delivering capability that understands how to enable and empower this autonomy that you see these people are, they're able to experiment quickly. But from a risk perspective, just have this discussion as well. Someone implying that DevOps essentially meant that every single developer and their mind had God rights to the entire ecosystem. And essentially, it not just like YOLO requirements, it was like YOLO operations to the to the enth degree, can you help me understand kind of from a from a guardrails perspective, a platform perspective? You know, really what that means, from a discipline perspective, you talked about the there's a discipline here, when it comes to how we do this that moves us from this this YOLO world into a bit more of a mature world, can you tell us kind of what right starts to look like, on that journey.
Joe Beda: So, I think oftentimes, I see success, when companies are clear about sort of like to do this thing, you have to meet this bar, right. And I think that's part of being responsible, oh, if you're gonna, you know, do this, you have to be XYZ certified, you have to, you know, go through this checklist and make sure that you've hit all the marks from a security point of view. And that's not going to change. And I think that's the responsible. Now what we can do is say, if you use this tool set, and if actually, you remain within this particular curated set of tools and platforms, then we can check off all of these things without a lot of work on your end and everything moves super smoothly. And I think that's a lot of where people want to get to with these platforms is provide a path whereby the time you get to the end of that path, there is no checklists because the system actually enforces the checklist along the way. And I think that's a lot of what you see with like. So, the traditional passes allow you to be able to do that it's a much more curated path. And if you can stay on that path, and you're all good to go, when it comes to meeting the bar in terms of shipping, or, you know, passing some sort of gate and moving to the next level, whatever that level may be, I think that you also need to give the development teams the freedom to say, you know, it's worth taking on the pain of going through a manual process of breaking glass and bringing in an extended tool set. And I know that when it comes time to ship, I'm going to have to justify those decisions. But bringing on that extended toolset means that I don't get the automatic checkbox checking. But it's worth it based on the tradeoffs that are making here. And so that then creates back pressure on the teams that if they just start going around grabbing random stuff off the internet and shipping it, eventually the bill is going to come due and they're going to have to actually then go through the manual sort of long form 1040 long form to be able to actually prove that they meet the bar from a security point of view versus Hey, the team that just sort of stayed within the curated path, they had a much smoother journey through this entire thing. And so, it's about bringing the right process to the right people at the right time. And then providing as smooth a path for the 90% case as you can.
Paul Puckett: So, bringing the right people to the right, you know, right process at the right time. That's hard for these teams, you know, that are starting want to own it, I want to get back to kind of the first question that I asked you like, Hey, did you imagine like, what in the world this would become, you know, wanting to deliver value, you know, even to application developers and trying to create that little that kind of abstraction layer, if you will. So consuming infrastructure? Did you ever imagine that project nine out of you know, none of seven, or seven of nine, was going to was going to lead into a culture change discussion, the way that it has, like most of our discussion here has been around people in process, could you have ever imagined that that would be the thing that you're doing now?
Joe Beda: You know, it's a surprise to me, to be honest, I think we definitely saw what we're doing with Kubernetes is disruptive at a technical level, I think early on when I was building Compute Engine, you know, we had a new, you know, director in charge of the thing, and I was demonstrating it to him like, hey, look, I run this command, I get a VM, I can SSH into it. And his take on that was like, Okay, so now what, right? And I think so much of what we've been doing around DevOps around tooling has been around answering that now what question around sort of like, I have a VM now, what do I do with it? Right? Is, is the hard thing, right? And I think, you know, everything from, you know, from Puppet and Chef and Ansible, install to, you know, to, to Kubernetes is really about answering that question. And that was very much a sense that, hey, we can change the game at a technical level there. I also had a sense for my own journey moving in, I started off in Microsoft, and then went to Google, and now at VMware. And so I've seen sort of like the shrink wrap software, when I was working at Microsoft to building and shipping SAS and how that changes the way you think about products and the way you think about agility, to you know, VMware, which is like, you know, really, is it this interface between, you know, technology and the way companies work and the real sort of reality of enterprises and institutions. And, uh, you know, along the way, you really get a sense that the technology and the people systems are just intimately interconnected. And you can't separate these things. And so, I probably should have seen the implications back in the day there, but you know, it, you have to live it to really see it. One book that I'm a huge fan of is, is Donella Meadows thinking and systems, which really, is this idea of like, how do you think about, you know, inflows, outflows. This is applicable to anybody doing distributed systems at a technical level. But as you read that you start seeing echoes in society in terms of how do we deal with, you know, the problems that we're facing, and you start seeing teams of people as systems also, and so I think, you know, as a as a leader, you know, within VMware, but also, you know, looking at the toolset that we have my job is not to, you know, is to create a functional organization that can solve problems to build that machine. And things like Kubernetes, and process and all these other things are tools that I use to build that machine, that system. And that system is both people and technology working together.
Paul Puckett: You know, great quote from an old Army General was talking about so much of the process that we've built within the federal government was built around the limitations of the PC era. And so, we see these technologies come where we talk about virtualization, fundamentally changing the way that we see, you know, our ability to leverage resources, and then we start talking about that shift into, you know, cloud computing with on demand infrastructure. I think we're also was starting to see that big shift even with Kubernetes. I think that's pretty wild Joe to think that you played a role, and now this thing that's making us talk about culture change. But for someone that's trying to understand this technology, again, at that, at that beginning of the journey, what do they need organizationally to start? I think you've kind of alluded to kind of starting small and some top cover. But where do we kind of start, and what are kind of those, you know, three, four things on that checklist that you think teams need to start to leverage this technology as a means to deliver, you know, the mission outcome that they want.
Joe Beda: I would say start with a team with a scoped mission to begin with. Try and reset the relationship to that team instead of one of, of, you know, that that is often adversarial to one of partnership, and, and then start to measure not just the outcome, so like, in the business world, and you know, and I'm sure, you know, in the in the, in the government world, also, you know, defining what your mission objectives are, is absolutely critical. I mean, we'll talk about things like OKRs. And structuring good OKRs is a really, really difficult thing. You want to be able to say, what are the outcomes that I'm looking for? And then what are the things that will show that I'm on the right path to be able to get there, those are the key results. And so, I think a big part of this is, is starting with a clear objective for that team. And then being able to say, like, what are some proxies that I know that I'm on the right path to be able to do this? And so, one of those things might be like, how fast can a team you know, can the team get started? If I bring a new developer onto this team? How fast can they spin up and be productive? How, you know how, like, how long does it take from a an aha moment and an idea about something to go from there to actually being exposed? Those are some of those sort of agility proxies that, hey, it's not the outcome that we're looking for. Right? We're looking to produce some sort of application or whatever. But in terms of driving velocity, I think, you know, based on the experience of so many people across the industry, those are good proxies, that show we're on the right way. So, I think being very careful about picking those key results, and then making sure that you're that you're creating the right collaborative relationship to start with. And then once you have that success, that seed crystal, you know, then you grow from there, right, I think, you know, it's easier to show oftentimes, than it is to tell. And so, if you can point to a team and say, this team, is seeing some success, how do we actually expand that and replicate that, that's a lot easier, and a lot more effective than, you know, writing yet another PowerPoint slide, or white paper, or whatever about what we should do. And so, a lot of this is just get out there and demonstrate it. And that often starts to speak for itself.
Paul Puckett: Now, I very much agree with you on that I have always said kind of this, this culture chain really starts with outcomes, because people have to have to really see you know, an outcome and in something valuable to, to then kind of entertain the idea of how can I see myself, you know, being a part of this winter also be role, though, in those people that are kind of the first movers to understand that part of their job is to create perhaps a repeatable path behind them? Can you Can you expound a little bit on kind of there's this, there's this other role that those teams are really playing within an organization?
Joe Beda: Yeah, and this is so tricky. I think, you know, this idea that, that you're both solving the problem, but you're also, you know, building a better world for those that come after you. It sounds kind of hyperbolic there. But there's always a balance there. Also, because I think it's too easy for teams to get into sort of a, a perfect platform mindset, let's build something that's going to solve all problems for everybody, only to find that they haven't proven it out along the way. And so, I think this is a place where again, there's a matter of taste, or like, how much do you build for the future? And how much do you build for now? How much do you know, how far down the road do you look? Right? If you're looking 10 years down the road, you may build a platform that, you know, doesn't work for now. Whereas if you only look a quarter down the road, you may find that you're thrashing based on problems that if you took a little bit of a longer view, you could sort of, you know, factor those problems away. And so, I think there's a certain amount of making sure that you're thinking on the right timeline, in terms of like, you know, whatever that is for you, but like, but not too far ahead. Because I think, you know, I've definitely been on teams, you know, I, one of the teams I was on at Microsoft was the Longhorn team for what eventually became VISTA. And that was an experience of people were so ambitious. And they were thinking about, hey, we're going to build the platform for the next 15 years, that they really overshot, and they didn't focus on sort of like, well, what is the reality that we have now? And I think, you know, this is a this is a relationship, strategy and tactics, right? You want to make sure that you set your strategy so that it's realistic and achievable. But it's important that you show progress along the way. And if your strategy is not believable, if it's too far ahead and people execute against that you may actually find that you don't get any short-term results in that you just spin your wheels, and you never really even get there.
Paul Puckett: Is this one of those differences between the commercial and the Federal world? Where kind of that time to value is seen differently like that, you know, people that are engaged with it just like okay, well, then how long do I spend, you know, researching the problem being strategic about it? And then, you know, when do I pivot to tactical? How far? How far ahead? Should I be? Should I be looking? Is there a period of time looking ahead, that, in your opinion is too far, and I'm sorry, for trying to kind of have good, like, pinned down a cloud here, pun intended, but like, what is that right, kind of like window of opportunity?
Joe Beda: What I would say here is, this is again, systems thinking here. So, one of the things that that inside of Google with Borg, there was an internal economy, Google could not build machines and get them to data centers fast enough. And so that means that machines were precious. And so, when something becomes a precious resource, people tend to hoard it. And that ends up making the supply chain issues that much worse, right? This is a lot of what we're dealing with right now with COVID is that when you end up with shocks to the system, it causes essentially ripple effects like wash boarding on a on a gravel road. And so, when you introduce a lot of friction, when you make something expensive, and approvals and going through process inside of the federal government is expensive, people tend to hoard, they're like, hey, it's gonna be a pain in the butt to be able to do this. So, let's just do it once and for all, and that once and for all mindset is a hoarding mindset. And that means that they're trying to bite off more than they can probably chew. And so if you can get faster feedback loops, but that comes at, you know, by reducing friction, and I think this is why it's so difficult within a government mindset is that like, like, those things are written into law, right, a lot of this friction is, is not optional, whereas what you find in the in the corporate world is that, you know, you still have in, you know, larger companies that are working to reduce that friction, but the pressure they're feeling is from startups who essentially have zero friction. And so, the less friction you have, the more agile you can be, the more just in time you can be and the less hoarding there is. And that means that people can set their sights at a much more realistic horizon versus trying to do it once and for all. And I mean, I don't know the solution to that that's a hard thing. But just getting people to recognize that problem, that friction leads to hoarding friction leads to hoard efficiency is something that I think you know that that's a dynamic that I wish more people would understand.
Paul Puckett: Yeah, there's definitely like a total cost of ownership that's in there, right? Because you talk about things being expensive. And I find, oftentimes, in the government, there are all these different touchpoints across all these different organizations. And you start to see some aha moments when people start to map like, how much time am I actually spending in these steps along the way? And then from a cost perspective, what does that actually costing me? You know, in the army, we've started to really focus on how do I start to automate some of the toil that's taken away some of our soldiers time, you know, people to your earlier point, fewer precious. So how do I start to kind of remove those gates, but it's back to that kind of, you know, signing in blood, like I've pricked my finger, numerous times of like, going to do those things, when it comes to those reviews. I find that to be really, really challenging.
Joe Beda: I mean, just to riff on that a little bit, I think like, oftentimes, we think about like a process as a single gate that's monolithic. To be able to reduce that toil. Oftentimes, we need to refactor what our requirements are to some degree, and maybe we can say, hey, you know, this process is doing three things, two of those things, we can automate and move earlier and actually make it automatic. And that one thing, well, that's really expensive. Why don't we move that later? And we can actually then do that. And so, it's really this, you know, it's, again, it's like understanding the requirements, can we understand the intent? Can we back up and make sure that we can deliver that same intent, I think this applies to the technology also, so much of what we're doing in the cloud native world is really sort of one step back two steps forward. Security wise, a lot of the traditional security tools don't work in the hyper dynamic environment that you get with something like Kubernetes. But with things like get ops and sort of the manifest view of the world, we can eventually move to a world where, and we're not there yet. But I want to get there where you know, every single process that should be running in your data center. And if an interactive bash shell pops up, alarm bells start going off, that would be beautiful, to be able to get there. We're not there yet. But to do that, we need to be able to take a step back and then rethink things and then be able to restructure to be able to get out of the local minimums that we find ourselves in right now. And that applies to the people processes. Also, you need to be able to refactor, you need to be able to make sure that you can get to the same destination via different ways.
Paul Puckett: Is your comment about just kind of like the alerting side of the house from a security perspective. You know, oftentimes in a lot of worlds, we have so many alerts, we don't know when to what to make sense of what makes sense of what, so you said we're not there yet. What gaps are you hoping start to get filled in that ecosystem, and what role do you see Kubernetes either playing or enabling on that journey?
Joe Beda: I think a big part of this shift that we're seeing is moving towards higher level intense about what we're looking to accomplish. Oftentimes, when I describe how Kubernetes works in the declarative nature of Kubernetes, I'll talk about self-driving cars, when we think about a self-driving car, and again, they're not there yet, but I think, you know, our idea of a self-driving car is that, you know, we could say, Hey, you get in the car, and you say, Turn left here, turn right here go 40 miles an hour, it would still be a self-driving car, because it's driving itself based on some level, but like, that's not what we want, what we want is to get in the car and say, take me to Grandma's house, take me to the grocery store, what have you take me home. You know, that is, that's the level of intent. And so I think a lot of times when we see, you know, alert fatigue, when we see, you know, a lot of the friction that comes out of these systems, it's because we don't have that higher level intent of what we want to do, we're still talking in terms of turn left turn right versus taking you to the grocery store. And so, I think a bigger the challenge is up leveling the way we describe our requirements, so that it's not, hey, I need, you know, 10 machines in a data center, it's like, I need to solve this problem, right? That's a higher level of intent there. And then, you know, yeah, and so that's, I think, the one critical piece of the puzzle here, and eight. And those things are always multifactorial. So, like, there's always other aspects of it.
Paul Puckett: So you bring up a really interesting point, I kind of want to wrap it, wrap it back to kind of the you know, the ends Ways and Means discussion, which you just articulated, there's like, you know, I want to get to Grandma's house, like that's, that's the end, the way is the means of like turn left turn, right. And oftentimes we're measuring, am I turning left? Or am I turning right? Like not? Am I getting to my destination? So, it kind of in the same vein of the security side of the house, it starts to shift the role of people and understanding what business they're in, right? Kind of back to, you know, DevOps DevSecOps, it's getting people together to create these feedback loops. Because like, well, what business are you in go developer operator, it's, there's some mission objectives that you have here. So kind of just kind of bringing it back home is like moving forward, and where we need to go, what are some of like the big shift mindsets that you think are really disrupting IT organizations today, when it comes to understanding the business that they're in, and maybe like, not, not pointers, but like maybe a few kind of encouraging components of like, focus on these things to truly realize your value within your company, your industry, your business, your mission?
Joe Beda: I think, you know, I can speak about sort of the commercial side of the house. And I think there's definitely lessons to apply to the, to the government side. You know, the previous world was one where the only place that application teams could get resources was the other central IT and cloud change that because with the advent of shadow IT, all of a sudden, some engineer could go to a cloud, swipe a credit card and get moving while they're still waiting for their ticket to get resolved. And so that created competitive pressure within the organization here to be able to do that. And what we see is that, you know, within enterprise organizations, you often have IT teams taking a sort of a siege mentality of like, wow, the world has changed out from underneath us, we can no longer assume that our customers are captive, and that they can only come to us, it's no longer you know, selling drinks at Disneyland, right? It's like, now all of a sudden, they have choices where they didn't have before. The useful way to think about this is what are the lessons that I can learn for why my users are actually choosing to go elsewhere, and then compete and actually start to bring some of that stuff back in house. And so that shift in mindset towards a platform team, instead of like, hey, you know, you file a ticket, and maybe I'll give you a VM some point, it's got to be like, how do I provide a level of service that is good enough, so that, you know, people don't actively try and subvert all of our controls to be able to do this stuff. And so, it's that idea of like, yeah, you got to compete, you have to provide a prod and internal product to your teams, so that they can actually be effective, you have to reduce that toil. And I think the forcing function with enterprise is that oftentimes, those teams, they don't have a choice, but they do have a choice, right? This is the idea of like, you know, they, they, you know, it depends on the organization. Like if your health care, there's a lot more but like in a lot of organizations, like it's like, you know, wink, nudge, nudge, okay, you're going to cloud, you're not going through Central, it will look the other way, because you're so effective. That's creating that pressure for those teams to be to start creating that that platform mindset, that product mindset. And, you know, if we could bring that into other domains that don't naturally feel that pressure, either and that's where we're going to start to see that success.
Paul Puckett: Now I completely agree with you is at the startup of the agency, we're talking about how you're going to deliver capabilities. And the goal is to deliver enterprise cloud services that the army loved. And there are definitely journeys along the way, where people like, oh, well, they have to come to us, because it's policy. And I'm like, these people want outcomes, they want to be efficient, if we are toil for them. That is the that is Shadow IT, you know, to the nth degree, people are just going to do what they have to do in order to get the mission done. So that change in mindset of how do I actually capture a market that makes people want to use my services? It's a fundamentally different mindset within the DoD, where typically, Policy Governance and you know, there is no one else to go to kind of this like that, you know, you have to go use it. So that I find that just so invaluable. I really, thank you for doubling down on that mindset. Joe, we we've been at this for a good little bit. And I think it kind of makes sense to close on that. It's still the last question is, the whole entire point of this podcast is talking about mission first, right? Now the new plank owner of mission first podcast is, you know, customer number one, that what do you hope that this podcast can help people realize over time of that role of people process and technology, when it comes to the mission that we have in the federal government?
Joe Beda: Well, I, you know, I want folks to more than anything, recognize that technology doesn't stand on its own, that it really is this intricate interplay between the culture the processes, and, and the technology to be able to produce that outcome. And so, there's no silver bullet here, right? This stuff is hard. It takes experience, it takes experience you know, it takes trust, it takes iteration, you're gonna make mistakes along the way. And, but being clear in terms of what your goal is, and focused on that outcome, you know, that's how you that's how you get success, not based on you know, well, I adopted this tool, so magic is gonna happen. That's, that's never gonna get you there.
Paul Puckett: I think that's just so profound coming from one of the founders of Kubernetes, a technology that is disrupted, commercial and federal, you know, businesses writ large change in the way that people organize change in the way that people look at outcomes. To hear you say that is something that they're really kind of carries a lot of weight with me. And I hope that people take that away. So, Joe, I really can't thank you for spending some time together. I'm honored to be able to chat with you for this time. So, thank you so much for being here.
Joe Beda: It's a great conversation, and I'm always happy to talk about this stuff. So, thanks for having me on.
Paul Puckett: Absolutely. So once again, my name is Paul Puckett. I'm the director of the Enterprise Cloud Management Agency with the United States Army, and you've been listening to Mission First.