Watch episode two of Mission First, our Department of Defense (DoD) and National Security podcast series focusing on the mission, not products. During this session, Jeremiah Sanders, Co-Founder of Kessel Run and Senior Federal Strategist, VMware and Alex Barbato, Senior Engineer, VMware Tanzu discuss:
Tune into our monthly podcast series, Mission First, where we will be focusing on the Department of Defense and National Security mission and not products. Hear from thought-leaders in the industry as they discuss complex challenges and topics buzzing within federal government IT.
Craig Bowman: Hello and welcome to mission first. This is a podcast that focuses on government mission, not products. So, today's topic is continuous authority to operate or continuous ATO. Is it possible and we talked about a lot but isn't real? So today we have the pleasure of having Jeremiah Sanders on our podcast. Now Jeremiah was a, an Air Force Flight Commander, a Branch Chief for the Network and Information Assurance Division, the Deputy Chief for NROs and Rock Mission Support, Branch Chief for the Air Force Acquisition Command and Control, a Material Leader for the AOC Weapon System, which is like a Program Manager for the AOC Weapon System. And he was Deputy Commander and Co-founder of Kessel Run, which is the focus of our conversation here today. Jeremiah, you got three degrees, right?
Jeremiah Sanders: Yeah, I do.
Craig Bowman: Yeah. And so, you're basically making the rest of us look like we've been slacking our whole lives. So, thanks.
Jeremiah Sanders: Well, the military sent me to school. You know how that goes.
Craig Bowman: Alright. And we also have on the podcast, Alex Barbato. So, he's a former Staff Sergeant for the Air Force Crypto Linguist. He also worked on some implementations for Amazon Alexa. So that little device that sits in our house and monitors what we talk about every time I have an argument with my 10-year-old daughter, I have him to thank. And before that, he was introduced in sort of the private sector where he worked on fitness apps and advertising. Now he's a Solutions Engineer and Outcome Enthusiast, for the Public Sector Division of VMware’s Tanzu Team. So welcome, Alex and welcome Jeremiah, to the podcast. Appreciate it. I don't like wasting time. So, I'm just going to jump right in here. So, Jeremiah, perhaps I think the best place to start is to just kind of level set with our listeners, can you can you kind of give us a working understanding of what is continuous authority to operate or continuous ATO, which I know is kind of evolved into this continuous risk management framework. But maybe if you kind of level set, what does that mean?
Jeremiah Sanders: In federal government, and particularly in the Department of Defense, there's a rigorous process in process owners for accrediting software. And it's the mother Mae eye that allows a development organization to get their software or infrastructure components onto the network. And it is a stage gate to production. And so, when we talk about continuous authority to operate, it changes the paradigm where typically accreditation would be done at the end of development, you know, considering software being done, and then we're going to put it on the network from a long time. And that kind of bolt on process at the end of development was very manual. And in the Air Force, in our case, it was taking about 14 months for that accreditation process to ensue. So continuous authority to operate changes the paradigms to cover down on the equities of the accrediting official and do it in a more automated and repeatable way, in fact, in a continuous way that very much looks like accrediting the process of software development and delivery as opposed to individual software products.
Craig Bowman: It's interesting because you talk about continuous ATO as being a delay in the process. And I know the Kessel Run, it's widely accepted that it's the first organization in the DoD to achieve continuous delivery. So, here's what you're saying that to get to that continuous delivery, that authority to operate was the largest roadblock to that process.
Jeremiah Sanders: It was one of the very first challenges and out of the gate is an organization that started with the notion that we are going to achieve continuous delivery in the Department of Defense. We knew that solving the accredited the timeline of accreditation was a big part of that challenge. But let me back up and explain you know, what is continuous delivery and why were we are seeking to achieve that from day one. So continuous delivery, in simplest sense, is the ability to affect change in code in production. And anytime on a whim. So, any software developer or system administrator having the ability to go in and make a change to software that goes into production without the typical Stage Gate and of accreditation isn't standing in their way. So continuous delivery? There's an organization called the DevOps Research and Assessment organization. And they look at about 1500 different IT and software organizations in public and private sector worldwide. There's about 35,000 survey respondents, and that answer up on how often are you deploying code? What is your lead time for changes to code? How many times are you pushing changes to code that fail, there's a lot of metrics that they're looking at. And from all this data that they collect across the world, they can figure out who are the high performing IT organizations, and they go, they've gone a step further, in the book accelerate, I would highly recommend that book. In that book, they delineate it and scientifically proved out the math behind 24 attributes of high performing IT organizations. And the single most key differentiator of high performing IT organizations is the ability to continuously deliver. So, coming off of in the 2017 timeframe coming off of what was it a dramatic, large scale DoD acquisition program failure, a modernization effort for the air operations center weapons system that had been going for 10 years, $430 million dollars was stuck in development and delivered a single line of code. We had the opportunity to do different and we were going to be different. So, we had learned that we wanted to model ourselves as an organization after the 24 attributes that would achieve us, you know, put us in the status of being a high performing IT organization have having better outcomes in development and delivery of software and achieving better mission outcomes for our warfighters. So, with malice aforethought, you put continuous delivery, the ability to continuous delivery is paramount right in our vision statement. And the first big roadblock to that was the standard bolt on at the end in manual DoD accreditation processes.
Alex Barbato: I guess, Jeremiah, I want to like maybe play a little inside baseball and get your perspective on. Like ATO, right, ATO is hard authority to operate hard. My understanding is Kessel Run created the construct of a CTO like that means a specific thing in the domain of Kessel Run, then, now, I think we've started to start to use the term continuous RMF. Because it's more widely accepted, it's easy to understand there's nothing actually in the risk management framework that precludes you from doing continuous delivery. It's actually written in there that you can do it. But I guess I if I've always been curious, like, was there like a meeting where you talked about what you know, what are we going to call this new construct that you're like, working on? I don't know. They might not have been that fancy, but I'm curious.
Jeremiah Sanders: Yeah. So, the genesis of what was Kessel runs continuous authority to operate in your right now, you know, you look at the DoD risk management framework. It is it is a, I think, a better frame of reference to say continuous risk management, you know, the continuous RMF in in the development and ensuring a provenance of code through the continuous integration pipeline, and then in production, the continuous auditing, logging and response to any security issues. But to back up to our first continuous authority to operate the first in the DoD. We actually stolen improve something that was started in the intelligence community, specifically, the National Geospatial intelligence Agency, had a working platform as a service on the seeker net SIPRNet that we didn't have when we started. And they had this concept that they were working on called ATO in a day. It was the notion that, you know, I mentioned our Air Force timelines for accreditation, were around 14 months. And they the NGA, were on the path to driving that process down through a high degree of automation and repeatability to the concept of ATO and a day. When we came onto the scene. The NGA was a great partner for us because as I mentioned, we didn't have continued CI/CD tooling. We didn't have a platform operating anywhere. And our starting point, and Kessel Run for the first several months, we built and delivered apps on NGA’s pipelines and platform. And so, we started with their pipelines, brought them into Kessel Run, enhanced with more tooling and more automation, along with the help of pivotal at the time now Tanzu Labs. So, we bolster everyone NGA has started and grew it, and then through that process, improve the efficacy of what was ATO on a day to make it truly continuous. Part of that is the automation and tooling improvements that we did, and building a practice of how to do continuous authority and operate. The other part was the culture and process change in that the accrediting officials in the Air Force the senior executives, I've never seen anything done like this before. So, working for the SAF CIO at the time, was Lauren canals and Berger, the she was in She's currently the SAS CIO, that was at the time innovation authority for the SAF CIO. And we approached her as well as the accrediting official for the AOC weapon system, Danny Holtzman, with this notion of being able to cover down the equities that they would be signing up for and accreditation, and doing it actually in a much more automated and accountable and insightful way we gave them and their teams, the security assessors, a full open kimono access into our development environment into our ops environment. And, you know, when it comes to what would typically be a static accreditation package, the pile of paperwork that would sit on their cutting officials dashboard, you know, a year in the queue, we were giving them that insight in an automated way through a new set of tooling. And essentially, our plans were done on the go in an automated way. And we were doing infinitely more exponentially more security testing, you know, gone are the days where you would run an HP fortify, scan, and print out the results. And that was the package that went to the accrediting official. We were doing four to five scans and many other tools scans with every code commit by every application team or the platform team multiple times a day. And all of that documentation then was accessible to the accrediting official. So, it became much easier for that organization to understand that we were doing a new security paradigm, but it was a better security paradigm where speed is the new security, we could affect change and implement security updates in production much faster than the traditional sense and improve the security posture of our system baseline.
Craig Bowman: So, Jeremiah, you've said a couple of different things there. That's interesting. And so, you talked about how many times you were scanning. If you were if you're a new organization, and you're thinking about how do I get continuous ATO? What's your recommendation for how they should start? I mean, I assume you didn't. You didn't like wake up one day and say we got to do all these things in order to get continuous ATO is probably a process that you worked through with security. Now looking back, what are some of the like milestones that an organization like a federal organization says they I'm setting up a Software Factory, I want to do 81 A Day or continuous risk management framework? What are the sort of like the milestones they should be striving for if they want to try to follow what you guys did Kessel Run?
Jeremiah Sanders: Yeah, so that's a great question. And I'll preface by saying that when we started as a government organization doing a government led integration approach, and we had traditional system integrators on our team, we started bringing in other small business companies that had specific attributes and competencies that we were lacking as a government. I mentioned their relationship with what was Pivotal Labs at the time now Tanzu Labs and that was really the, the core of our know how and learning process. So, we knew we wanted to achieve continuous delivery. We knew that meant we had to get to, quote unquote, continuous authority to operate. But we didn't know how to do that. And it's not just a matter of implementing technology. When we look at digital change through the lenses of people process technology, a lot of it hinged on the hearts and minds expedition of Working with the, you know, the security assessors and the accrediting officials in we were all in it together at that point to realize a vision that hadn't been done in the Department of Defense before. So that that was really the starting point. And I would say, for other organizations, now, there's a bit of you know, how that has been shared more than once across other DoD organizations. And Alex can speak to that. And actually, Alex, why don't you take him in explaining some of the other organizations that have gone on this continuous authority and operator CRMF? Vision not just in the Air Force, but SR services? And how are they starting? And how are they growing that capacity?
Alex Barbato: It's an interesting journey that has sparked now, along this path of, you know, moving faster, delivering outcomes faster. And frankly, I'm, I'm excited for it. I'm grateful for the work that your team put in that the folks that were on the ground at NGA did, and probably many people before that. But I think the two core components that we see teams really coming out and having to tackle when they start their CRMF journey today are one, how do I get in? This isn't in any particular order. But one? How do I achieve the same compliance outcomes that I need to see, right, because the construct that Kessel Run or NGA had was one and it has evolved, and it's changed over time? And it continues to evolve to where we are trying to now like really exemplify that model of, we talk inheritance, we talk reciprocity, we talk like you know, controls, all of that should build up. So we see that that model has refined a lot refined and refined greatly, to where now we start to see really clear lines between infrastructure and heritability to platform and heritability to like, hey, now my app only has to do 30 things like this is way better than the, you know, 72 pieces of paper that I had to do before. And we start to see, you know, it, it's RMF. It's, there's no, we don't need to call it anything else. If we don't want to, you can call it whatever you want. But there's that. And then there is also the actual security outcomes, right? I think organizations broad broadly speaking, I mean, we can say DevSecOps, we can say CTO or whatever. But I think the government is really becoming aware and, and demanding that they can operate above their value line. They're tired of being stuck in this space, where they're managing things that aren't adding mission value to them anymore, that I don't want to worry about. You know, every org is different, like they're going to draw that value line in different places. But in your case, like if we take for example, you know, like, like army Software Factory, or Kessel Run, pick your org, right? You actually Maru pick wherever you want to go, right? But they're building applications, right? So, they should be focusing at the application layer. So how do you build the automation that allows you to behave it that way? What's the platform that doesn't, you know, lock me into anything crazy, helps me build cloud native applications and secure ways. And gives me a ton of value, right? Like they're not opposed to paying for value. But those two things mash together, I think is the magic of today.
Jeremiah Sanders: Alex, you mentioned you kind of hit on the bottoms up inheritance and building of the continuous authority to operate. The underpinning of that is a high degree of automation and bottom-up security inheritance. So that that was one thing that we implemented at Castle, one that's been refined and repeated is for every infrastructure stack that we were going to land on, we did essentially, a traditional infrastructure layer or accreditation package. And then we had this new implement what was new to us and hadn't been done in the air force, this past layer platform as a service that was essentially just infrastructure and developer tooling, as code. So, it's a piece of software. And we wanted to be able to update that piece of software, that runtime environment and will so we got a continuous authority to operate for that layer of the architecture as well. And you're absolutely right, in terms of abstracting complexity away from application development teams, particularly, as you know, we as a very fledgling government software development organization, and any high performing it or software organization, you want to abstract complexity away from the devs so they can focus on the mission or business value new creation through functional code and the associated data, not the demands of infrastructure and runtime and locking everything down from a security perspective. So high degree of automation in the stack and the security tooling around the stack from understanding provenance of code in the development environment itself, and in the ops environment, all key and it wasn't that we had it all up front, we actually, were learning how to do all this didn't have any of the tooling when we started and had to beg borrow and steal I mentioned from Nga, and improving along the way. So, there was a lot of a holes and room for improvement in what we initially instantiated. And we've continued to evolve. And now I'm, you know, in industry and seeing it perpetuate through the Department of Defense and other federal agencies that we've been working with. You mentioned a couple of them do SpaceForce a Kobayashi, Maru software, factory, Army, software, factory, Marine Corps, make boss, platform and environment. They, every one of these engagements has helped to better refine this overall continuous risk management framework construct those repeatable playbooks to go along with it. And so, you know, other organizations in public sector are in such a fortunate can condition that they aren't trying to do something for the first time. And they have, they can phone a friend for help.
Craig Bowman: So, Jeremy, I've got a question for you. What about what about STIGs? Right. I know that when, in a previous life Ford came here, we were working on, could we automate STIGs as part of a CI/CD pipeline? Was that ever part of the equation for Kessel Run? And how did you guys integrate, you know, either STIG detections, STIG automation into the Kessel run program?
Jeremiah Sanders: Yes, certainly. I mean, because Kessel Run, you know, from an architecture or implement perspective was networks, routers, switches, servers, workstations, Mission applications, all the above, it was a full stack and out to the end user environment for their operations center. So, seeing evolve those network components, the server implementations locking down the operating systems, we had to abide by those rules. So certainly, as we built from the kind of bare metal up the security architecture and the bottoms up inheritance that we talked about, we looked for every opportunity to automate and make repeatable because we wanted to, one of the things that we were learning to do was, for example, as a way to effect a better security posture over time is to have the ability to blow away servers and, and repave them, essentially, reconstitute them in a very automated fashion and do that kind of round robin, every few weeks. So, we wanted to, we absolutely had to comply with the STIGs. We automated the way that we implemented STIG lock downs in in those various systems. And then from a tooling perspective, yes, we were configuring tools that would verify series as we were doing continuous integration. And that happened multiple times a day is really interesting. I was gonna ask Alex, what, you know, what have you seen in terms of STIG implementation and other organizations you're working with?
Alex Barbato: It's interesting. I don't I don't have any, any other perspective to like automating STIGs and what you said like, it's super important. The only way that I would say that I am trying to look at the picture more and more today is, is focusing on compliance outcomes and security outcomes. And there's overlap. And there are disconnects. Because not like, I think this I don't even think this is a controversial take today, today to say something like this. But I think we can effectively say that. There's like the model in which that we're expected to like go line by line through STIG after STIG after STIG when it's not even possible to have STIGs for every possible appropriate configuration. If the model doesn't work, right. Like it's not it can't be effectively implemented. And I think that's, again, I don't even think that's controversial, because that just like the data shows that with the number of breaches with the number of vulnerabilities, the fact that we've had to adopt this network perimeter model that is like, as long as you're not in the network, we're okay. There's a lot of anti-patterns that this has introduced. Right. So, I do see somewhat of, you've got to, you've got to meet the teams that you're working with where they are, though. You can't just like throw the baby out with the bathwater. Right? Like, there are good things in there. And different authorizing officials are going to look at this differently. Right. And I think the one that authorizing officials that most excite me on the on the sticking controls front are the ones that are really focusing on Hey, are we meeting the intent of the controls? Am I creating an environment that's like, you know, in the spirit of RMF, NIST 853, right? Like, my meeting, the intent of the controls, and my implementing STIGs, you know? And then okay, cool. Now let's focus on automating the appropriate security measures the appropriate monitoring, right, and really focusing on risk, rather than I think even myself, I've fallen, fallen, I don't want to say victim. But I've definitely gotten myself in the mindset that I need to automate all the STIGs and like, that's the top priority. And I'm like, wait, like, wait, maybe I'm going right back to where we just weren't, like, I need to focus on where the vulnerabilities are, where the most amount of risk is in front of me. And when you can have those conversations like you've achieved nirvana, then you could focus on you know, if it's a STIG, it's a STIG great. And you need to, but you need to make that visible. How do you make that visible? What makes sense? I don't know, that varies. But I loved a wide variety of views, and ways that we're seeing this be implemented today. From I want to see controls, do I want to see stick automation till I want to see just bare bones scans and various types of, you know, automation in that domain, you know, focusing on infrastructure as code, there's so many ways you can tackle that, and create visibility monitoring tools, I mean, it doesn't matter. But as long as you're creating a pane of glass, that's what that I mean, that's what's making me excited today,
Jeremiah Sanders: You brought up an important point, Alex, and that is the transition from what used to be largely a compliance or check the box routine to get accreditation, to one of actually considering risk on a continuous operating basis, truly security operations, right. So that paradigm shift is this still a work in progress, when you take a step back and look, DoD or national security organizations writ large, there's a lot of room for moving beyond just the compliance checklists aspect of accreditation, to what you're saying. And there's another aspect of it that that I'll bring up to, organizationally for us, whether it was the infrastructure operations teams, or the security operations teams, or the platform operations teams, as they embrace this new model of DevSecOps, you want to use the buzzwords this, the things we were doing to achieve continuous delivery. So that we can iterate very quickly, the application functions and deliver value to users and validate that we were building the right thing that the application teams were building the right thing. The rest of the organization, infrastructure, platform ops, security ops, changed our mindset from what was either, you know, a checklist oriented Alliance mindset, or a very administrative function mindset to one of owning jointly, the outcome of helping our users and putting that work in a very visual and open hourly, hourly way for everybody in the organization to be able to see what was going on. If a development team was unable to get to production, for some reason, everybody knew it. And it was a big mindset shift for platform operators and security operators to become service oriented in that they saw as their users, the application devs and the downstream application users. And that was a mindset shift for them. Where, frankly, when you look at security assessors or the accrediting official or cyber teams that are involved primarily in the in the development of acquisition systems that are taking years and years before they deploy anything. Their mindset is not one of being responsive to actual user needs, because they're not delivering nothing. And there was a tremendous change in our organizational culture to essentially everyone gained this warrior ethos that it was their job to help the end user at the end of the day.
Craig Bowman: Yeah, it's interesting. Listening to the two of you talk. I'm just gonna take some takeaways. So first of all, Jeremiah, you said you NGA had this idea, right? You guys heard about this. ATO in a day. So, you had this sort of like North Star that you knew you wanted to get to. You brought in a team of people that have done it before. So, you brought in experts, and you're moving along in that direction. And then there were people process technology, all of those had to sort of be worked on in order to get this thing done. So, I'm trying to think like for listeners who say, hey, we want to get to continuous delivery, it's looked to the look to the models of people that have done this, they've done it successfully. Realize is not a technology decision. It's a people process, technology decision. And in that, you know, Alex, you talked about the bunch of different teams that, you know, there's all these different teams involved. And you said, there's so many different ways people are trying to solve this problem. It reminds me of the last podcast we had, where Paul Puckett and Joe Beda, were target talking to things about that podcast, Joe talked about just the power of Kubernetes is that there's so much that people can do with it, there is no like one way to do it. And everybody's like doing their own rolling their own. And one of the, one of the challenges of Kubernetes is that everybody can roll their own, everybody does their own thing. And what Paul really was bringing was what you guys are talking about which right here at the end, Jeremiah, which was this is a people problem. This is a team's problem, that once you get the North Star, and you get the team and the experience, and you start working through this, it's getting all this buy in. So, my question to both of you then, knowing that there's all these different ways you can do it as people process technology. There's no like blueprint for success, even though they're guide points, as I call them. Whose responsibility is it in DevSecOps to ensure security? Is it the apps folks? Is it the network team? Is it the CISO and the SOC, who are traditionally network? Whose responsibility, is it? And should they be talking more together?
Jeremiah Sanders: I'll say that it's everybody's responsibility. And we had many an application developer that had to learn that lesson as well, that part of being able to deliver at a sustainable pace is to ensure that you deliver with quality and you deliver with security so that now you're not just responsible for building features, but to deliver those features in a secure way. So, application developers, the network ops team, the security ops team, the CISOs, everybody has the responsibility for security. And it meant that in our organization, there was a lot more communication, about security across those teams, a lot of that communication was afforded through automated tooling. And a lot of it was getting up out of your seat and walking around and talking to people and, you know, embedding security team members into App teams to help them as they were onboarding and learning how to, to you know, imbue the software security elements into their applications from day one. And one of the 24 attributes of high performing IT organization that is Dora pointed out is shifting left on security. And that means the entire organization has to be security minded from the get-go. Day one and first line of code.
Alex Barbato: I just want to play devil's advocate and quote, quote, Corey Quinndevit, and that he loves to tease out the, you know, saying security is everyone's responsibility means security is no one's responsibility. Yeah, and I think that's dead on, I think it's dead on. And I only am teasing here, Jeremiah, because I think you're being new. But there's a Nuance to what you're saying. And you, you yourself said, and I want to like maybe poke a little bit at Craig too earlier when he said the takeaways of. I think really, what you're saying is, we need to focus on outcomes, and we need to incentivize, to outcome outcomes to mission outcomes. And the only way you can be confident you're delivering a mission outcome is if you're connected to the mission, right? I can't tell you if I'm delivering a mission outcome. If I get feedback, once every five years, right, like, that's not that doesn't work. That loop is far too long, right? So, we talked about shifting security left, it's shifting behavior left to being able to adapt quicker across all of the teams. So, my say you get rid of security, oh, please don't, please don't. Like I'm an application developer, like I do not, do not have the qualifications to come in and tell you the proper ways to implement mutual TLS. Or you pick your pick your security thing, right? I still need those security people here. But I need them to be incented to behave in a way that's compatible with the way that I'm behaving. And they need me to be incented to behave in a way that's compatible with the way that they're behaving right. If they're able to just say no, you don't meet this checkbox. I don't care Like, oh, come on. Now, if all of a sudden, like, you know, something really bad happens, I promise you that node is going to go. And we're going to deploy the system like I promise you. So, we should practice like we play. And if and if the way we're practicing doesn't work, then you can't tell me no, you need to tell me hey, blinded a path to green, I need to maybe there's a different way we can tackle this, let's work together, I'm incented, for you to be successful and Bide on your risk. Similarly, app developer, it's not okay, for you to work around security and try and get around them, and to not leverage your expertise to try to deliver secure outcomes, right. And that goes to a lot of that automation we built, right, like, developers kind of get it easy in some ways, and that they can implement scans or testing, and they can get that really quick green read, right. And that's, that's okay. That's what they should get in some degrees. So maybe slightly long winded. But I see you nodding your head, Jeremiah. So, I think you agree, like, there is still, you know, some level of responsibility, but it's about incentive. So, you've got to make sure.
Craig Bowman: Alex not just worried, like, you think about the traditional SOC operator, right? They've been focused on network security. So, things fire off, they get tickets, and they got playbooks, and they're implementing, and they're, you know, they're going into a router, and they're making a change. This is a totally different paradigm. Right? So, this is not that paradigm. These the sock operators are not trained in a DevSecOps environment, or CI/CD pipeline. If they got a ticket, let's just say everybody goes, well, we'll just feed we will feed the logs into Splunk. Okay, it goes into it, it goes into a, you know, some kind of a sim, and now your sock operator gets this thing and a, there may not be a playbook for it be not sure they know what to do with it, even if there was they may not be tied into the CI/CD pipeline to be able to fix it. And a lot of the security breaches that we are worried about with Dev SEC ops have to do with API's. Right? We used to have like four different applications. Now we've got 1000s of containers, and API's calling everywhere. And so, what does that look like? Either one of you, like, how do we merge the world of, you know, when the CI/CD gets released, and you've gone through this checklist, Jeremiah Kessel Run, and it goes into production. And then they go, oh, this new, this new vulnerability comes out. Sometimes the developers know about it before the sock operator even knows about it. So, they're already making the change, before the sock operator even had a chance to get the ticket and respond to it. So that's what I mean, like, I'm trying to understand like, what is the future for security? If security is 1000s of containers and API's making calls, and our Network Operations are not in our sock is not traditionally CI/CD? I just don't know if we're making that connection, I guess.
Jeremiah Sanders: We've used the word we've kind of beat to death to the word automation on this call. And but it's absolutely key, you cannot achieve this new paradigm. Without it. What you were referencing earlier about? A new paradigm is absolutely right, you know, a traditional program sock operator where the system accredited negation goes into effect, and you do system updates, maybe quarterly, major version updates every five years. That's the life of a traditional security operator. But for in the new ecosystem, where you're taking provenance of code from a code repository, on the internet, insert a CI pipeline into production in under an hour. Now, your ecosystem, the context of operating from a dev and securing that operations perspective, in the in that context is much bigger than traditional teams used to worry about. And you're absolutely right about the proliferation of network traffic associated with API's. I saw a stat the other day, I think now 83% of network traffic is AP is API communication on the internet. So why APIs are the new frontier for security and achieving that security paradigm we talked about speed as the new security. So, this all may seem very scary, but really being able to take the speed at which you can operate to your advantage when there's a new vulnerability. I'll give them an example. We used to get CVEs and for the legacy AOC baseline. You know, from the time we had a patch available for a known cat one CVE. We would we would our legacy systems or manually built we would patch hundreds of servers and enter in a dev and test environment, make sure that we didn't break any of the legacy system and then push that patch to 22 locations. worldwide in end to end, patching of our of our systems will take around five months. Now, a CVE comes in all of a sudden, that is the next most important thing for everybody in the organization to fix, including the application developers, because the way that we tool, the CI pipelines, if the security tooling in the pipeline sees that there is now a problem, your code come in, can't go into production. So, application developers have to stop what they were doing from a functional application development perspective and more the security issue. And everybody goes front of mine to solving the security challenge and upping the security posture in ops. So that we were having response times to see me ease then in single digit hours as opposed to months or years at a time.
Craig Bowman: Interesting, any thoughts on? Looking back at that major breach at, you know, what does the future look like for dev SEC ops, when that is the new attack vector for our adversaries.
Alex Barbato: I mean, the irony is, is that those that aren't listening, or that that aren't listening right in this very moment, or aren't astutely putting together what you just said, to infer to like one industry event, like there was a really, really big, open source supply chain vulnerability that just happened. That is debatably, even bigger than what I think you may have just been reformed.
Craig Bowman: Yeah. I wasn't even referring to now.
Alex Barbato: There are, like that ship sailed like we there is a dependency, there's a problem in in our entire supply. Not a problem, per se, but like, there are vulnerable, there are risks and potential areas of attack, and introducing like malicious intent along the entire supply chain of how software is built. That is the world that we signed up for when we said that we believe in an open web that is built upon open standards. And we you know, have this modular system, look at Kubernetes. Right? It's what we live in now. So, the trick is, how do you minimize those surface areas to allow them to be true building blocks, right? And allow you to focus on that modular layer? So, if we talk about the attack that you're looking at, like, if thinking just about how I do air gap delivery and creating flame frameworks that actually create a bit more like be like six door right or I mean, not getting too technical. But there are like many efforts around how do we actually secure the process of going from raw code in but then there's like, what did I just pull into that code? Where did that come from? Right. So, it's all of those layers. But it's about constantly creating incentives to assess those risks and make sure you're not just following a playbook that says, you know, well, it told me, I can't have this port open on this piece of hardware. And you're like, wait, wait, wait, wait. I'm in the cloud. But he talked about, like, what am I even assessing right now? Like, I gotta look at what's relevant to me today, and have incentives to because back to your sock question to pull it all the way back there. I fully believe those SOC professionals, there are a number of very, very smart individuals that want to be moving faster and want to have leaner, leaner behaviors, right? And be able to more accurately implement changes, right? But we have to create the systems that allow them to behave that way, and get constant feedback, right? We so often look at that end user application developer is making feedback loops. But now we're in a world where there are many platforms that you're interacting with, to achieve that one outcome. Right. So, we have to behave that way.
Craig Bowman: Yeah, really interesting. controversial question here. Jeremy ready for it? How far should we centralize DevSecOps? Right. So, there's all kinds of efforts right, some of them centralized deeply, some of them centralized. What is the right amount of centralization of DevSecOps that makes sense to you?
Jeremiah Sanders: So, I tend to think about that. In a simplistic way, think about it from an infrastructure stack perspective. When you look at the complexity of running infrastructure and platform and the security operations and CI tooling. I don't think every organization in the in the government needs to do that, nor will they have the human capacities or the funding to do that. It's an expensive and resource demanding proposition to build and deliver consistent virtual consistency across potentially many different networks or security domains on the infrastructure, the platform, the CI tooling, and the operations there of those layers, when you look at the Mission applications, or the business applications, which we really care about in terms of delivering outcomes to users, that I think should still stay relatively decentralized, because we need the ownership of those application teams to, to have continual connection with and a sense of service or servitude to an empathy with their users or constituents. So, I don't think that there should be like one central application development house to rule them all. And not think about the context of the applications that are being built that can still largely be decentralized. And then I think a few powerhouse organizations should be running infrastructure and platform and the associated CI/CD tooling for potentially many, many government organizations writ large. And that's the, I think, ultimately, at the end of the day, it is less costly to do it that way. You know, getting the talent pool needed to operate at that scale, in truly support day to DevSecOps. There's a It's not infinite, but there's an economies of scale, that's fairly tremendous when you get a homing platform and infrastructure organization that provides the collaboration and developer tooling for many, many application teams that can be from dozens of different organizations, leveraging this underlying capability as a service is the most cost effective, and least risk way to do it. If we want to improve the security posture of government networks and application operations, do it that way, centralize the kind of bottom two thirds of the tech stack and decentralized application development.
Craig Bowman: Very, very interesting. Well, I'll tell you what, go ahead, Alex,
Alex Barbato: I just want to briefly like chime in for any of our leaders out there that like might hear this, because I tend to agree with Jeremiah. But there's a word of caution that I hope that you'll hear in that funding is hard. The apparatus in which the like federal government operates, makes what I'm about to say very difficult. So that's my disclaimer, right? Check the box, it's done. I know what I'm about to say is cheap, no tip. So, if you have to say, thou shalt use this platform, you've already lost like it's over yet. Like it's done, you lost. You didn't get it? Because you're further creating that system that says, we just like we build it this way. This is what we do. And it's like, no, no, the right thing should be easy. It should be desired. And people should be banging at the door asking for it right? Again, funding makes it hard, etc., etc. But please build it. They like and don't build it. And they will come model right. Build it with them. And they will want to keep going with you.
Jeremiah Sanders: Yeah, you're absolutely right, Alex, I could not agree with you more platform has to be treated as a product. And it should be iterated upon and build along with the application teams that need to use it. And in our case, that was one of the things from the get-go. We did not want to build a platform that we forced people to use. We onboard a dozen other acquisition and operational organizations onto our platform at Kessel Run. And they chose to use it. We didn't force them; they chose to use it. Much like the end users were choosing to use the applications we were building, we didn't force those eaters. And the choice was because we made their lives easier. Whether it was users of the applications are the users of the platform, which you know, in its own sense is its own product. We didn't force it upon anybody. It made people's lives easier, and they chose to use it. That shouldn't be the model that drives I'm glad you brought that up.
Alex Barbato: Tell them they got 30 controls to do, and they'll say I want to see whatever that is. Sign me up. I'll be there.
Jeremiah Sanders: Now the other 800 controls are taken care of. Yeah.
Craig Bowman: Alex, that's a really great tool. Want to end on? Let me just say, as we wrap things up, let me just throw it up to either one of you any last words for government leaders that are listening, they're thinking about or trying to set up their own software factory, there's still a lot of them. Any last words of advice by the one of you, and then we'll wrap up this podcast?
Jeremiah Sanders: I would say the, the government has a great opportunity to learn from commercial the commercial sector and from organizations in the government that have been successful in adopting the models of modern application development and running of cloud-based infrastructures that allow for continuous delivery and better mission outcomes. And you don't have to do it on your own. The success that my government organization had, because to be very blunt, whether it was my military workforce, government, civilians, or the traditional system integrators I worked with, none of us had the wherewithal to embrace and leverage the cloud technologies we were using, or the practices and ways of modern app development. So, it was an on-the-job training experience that we went through, in and that was a big part of why I came to VMware to further that growth pattern in in the public sector, and leverage that to the best extent that you can to be successful.
Craig Bowman: Fantastic. Well, Jeremiah, Alex, thank you both for joining us on mission first. Thank you for all that are listening. In our next podcast we're gonna be having, we will have Ken Bible, the CISO from DHS actually sitting down with the CISO of VMware. And they're going to be talking about what it means to enable a remote workforce during a pandemic. What were some of the challenges that the that the government faced, and that commercial company faced in trying to enable the mission of both while still keeping the environment secure. So, I'm Craig Bowman, your host, thanks for joining and we'll see you on the next mission first podcast. See you guys.