One of the biggest vulnerabilities state and local governments face is the huge installed base of legacy hardware and software. They are very expensive to maintain, draining resources that could be used to offer new, internet-based services. They often are riddled with security flaws, from bad code to unsecured ports –it’s next to impossible to keep up with all the patches that need to be implemented.
Speaker 1: On behalf of FedInsider and Carahsoft, we would like to welcome you to our mini-series headlines in cybersecurity, which aims to translate the years hot buttons cybersecurity news stories into actionable steps state and local governments can take to protect themselves from attacks and recover when disaster strikes. Today’s podcast brought to you by CyCognito is focused around the modernization of legacy security systems journalist John Breeden will moderate as Kevin Walsh, Director of IoT and cybersecurity at the US Government Accountability Office in Steven Cates, Senior Director of solutions architecture at CyCognito. Discuss barriers to updating legacy systems, the challenges to improving security for them, the possible role of the federal technology modernization fund in making changes and how integration of new technologies can provide more resilience.
John Breeden: Hello everybody. Thank you for joining us. I'm John Breeden and I will be moderating what I know will be an interesting and lively discussion about modernizing legacy systems in government. Today, we're going to be talking about legacy systems in government, how do you define them, how do you evaluate them, and how do you know when it's time to replace and modernize them. So it's both a critical and a highly technical topic, but have no fear, because we have two of the leading experts in this field to help break down everything for us. So let's meet them and then we can get started. I wanted to extend a warm welcome to Kevin Walsh. He is the Director of Information Technology and the cybersecurity team at the Government Accountability Office. Kevin, I know you have been studying legacy systems in government for a long time. And it's an honor to have you on the show to talk with us today about this topic and some of the GAO's findings in this area.
Kevin Walsh: Thank you. Good to be here today.
John Breeden: And I'm also pleased to welcome Steven Cates, the Senior Director of solutions architecture with CyCognito. Thank you for taking the time to talk with us today about this critically important topic, Steven.
Steven Cates: Thank you happy to be here. Excited for the conversation.
John Breeden: Great. So thank you again, both for being here. I do want to dive into this topic because it is so critical in government. But you too both have such impressive backgrounds. I thought maybe we should let you introduce yourself to the audience a little bit so that people can get to know you before you start dispensing your very sage wisdom. So Kevin, let's start with you on this. Can you tell us a little bit about your background in government? And how long have you been the director of the IT and cybersecurity team at the GAO?
Kevin Walsh: Sure. First, thank you for the kind words. And the invite today I have been a director on GAO's information technology and cybersecurity team for just a little bit over a year. So still learning the ropes, but have been in this area of work since well, as the slide says June 20 2006. I grew up in in the GAO doing work on OMB it dashboard, which eventually morphed into heavy work on photogra and close coordination with the US Oversight and Government Reform Committee as well as the Senate Homeland Security and Governmental Affairs Committees. I really, really like talking about the legacy system stuff as well. It's been great and fascinating topic. And it is especially critical and important in this day and age as demonstrated by recent events. So looking forward to talking about it today.
John Breeden: Great, thank you so much, Kevin. And Steven CyCognito has gotten a really great reputation for examining networks and helping agencies and other organizations plug their security holes before the hackers can get to them. Can you tell us a little bit about your background and how you became the Senior Director for solutions architecture at such an impressive company?
Steven Cates: Yeah, absolutely. So most recently, before CyCognito, I've spent close to a decade working in the attack surface management space, even before it really had a name helping to develop and define what that meant for organizations and how they actually go about implementing programs to protect their attack surface. Prior to that I spent a number of years in various consulting roles at large resellers and spent some time working for the FDIC and doing investigations with FDIC FDI during the financial crisis back about 15 years ago.
John Breeden: Excellent. Well, thank you. Thank you both for sharing those impressive backgrounds. So many of our audience members today are tuning in from state and local governments and since many of them are of course dealing with legacy systems in their organizations. Kevin, since you're on the federal side, some of our audience might not be completely familiar with the GAO and its role. Could you maybe briefly let us know what the GAO is mission is and why you're putting together all these reports?
Kevin Walsh: Sure. So when explaining this to the lay person, I would say that GAO is roughly equivalent to a congressional investigator. So we are hyper nonpartisan, which is rare in this day and age. Most of our work is done at the behest of Congress. So we take requests from both sides work together to make it bipartisan, and then do our do our investigations and work for them. So we are fact based organization, again, nonpartisan, and we are following basically wherever the federal dollar goes. So anywhere that receives federal government money, we have the power to go in and audit and work with the associated agencies. Most of my work, again, is done with the largest federal agencies, but I hope that we can share some parallels with state and local governments as well. Looking forward to it.
John Breeden: Absolutely. And most of your work gets initiated because of a direct ask from Congress. Correct?
Kevin Walsh: That is correct. We do occasionally startups, some work our own volition. For example, a few years back with Hurricane Katrina, everybody wanted to request it. So rather than pick a favorite, we started up that work in our own behest. But you are spot on our work is at the behest of Congressman comes from a direct request from them. Excellent.
John Breeden: Well, thank you. And Steven, just so we know where you're coming from in terms of cybersecurity as it relates to legacy systems today, can you tell us a little bit about what CyCognito does, and maybe some of the government programs that you're particularly proud to have worked on in the past?
Steven Cates: As a platform, CyCognito, we focus on what we call attack surface protection. For those of you familiar with attack surface management, typically, that runs the gamut of understanding and discovering what your external attack surface looks like, and the problems that may exist with it, we have taken the platform a step further, to actually test those assets to understand if there's a weakness, should you be concerned with it give you the evidence that we were able to exploit it. But then with that, most importantly, factoring in what is the path of least resistance for an attacker because we live in a world where the attackers have all the time in the world to try to find the one hole that you weren't, you didn't plug because you didn't know about it. So we focus on that 100% visibility of everything that's externally facing, testing it all, giving you the results back that are actionable, and the most important things that need to be fixed.
John Breeden: Makesa lot of sense. Thank you both for that overview. So let's begin with the talk about modernization. And, Kevin, I want to begin with you because I want to ask you about the report labeled GAO 19-471, which was an older report, where you looked at legacy systems in the federal government, and came up with a lot of interesting criteria about how to define legacy systems and when to begin modernizing them. Some people may not know this, but that was actually the second report. The first one was actually GAO 16-468. I think I got that, right. Yes. And it caused a bit of a stir when it was originally released. Can you bring us up to speed about the findings of the first report and why it made so many headlines around the world?
Kevin Walsh: Sure. So the first report, and you're right, man, what a what a, what a crazy report, that was what caused the headlines was we found and highlighted some of the oldest systems in the government. And most notably, we found an old DoD system that was still using eight inch floppy disks. And this is, you know, this report came out in 2016. So this is this is decades and decades old. Now. This, of course, caused all kinds of alarm. But to do these credit, this was a backup to a backup system. Now, the flipside of that, of course, is it was used for nuclear command and control. So there was some mitigation, they're in terms of, hey, this was this was, you know, a tertiary backup. But on the other hand, hey, this is this is really, really important. So, again, we looked in that report at some of the oldest systems. But as we as we did our due diligence and follow up, we realized we could have asked better questions, in particular, not only getting to what the older systems are, but again to that criticality idea, what are the most critical older systems in the government? Because let's face it, if you're running a print server in the basement, who cares, right? It's what's the really important stuff that we really want to highlight and to focus congressional attention and oversight onto so that's more of what the second report dug into. And I think we're gonna we're gonna talk a little bit more in a few minutes. But that report not only talked through one of the one of the oldest, most critical legacy systems, but we also got the agencies talking about what they considered or what they were thinking about as they were comparing and evaluating legacy systems, as well as the impacts of not modern icing. So I don't know how much you want to jump into that right now. But those are some of the big things that came out of that 19-471 report, in addition to a more nuanced understanding of does legacy always mean risk. The best thing that I can tell people is just because something's old doesn't mean it's broken. And the word legacy is a very, very heavily weighted word. But there is no real good definition of that. In fact, in a, in a congressional hearing, just last week, the Federal CIO, Claire Martorana, and for anyone who wants to stalk if this was before, his gap, Homeland Security and Government Affairs Committee on the Senate side on September 28, she was asked about legacy systems. And in her opening statement, she said that legacy is not all old systems. So all old systems are not necessarily legacy systems. And she acknowledged the lack of a good definition. But she said that she was most concerned with the systems that that met, and I counted them up, there are six things she's concerned with. The first was those that are out of support. Second is they can't be patched. The third is they have availability issues. Fourth, not meeting usable user needs. Fifth can't be policy goals. And the six is those without data security, and I liked these, these items that she was concerned with, I wholeheartedly agree with them. But again, it underscores that that idea that legacy is this term that we use, and we throw around all the time, but it means different things to different people. So when you're talking about make sure that everybody's on the same page.
John Breeden: Good advice and we're gonna definitely dive into a lot of that in just a bit. Steven, along the lines of some of the things that Kevin was saying, it occurs to me that, you know, CyCognito gets to look at a lot of government public sector networks. So in general, with the caveat that the term legacy systems is kind of nebulous, but when you look at systems like that, do you find a lot of legacy systems? And do they tend to cluster or silo up within certain areas? I mean, I've heard that they kind of kind of grow in stay in finance systems for a long time, for instance.
Steven Cates: Yeah, I see it all over the board for both sides of that. But you know, very much agreeing with what Kevin's saying with the legacy being, you know, toughen, what you're calling that I look at things more from an important factor of when the system is found, that is old, outdated, hasn't been patched, looks like it's abandoned, any of those kinds of things, we start to pile on with that, what are the issues that exists there. And then beyond that, what's the attractiveness to attacker because we classify that system so if it's something like an old marketing web page that was stood up on a patchy several years ago, and was never taken down, it was just abandon traditional scanning tools are going to tell you, you need to pay attention to this, it has a lot of problems. And it's going to have all the attention drawn there. Whereas something like a payment card mechanism on another IP address may have very few findings or none, and not look like it needs attention. The CyCognito approach that is actually the opposite. Because we factor in what the attractiveness is to attacker hacker doesn't care about the marketing website that touches nothing, they care a lot about the payment card mechanism. And one issue existing with the payment card mechanism means that the inroad they're going to take in, whereas that legacy of ASCII system, you don't necessarily need to worry about it goes very low down and level of risk. But that's hard to do for an organization unless they understand what every system is, because then you're getting into a very risky situation of saying something is low risk, without really knowing that it's low risk. So that's the big part we come in is try to help to grade those and guide where the efforts need to be put, and make it easy for an organization to actually see improvements in their security posture by removing the path of least resistance of where an attacker is gonna come in.
John Breeden: That makes a lot of sense. Thank you. So Kevin, I kind of wanted to ask you directly to follow up on something you were talking about not from the first report. So that DoD system that was still using the eight inch floppy drives in that made news and headlines and everything. So it was splashy. And it certainly did put a lot of people's focus on what is a legacy system. But technically, and the reason I'm asking you this is because I know you use this to change your criteria for the second report. Technically, it wasn't necessarily a bad thing that that particular system, regardless of what it was responsible for was an operation, right?
Kevin Walsh: Right. I mean, this is difficult, right? Because, on the one hand, older systems like that are going to be increasingly expensive to maintain. Right? You know, how easy would it be to get even a replacement eight inch floppy disk? But conversely, how easy would it be for bad actors to even replicate it as well. So it's a tradeoff, you know, you, it's not necessarily a bad thing to have these old systems. But we at least want people to be aware of them, as Steven was saying, you don't want something just abandoned, somewhere sitting there. And that that acts as a point of vulnerability. We want people to be aware of them, thinking about them, and revisiting that discussion and decision so that it just doesn't sit there for another decade, you know, it should be part of a regular discussion of good IT management maintenance.
John Breeden: Makes sense. And Steven, continuing on your thought from before about legacy systems and how payment system, which isn't a legacy system might actually cause more of a concern. But in general, when you look at a public sector network for security vulnerabilities, and you find legacy systems, do you mean, you don't always necessarily flag them as a high risk, just because they're a legacy system, you have to look at other factors as well?
Steven Cates: Yeah, and our standard scoring model, which is fully adjustable for what our client wants to score things as, but by default, the way that we look at it as attackers ourselves would be that that's something that I'm not going to bother trying to attack, because it's hard to find it, there's nothing attractive behind, it doesn't act as any other systems, there's no critical information I can get my hands on. All those come into scale to say, you know, this isn't really something that an attacker is likely going to attack us. And we say, focus your energy, focus your efforts and your time on things that really are going to be attacked and will be breached, and will cause very large issues for you. So we don't want everything to come out is just equal, every issue is every issue, those legacy systems can be critical. And by all means those could be but they also could be something that was abandoned and doesn't really have an impact if it gets breached. So we try to look at it from all those angles to use a smarter, a smarter grading and scoring system around security and prioritization of assets and issues and how those are related.
Kevin Walsh: Excellent. Okay. Well, thank you just one nuance there that I agree with Steven, but not every attacker are the same. And so agreed that most of the you know, the hacks that you see are going to be after those car payment systems. But government entities would have potentially different bad actors coming after them that aren't just after cards, or after information or even control surfaces. So, you know, while I agree that the script kiddies that are coming in looking for the card payment processing systems, absolutely, they're not going to care about the old nuclear command and control. But the other nations, yeah, actors might really be interested in it.
Steven Cates: So actually that a system that is a any kind of SCADA system command control system that's identified, that is scored differently as a legacy system than just a marketing server. So we do look at what's the actual devices being used for, and classify that. And that's what we understand the attractiveness. So there were there would be a difference there. But then also, based off of the industry, vertical of our customers, we do adjust as well, because, you know, to some people, that marketing website may be really important. And it's just not the most. So that's right. It's not a one size fits all that we give you. We start with a guidance point, and adjust from there based off of our conversations with what our customers actually want to accomplish.
Kevin Walsh: And to the best of my knowledge, there are no state and local governments that do have nuclear weapons. But you know, just wanted to put that in there. So, thank you for that.
Steven Cates: Yeah.
John Breeden: Thank you all for really getting into it. I appreciate your thoughts on this. So Kevin, let's move on to the second report, which is kind of the one that we're going to be talking about, mostly today, the original report, if nothing else, put legacy systems serving within government in the forefront of people's minds. But in some ways, as you've mentioned before, it was kind of unfair on some of the systems that were called out, which is why in the newest GAO report, you modified the criteria quite a bit, actually. So Kevin, with that in mind, how did you configure the criteria for the next report, which is Gao 19-471 to better define legacy systems in the federal government and their impact?
Kevin Walsh: Sure. So rather than going out and trying to tell agencies what legacy is and is not, we deferred to them. We went out NASA, the 24 CFO act agencies, which are the basically the biggest agencies out there. So think of Department of X, and you've got them and ask them, Hey, what are your most important legacy systems and we asked for three, some gave us more, some gave us less. And we wound up with a list of, I believe 65 different systems that they had flagged. And we asked for a whole bunch of metadata about these systems, system age old age of the oldest hardware, how critical they felt the system was how risky they felt the system was age of the old software, operating system, all kinds of different metadata about these systems. And we, we use those data ourselves to score and rank the systems based on what we thought were the most important. And what we wound up with, which is a little bit too small to see on the screen was a list of 10 of the systems that we thought GAO thought were the most critical legacy systems in need of modernization. And it wound up across basically 10 different agencies. So there, nobody doubled up, nobody had to have the most important systems. But we also were very, very careful, because we didn't want to create a target list for any bad actors. None of these systems in this report have any identifying information. So it doesn't say the name. Now, it does have descriptions in the appendices of the systems. And if you're familiar enough with any of these systems, you probably have a very good idea of which these systems are. But at that point, if you're familiar enough with the systems to identify them, you're also familiar enough with the problems of those systems. So it's not really a sensitivity issue at that point. But as you're, as you're reading through this report, some of the most fascinating stuff is for once in Energy Report, not just in the body, but also the appendices where we list out some information about the 65 systems. And then for the top 10, we profile each of the 10. And it's some it's some great stuff. And perhaps one of the most fascinating parts is we asked agencies, how much some of these systems would cost to modernize whether there would be any cost savings, because oftentimes when legislators are thinking about this, they just see, okay, Legacy system, we need to modernize it, and they see dollar signs and cost savings. But one of the one of the best examples in there is a system at I believe it was the IRS and the Department of Treasury, which had reported annual operating costs of I believe, about 16 million, which is you could probably hear him flipping through there right now. So about 16 billion in annual operating costs. And then to modernize it, it was going to cost more than a billion dollars. So oh my god, right? So if you're looking at like, okay, we're spending $16 million a year, but it's gonna cost $1.6 billion to modernize? Well, if you're just looking at a cost benefit, that doesn't make sense, right, you're going to take 100 years to pay that off. Okay. But when you're thinking about these legacy systems, and again, this is where it's just shades of grey, you're not just thinking about cost of monetization, you're thinking about better services to users, minimizing the attack surface, that Steven's been talking about. So it's a fascinating discussion. And it's, it's not always just black and white.
John Breeden: Excellent. No, thank you, Kevin, let me ask directly for our state and local audience. So when you were defining the terms of, you know, what makes a legacy system? Can you go over, you know, kind of what your thought process was some thinking like for some of our state and local people who are who are following along today, you know, when they're looking at their own systems, what should they look at? What factors should be the most important ones that they consider when they're saying, Okay, this is a legacy system, this may not be technically a legacy system yet?
Kevin Walsh: Absolutely. In very early on in the report, which and to your earlier point, I really acknowledged that GAO is some dry reading normally, so acknowledge the point, the things that we thought agencies, while the things that agencies told us that they consider when thinking through legacy systems is, how, how risky Do they feel it is? So how exposed is it? The criticality of the system? Again, that print server in the basement that nobody cares about, versus the card processing systems? You know, how important is it to the operations of your organization? The cost, not only how much it costs, you know, for software licenses and the like, but how much are you paying for salaries? How much are you paying to support the system? Is it getting harder and harder to find the necessary talent to support the system, and then performance if it's still humming along just fine, but you're getting additional, you know, legislative requirements every year that the state legislator or the General Assembly keeps on putting new requirements on you, and it's getting increasingly difficult to add them on or to bolt on those extra little pieces, then that right there might be might be cause, you know, for you to modify. So those are the four main considerations that we identified. And then the flip side of the coin, of course, are the four impacts which a lot of times when talking to others is better to talk about, well, if we don't do this, then here's what we could, we could experience and the two are length, you know, the risks, hey, increased risks of being hacked or data breaches to like, or failing to meet the mission, which is the second one mission performance, you know, are you going to meet your mission? Or is it going to prevent you from doing that? Third thing, again, staffing, if you are having problems finding the right people to do this or to maintain it, that's going to be an impact, you know, it's going to be harder and harder to find the right thing we identified in the report, the Department of Treasury, and IRS was hiring back retirees, rather than rent because they couldn't hire anyone out of school, no one was trained and knew how to do some of these programs that they were trying to use. So they hired back retirees, and I'm sure they made a pretty penny doing it finally, is costs, which again, all of these things are linked and related. As systems get older, they get more and more expensive to maintain, and to upkeep and to update. If you have a system that is no longer supported, you know, keeping it locked down by yourself is going to be more and more expensive and more and more difficult and cost more and more money. So the impact of not modernizing can really be tracked back to those four things risks, mission, performance, staffing, and costs.
John Breeden: Excellent. And I'm glad you brought up staffing, because that's an interesting, one of our audience members was asking about maybe putting a big firewall or something like that around like extra protections around a legacy system, which might work and it may help to eliminate some of your security issues. But if you're still hiring, you know, retirees to run the thing, then that's kind of a cost that you have to consider as well. So very interesting, Kevin.
Kevin Walsh: On that topic, you know that adding a black box or even air gapping systems, if you are facing a dedicated, persistent, bad actor, even that won't protect you. You know, if you if you look at Stuxnet and what happened to Iran and their nuclear processing facilities, that was totally air gapped. It was not connected to the internet, and it was still hacked. So you know, there's only so much you can do. It's not a matter of if you're going to get hacked. It's always a matter of when and hopefully, I'm not Steven, stealing Steven's thunder here. But yeah, putting a black box around them. It can help. It's a preventative stuff. But don't let that be the only thing.
John Breeden: Make sense. Well, Steven, let me ask you to follow on to what Kevin was saying. He laid out what how a legacy system is defined for government, I thought he did a really good job. But is there anything that you want to add? I know, you and CyCognito get to look at, you know how all these systems interact with each other. I've seen your maps before. They're very impressive. So how would you help state and local customers identify legacy systems?
Steven Cates: Yeah, so it gets into that tough definition of legacy again, but just some statistics, kind of, we're starting to go there, Kevin. But looking at like, we've found that 67% of organizations have been attacked by a blind spot, with blind spot oftentimes does overlap with legacy. And then 75% of them expected happen again, we find that most organizations have visibility into 50 to 70% of their attack surface. And that's it. Again, that speaks to large parts, the legacy of what we find that they didn't know about, usually my conversations people go like, Oh, I didn't even know that was still there. I don't know what that is, they look it up and say, oh, that was some from 10 years ago, we don't know why it's there. So I would say more. So on my side, it's identifying everything. And then the goals that people organizations start with CyCognito is reducing their risk overall. And reducing that risk can come out of doing something with legacy systems, whether that be putting in new compensating controls, spending the money to upgrade them, or migrating them somewhere else, whatever it may be. But we start to rank together those same issues that occur over and over and over again, on multiple different systems, or one system that has a lot of issues, and can help an organization to understand if you took care of these things. What was your security posture look like? And a big thing that I work with on customers. Honestly, most everybody that comes in they’re at risk of attack risk, when they start. I think I've seen one ever that was not enough. So I say where you start your journey is, yeah, try to get away from an F and get to a D. That's improvement. And that may involve a lot of legacy systems, but it may not as well. So that's that's the tough thing because of where legacy is defined. And then also from the outside. Some of those legacy systems, we never see them at all, from the outside scanning. So we don't even know that they're there. And if we don't know that they're there, attacker may be able to find them if they're really good and can recon and get into the network. And then do some more recon, we focus on the recon from the outside entirely. So if it's not exposed to us, then we also can tell you that. Do you know this legacy systems there, but it's not exposed from the outside? Whitely exposed anyway, so.
John Breeden: No, it's is really good advice. And I like getting from the F to the D. I mean, and how much of an improvement that is for a moment, though you sounded like my high school algebra teacher. But yeah, I guess it's a constant. Thank you. So let's talk about the findings of the report. Kevin, why don't you and I think this is going to help our state and local audience to kind of get an idea of the kinds of systems that you thought were legacy and, you know, affected in government? So tell us about the top 10 systems that were identified in the report and some of the things that they had in common.
Kevin Walsh: So the federal government well, to manage expectations, in terms of commonality, these are all going to be relatively different. The federal government just has such a wide range of responsibilities and scope. But well, I'll highlight for you. So we highlighted one system at the Department of Defense. And in particular, it was at I believe, the Air Force, which they use to configure and control this management sport to wartime readiness. So of operational aircraft, excuse me. So basically, this is a behind the scenes system that they use to track how ready jets are to fly. Okay. So, again, very important, it's not anything that we're ever going to see by going to dod.gov. But it keeps track of the maintenance requirements and how different aircraft are configured. So pretty cool. Pretty important. It was in terms of annual operating costs about $22 million per year, cost of modernization, 12 million, and I believe, since they, since this report was issued, I think DoD has actually modernized this one. So on an annual basis, they were spending 22 million, the annual labor costs were about 3.6 million cost to modernize about 12 million. And they reportedly saved I think, on the order of 20 to $30 million per year. So this is one of those, you know, rare, rare instances where it made not only a lot of sense to modernize, but they even save money to do it. My other favorite ones to highlight one was at the Department of Interior. This was a system and to use the term Steven used earlier, this was a SCADA system. So supervisory control and data acquisition SCADA. So this system controlled dams and power plants. Now, it was older, and I believe they had it air gapped, I'm not positive on that. So take this with a grain of salt. But basically, if this system went down, I think catastrophic releases from a dam that had, you know, not small cities downriver from it. Annual operating costs, again, of this one is $400,000. So this is not all that expensive to run. So costs, you know, in that in that matrix of the four things to consider, you know, risk, criticality cost and performance, this one really wouldn't have much on cost. But in terms of criticality, and risks, well, this one, almost pegs the chart on it, right, because if you've got a catastrophic dam release, that's scary. The third one to highlight would be the Department of Treasury. That system I talked about earlier, it costs $16 million on an annual basis to run, the cost of modernize was 1.6 billion. And right on its face, hey, that doesn't sound like a good payback. But this system was initially implemented in 1968. Okay, and it is used to process taxes. Now, anything that that that's that old, you know, can you imagine what the Treasury and the IRS are having to do to get this system to accommodate all the changes to the tax code that have been made over the years. It's also written in assembly language code and COBOL. So finding programmers who are not just passingly familiar, but with COBOL, and assembler language, but able to write to the tax code in those languages, that's just going to skyrocket your costs. So this is one where you know, costs are going to, you know, the longer you procrastinate, the more it's going to cost you not only to monetize it, but on an annual operating basis as well. And perhaps modernizing it, you'd get a whole lot more functionality on a newer code base than you could on something that old. Finally, just a just another system to highlight is OPM told us that this should come as a shock for no one who was reading the news in 2015 or 2016. Seen that a lot of their IT was outdated that so the hardware and software and networking components which contributed to the breach. So in 2015, OPM record reported that massive, massive breach into their systems were basically the personnel records of 4.2 million employees current and former was exfiltrated. And so basically, OPM didn't have those data encrypted. Because the hardware and the software couldn't support it. If they had more modern hardware and software, it could have been encrypted at rest, which would have meant that the bad guys, whomever they may be, you know, even if they got it, it would have been much harder to do anything with it. But everything was stored in the clear. And so once they got it, game over, they were able to just make whatever uses they may have it. So that is four of the 10 systems that we profiled, and I'm not sharing anything sensitive these stories are in that report, the other six are there for your readers to digest at their pleasure.
John Breeden: Excellent. Well, no, I think you picked some excellent examples, especially the OPM one, I didn't realize that the OPM attack and breach could be directly attributed to older hardware. I mean, if the older hardware, and they had newer hardware in that case, then they could have maybe had encryption on it. So that's amazing to hear. I didn't know that.
Steven Cates: Yeah.
Kevin Walsh: I don't know that it would be directly attributed, but it was definitely exacerbated by it. Right?
John Breeden: Definitely. So Steven, along those lines, I mean, Kevin gave us some fantastic examples, when you're looking at legacy systems. And when you're talking to your customers, you know, you've identified systems that are kind of legacy, like, what are some of the common dangers that are associated with running legacy hardware, like in the OPM example?
Steven Cates: The legacy systems, I think that they really pose on the surface, they have the same probability of somebody finding a backdoor in or a way to bypass the code that that's gonna run the risk with anything legacy or not, when I think I factor in more when I look at legacy are things like older technologies and use like the encryption, I see a lot of systems that I would call legacy that are using the triple does encryption. And back when it was implemented, that was great. But now I can decrypt that with my cell phone. And that's a very different world we live in today, when we talk about the amount of compute power that people can crank through on GPU to try to crack passwords and hash is, it's very different than it was 5, 10, 20 years ago. And if the legacy system is still using the encryption technology, and storing passwords in what was considered a secure manner back then, that is leaving a wide open door for somebody to come in easily. I think those are big things that need to be considered.
John Breeden: Yeah, definitely. And Kevin brought up something that I want to follow up with you on Steven Kevin, in the of course, the report mentions this, a lot of financial things. So you know, a system costs this much, many millions to run, and it costs this much to modernize, and so forth. And some of the examples are pretty egregious. So you know, if you have a system that's costing you $20 million to run, and it'll cost a billion dollars to modernize, then maybe that's something that's kind of a non-starter. But are there times when an agency is not going to achieve a direct cost savings with a modernization program, people kind of think that they modernize, everything gets better, and their costs come down. But might there be a situation where it's simply a matter of security, and the cost might remain flat, or possibly maybe even increase? But they still should consider modernizing because of the criticality of the system?
Steven Cates: My opinion, yes, I understand why the decisions are made not to and that the risk is accepted. But it's one of the challenges of the world that we live in today. Because it's not very expensive, from an attacker standpoint, to exploit those older systems. And to be able to run an attack against them. They said, like the where we are today with Compute and GPU processing power is very different than it was 10 years ago. So it cost me you know, I've got a cracking rig that cost a couple $1,000 and does 10 billion cracks per second. That was unheard of five years ago. And so something that we would have thought have been saved, then it's not at all now and it's going to be extremely expensive to upgrade to protect against what hackers are capable of today. And then you also got to worry about what they're gonna what's gonna happen in five years. With the new advances in quantum computing and the ability to crack systems then at that point, it's going to change again. So why I don't know the answer to the question, I have my own opinion of what it should be. Because I think you should always consider how big of a risk something is and adjust accordingly. And I try to help customers to eliminate a legacy system that's just out there. And it's only going to be a matter of time before that is a problem. I point to those and say, those are the things you need to focus on getting rid of them. And maybe that means my maybe it's somebody need to take a new approach and migrate actually how that whole process is done. Is this something you can move to the cloud? Does it even need to be sitting on an eight inch floppies? Can this thing how much it's gonna cost to rewrite all this in a current modern language that we can get new contractors to fix it when we need to make updates? You know, what's the cost savings there that that we need to consider as well? There's so many factors that go into it. It's really tough, and not always being on the business side of that decision. I see it from the security side and say, Hey, guys, this needs to be done. But I get it when people come back and say, No, we're not going to you right now. I don't agree with it. But that's the way that that business and finance works. So I understand.
John Breeden: Yeah, no, it makes sense. And, Steven, that that kind of leads us into our next section about older systems. So I wanted to start with you on this one. So when you're working with customers, and you run across the legacy system, I mean, what are the recommendations that you saw that you often make? I mean, can some legacy systems be patched or have their security increased or is it almost always necessary to do like a full replacement?
Steven Cates: Yeah, some can be patched, some can have compensating controls put around them. There's technologies out there that do virtual patching as well, that you don't actually have to touch the OS at all, and can just block for that type of attack coming in and table but realize that and stop that attack. So there are other things that are out there technology stacks that can help with this kind of thing. It doesn't always mean that the system does need to be taken offline or migrated somewhere else. But that's always a conversation that I find that companies when I'm going to, we're looking at this. It's not even something that they have started having a conversation, this was a more often than not, I hope this is shocking to people. But when I find an old legacy system outstanding, and I'm talking with somebody who's a director or a CISO, and that I show it to them, they say, somebody lied to me, they said that that was offline. In real life, this, this should have been gone two years ago. And it's just they hid it from me. So that happens a lot too. And no, and nobody just don't take it offline. And once you've been told it's gone, you stopped looking for it and you stopped testing it, you're actually even in a worse position than you were when he knew about it.
John Breeden: That makes a lot of sense. So Kevin, following up a little bit on what Steven was saying, you kind of mentioned this a little bit before. But is it true that just because the system is old, it doesn't necessarily need to be replaced? Could you maybe give us some examples of where old is not necessarily bad?
Kevin Walsh: Yeah. So I mean, even the eight inch floppy disk that we've announced around about DoD nuclear command control, I wouldn't say that that was necessarily a bad thing. Right? There were advantages to having you know, this, this extra backup, where a if everything really, really hits the fan, you know, we've got this dusty old floppy disk sitting in a filing cabinet somewhere, we plug it into the archaic system. And, hey, we've still got a backup to the backup to the backup. Okay. But I don't think anyone who's familiar with the topic would say that just because it's old, means it's bad or it needs to be replaced. We want people to be having a thoughtful, deliberative discussion in the federal government, Adobe years ago, mandated operational analyses, which are supposed to be these annual discussions of, hey, let's talk through each of our systems and think through, is it time to call it quits? What do we need to do with this now? Sadly, for us, that's turned into? Well, I suspect it's turned into a rubber stamping exercise with Yes, keep it going to the next year or the next year and some agencies, we go and ask them about their operational analyses. And they say, are what, but the intent or the idea behind it, I think is still commendable, you know, think through and talk about all of these older systems on a regular basis. It doesn't have to be annual, but you know, however frequently you think works for your organization. Just have the conversation. And don't be afraid to say, you know, it's limping along but maybe this is the year we finally we finally do away with it and finally monetize it. Or hey, this one's this one's still limping along and I think we can we can nursing along another year. Now, this kind of gets to the idea of funding and how gov work in the federal government, you know, we get appropriations on an annual basis. And at the end of the year, there's all kinds of pressure to this money is going to go away if we don't spend it, which also doesn't lead to the to the best it decisions. And I'm sure state locals deal with the same exact kind of thing. But being prepared and thinking through these old systems will at least at least mean that when you get those, hey, we've got to spend it, you can go down your list and say, Hey, here's the next one that we prioritized. Or here's the one that we said we should do next. And so you've already thought about it, you've already had the conversations, you've already hopefully got your stakeholders on board, to have that discussion and to know what your priority is. So to sum up old doesn't necessarily bad. But old means you should really be thinking about it.
John Breeden: That makes sense. And I think that goes into a lot of the systems that were in your appendix. I mean, there were 64 systems in the appendix that were not listed in the top 10. So what you're saying, Kevin, is that, you know, some of those may be able to continue going along, but make sure that left alone does not mean left alone forever.
Kevin Walsh: Right. And the federal CIO, in her testimony last week, said that she would love if every one of the government systems was fully modernized and perfect and beautiful. But that wouldn't be how, you know, we, as a government have chosen to spend our money, right, we, as a taxpayer, I'm okay with some of these systems limping along, we don't need the latest hot rod supercomputer to run some of these systems and the people who are making these decisions to keep them going. I mean, how these are some tough decisions with scarce resources, so much sympathy to my colleagues at the state and local level, because it's not easy figuring out exactly what's going to limp along for another year, and what's gonna get the axe or, Hey, if we keep the system running, we've got to let go of full time, you know, employed? That's rough. So yeah, we have to make hard decisions in IT. And especially when dealing with these other systems, it all comes back to having the discussions, how critical is it? How risky is it? How much is it going to cost? And how's it doing on performance?
John Breeden: Excellent, well thank you both for talking with me today about legacy systems and government. It's been super interesting. So Kevin, the GAO was recently talking about the latest report in Congress. Although we are talking about the latest report, it is still two years old. At this point. Are there any plans for doing a third report or maybe even turning this into an ongoing series? Since, like you say the issue is so critical in government?
Kevin Walsh: Yeah, oh, man, I would love for this to turn into some sort of series, I don't think it would make sense to do it on an annual basis, you got to give agencies time to, to react and to adjust what you've, you know, recommended. But as you know, this is an important topic, we see the systems in the government getting older and older. And this government wide perspective is very, very informative and influential. So I would, I would love for there to be a third report this series.
John Breeden: If you did end up doing a third report, would you refine the methodologies again? I mean, you changed from the first report to the second report to make it more accurate. Is there anything that you would want to change again, to make the third report even more relevant for legacy systems and government?
Kevin Walsh: Looking into the crystal ball? I think we would, we would always want to keep evolving and keep growing. As their knowledge of the topic evolves, we're also seeing a lot of, you know, the government is shifting away from having their own hardware, even platforms, you know, would we, we factor in anything cloud related? Who knows, but I think a report using this, the same exact methodology wouldn't be the best service to our stakeholders or even the taxpayers. So I would want it to grow and evolve as we go. So perhaps.
John Breeden: Excellent. Well, thank you, Kevin. Appreciate that. And Steven, it looks like you get the last word today. We covered an awful lot of ground today, in a very short period of time. So to kind of wrap things up for our audience. Could you maybe reiterate the most important points from today? What are the key lessons that you want to make sure that our audience in state and local comes away with regarding looking and evaluating their legacy systems?
Steven Cates: Yeah, so I think the largest thing that I see is the internal fear of that system being a problem, and it disappears off the radar. So saying that when I'm doing early onboarding with customers, and identifying what their taxes, this looks like, we're finding those legacy systems that are out there, but we're not on anybody's list. Nobody's checking them. And unfortunately, they've missed the security controls too. And in cases where you have those systems that you have to leave them up for financial reasons for technologies and whatever reason might be you're going to accept the risk and keep on running. At least make sure that they're under the control of your security program, so that you can monitor them you know, when something goes wrong, you know, when there's a potential breach or issue, that's the first step one. And that isn't as costly as talking about migrating away from them and upgrading. But if you don't know about them, and their legacy, and they're easy to exploit, that is what an attacker is looking for. We refer to that on our side as the path of least resistance that attackers take. So first and foremost is finding those and understanding that they're there. And from just doing that, you've brought yourself up further in your security posture overall.
John Breeden: Makes sense. No, thank you so much. And thank you, Steven, and Kevin, for being with us today. This has been really amazing. All of your insights have been great. I learned a lot today. I'm sure our audience did as well.
Speaker 1: Thanks for listening. If you'd like more information on how Carahsoft or CyCognito can assist your state or local government agency, please visit www.carahsoft.com or email us at CyCognito@carahsoft.com. Thanks again for listening and have a great day.