Ransomware cyberattacks vaulted into public awareness with a vengeance when an East Coast gasoline pipeline shut down for days, causing a shortage and long lines at the gas pump when drivers panicked. But hospitals, schools and local governments have been dealing with ransomware threats for some time.
Speaker 1: On behalf of FedInsider and Carahsoft, we would like to welcome you to our mini-series headlines in cybersecurity, which aims to translate the years hot buttons cybersecurity news stories into actionable steps state and local governments can take to protect themselves from attacks and recover when disaster strikes. Today's podcast brought to you by Hewlett Packard Enterprise is focused around ransomware journalist John Breeden will moderate as Josh Leiling, Assistant Director of IT and cybersecurity at the US Government Accountability Office in James M. T. Morrison distinguished technologist in cybersecurity for the office of North America, CTO and Hewlett Packard Enterprise discuss their own efforts to combat ransomware and strengthen their system protections against it.
John Breeden: And hello, everybody. Thank you for joining us today. I'm John Breeden and I will be moderating what I know will be an interesting and lively discussion about the critical topic of ransomware. And how to best defend against those kinds of attacks, how to protect your critical data, and what to do in terms of resiliency should the worst happen. It's both a critical and a highly technical topic. But we have two of the leading experts in this field to help fully cover this topic for us. So let's meet them and then we can get started. I wanted to welcome a Josh Leiling. He is the Assistant Director of IT in cybersecurity at the Government Accountability Office. Josh, it's a real honor to have you with us on the show today to talk about ransomware.
Josh Leiling: It's great to be here, John. Thanks for having me.
John Breeden: Excellent. And I'm also proud to welcome James M. T. Morrison, a distinguished technologist for cybersecurity with the office of the North America, CTO of Hewlett Packard Enterprise. James, thank you for taking the time to talk with us about this important topic.
James M. T. Morrison: Thank you very much for having me, and I'm looking forward to the conversation.
John Breeden: Well, thank you both for being here. I do want to dive into this topic. But you both have such impressive backgrounds and credentials that I thought we should first let the audience get to know you a little bit better before we dive into the dark heart of ransomware. So Josh, let's begin with you today. Can you tell us a little bit about your background in government? And how long you've been the Assistant Director of IT and cybersecurity, at the GAO? And what are some of your current responsibilities at the agency?
Josh Leiling: Absolutely. Thank you for the question, John. I began my career in government in 2007 as an IT analysts at the Government Accountability Office, and over the years I've contributed to and led projects looking at ways to improve major IT acquisitions such as farm loan systems and weather satellites have also looked at addressing major IT management challenges such as cost transparency and workforce planning, as well as projects related to enhancing cybersecurity across the federal government. And virtually I've looked at all major federal agencies and their sub components in some way, shape, or form. Over the years, for past three and a half years, I've had the pleasure of serving as an assistant director in our cybersecurity function, where I lead about anywhere from four to five ongoing project teams. And also I provide subject matter support to other mission teams within the GAO, for example, we have a Natural Resources and Environment team that does work at the Department of Energy. And we have a defense capabilities and management team that does a lot of work with the DoD. And so those projects often require some technical assistance in navigating it and cybersecurity related issues in their audits.
John Breeden: Okay. Well, thank you so much, Josh. That sounds really interesting. And James, we also appreciate you being with us here today. Can you tell us a little bit about your background and how you became the distinguished technologist for cybersecurity with the office of the CTO at Hewlett Packard Enterprise?
James M. T. Morrison: So I've been working for HP about two years now, prior to that, I worked 22 years from the FBI out of the Houston and Albuquerque offices, working cybercrime cases, you know, across, you know, across the Southeast Texas area in particular, worked a lot of, you know, national security criminal cases was sort of, you know, instrumental, you know, part of that group that watching cybercrime for the rise up out of its infancy in the 2013 2014 year, in the beginning of ransomware. In particular, reverse engineering malware was one of my skill sets. And so, one of the things we did early on was dissect ransomware and then attempt to track it back to its origins and work those kinds of cases with HPE I've been working with across our industry, both from a federal sector as well as with the SLED group So, you know, state, local, and education to improve our offering. And then also to start to have more of those, you know, one on one security conversation, where are we going, you know, what kind of transformations are groups trying to make, and then being sort of that trusted security adviser across North America. So that's, that's been sort of my goal is to, you know, really just not be as much, you know, product driven, but more, you know, services driven and, and HP is really starting to stand up more of that accounting, or professional services, and security services organization to do exactly that.
John Breeden: Makes a lot of sense. And one impressive background, James, thank you. So thank you both for sharing those, those background stories. That's very, very impressive. You have great credentials for the show today. So many of our audience members are tuning in from state and local governments today where they are dealing with the problems associated with ransomware. In many ways, they're very squarely in the crosshairs. So we will see if we can give them some good advice and help during today's show. Josh, just before we jump into that, and since the GAO is a federal agency, and most of our guests are coming from state and local governments today, some people may not be completely familiar with the GAO and its role. Could you maybe briefly explain what your agency's mission is, and the kinds of things that you research and report on for Congress?
Josh Leiling: Sure, I'd be happy to John, thanks for the question. So the US Government Accountability Office, the GAO, we're a nonpartisan legislative branch agency that provides Congress and the heads of executive agencies and the public with timely fact based information that can be used to improve government and save taxpayers billions of dollars. So from the IT side, you know, we do a lot of reviews of major IT acquisitions and, and looking at things like modernization of legacy systems and the like. And what we'll do is we'll go in and, you know, similar to IGs, within a state or, like, your internal auditor, you know, will identify potential issues and make recommendations that agencies can, can decide to implement and hopefully improve their operations, we've got a pretty good track record with, you know, on average 80%, or more of our recommendations getting implemented, and each year, we identify, you know, upwards of 50 billion or more in potential financial benefits. And that doesn't even include the number of non-financial benefits that we provide through our cybersecurity related recommendations.
John Breeden: And most of the work that you do comes because of a direct request from the US Congress.
Josh Leiling: That's right. So we get our requests, typically from committees of Congress to maintain that bipartisan nature or non-partisan nature of the work. So we do occasionally do work for a member if, if they have a particular area of interest, but we prioritize generally, the requests coming from a full committee or subcommittee, we also initiate a small percentage of our work ourselves, for example, the SolarWinds incident, our Comptroller General, immediately upon, you know, hearing about that compromise, decided that we needed to get out in front and start working on that right away.
John Breeden: Makes sense. Well, thank you, Josh. Appreciate that. That's, that's a good level set for our folks. Now, everybody's up to speed on the GAO. Appreciate that. And James, just so we know where you're coming from in terms of cybersecurity, can you tell us a little bit about Hewlett Packard Enterprise, and maybe some of the government programs that you're particularly proud to have been a part of?
James M. T. Morrison: Yeah, so I mean, HPE has a very large footprint in the federal sector. And so we've got a whole branch of our group that that's their focus is, is how to, you know, help develop and improve the security offerings in particular, you know, across the federal sector. But we've also been doing like, I was saying earlier was we've been really trying to drive more conversation into, you know, the SLED market, and offering kind of that, that expertise that we have, you know, from a security standpoint, for all those organizations. And what we saw was, in particular, as, as smaller government agencies, state and local start looking at cloud implementation, we have the ability to, we purchased the Cloud Technology Partners, which is a group that kind of helps come up with the right, the right implementation for cloud in your world, because there are no Greenfield's. Right, there's no, there's no, you know, cookie cutter that works for, you know, one group, it doesn't work for another. In addition, we offer like said the same sort of services, you know, and from a security standpoint of what, what can we do to help you, you know, get into a better, you know, security model, whether it's, you know, compliance questions, whether it's security In the cloud or security, you know, on your endpoints, we really feel like we've got that, you know, expertise that you may not be able to find in other groups. We also one thing we did was last year, we started offering what we call the trusted supply chain, where we are able to offer to entire server lines made in the United States. Most of them are manufactured in Chippewa Falls, Wisconsin, facility, everybody's cleared. And so we have the ability to control that supply chain, and provide a what we feel is a much more secure offering to our customers.
John Breeden: Excellent. Wow, really interesting stuff you guys are working on. Thank you so much. All right, let's get into ransomware. So let's begin by level setting the situation, especially for our government guests, who are squarely in the crosshairs of ransomware attacks these days. James, let me begin with you on this section. I know that you and HP get to work on a lot of private industries work with a lot of private industries, and also government customers. So what are you hearing from them in terms of their concerns regarding ransomware? And what is the threat landscape like today, compared with say, just a few years ago?
James M. T. Morrison: Yeah, I mean, you know, ransomware, unfortunately, or, you know, it has become really the attack of preference for most of these criminal groups. You know, we were talking before we came on about how, you know, these groups are operating out of countries around the world, that are providing some sort of support, whether it's implicit or explicit. And therefore, they feel that they are probably immune to any sort of federal, whether it be FBI or, or, you know, prosecution from any country around the world. The problem we saw, especially with COVID, was we saw a huge increase in the number of attacks. Work from home actually expanded the attack surface. So now, when we talk to customers, the conversation is how do I protect my edge of my network? How do I protect the, you know, the security of my endpoint, when my endpoint is sitting on a desk at home? How do I make sure that the data that's being processed, you know, at the edge of my network, is able to be safely conveyed into my data center or into a cloud implementation? So ransomware, has really started to focus around, you know, not just, you know, attacking the data, you know, in the data centers, but trying to leverage a new attack surface from that edge network. In addition, what we started to see is in this was about last 18, to 24 months, ransomware has also become a data breach, and that the double extortion cases where they will steal the data prior to encrypting it, and then use that, you know, then they'll try to, they'll attempt to get, you know, the same level of another payment of ransomware, for example, from an agency or from a state or local government. There's actually and this is where it's kind of interesting is that we've seen issues also in in federal sector, and a little bit in the state local sector, is when they steal that data, they also sometimes look at that data to see if there's something in that data that might be potentially embarrassing or damaging to that local government. So let's say there's a report on police, you know, a police incident, or body cam footage, some of that stuff has been stolen through some of these attacks, these criminals will then turn around, say, if you don't pay us, in addition, we will expose this to the press, and enough further attempt to embarrass. So it's all it's all sort of a form of extortion using that data. So that's the landscape that we've really seen, and how it's affected state and local governments.
John Breeden: Wow, that's some intense, intense stuff going on. Thank you so much. So Josh, before the show, you and I were talking a little bit about ransomware initiatives that you in the GAO we're working on at a high level for now. Can you describe some of the things that you've learned or seeing right now in terms of the threat landscape, and the surveillance of the ransomware threat, especially as it involves government agencies?
Josh Leiling: Thank you. So. So I agree with James. So the ransomware threat has steadily increased in the past few years. And it seems like in 2020, there was a pretty dramatic increase in these types of attacks. For example, just to cite a few statistics from the research that I've done, the multi-state Information Sharing and Analysis Center reported in 2019, that they observed 153% increase in the number of attacks reported on state, local, tribal and territorial governments from the previous year. Then, in 2020, we had the Institute for security and technologies the ISP ransomware Task Force reporting that nearly 2,400 US based governments, health care facilities and schools were victims of ransomware. And that came from data from the security firm MC soft. And then similarly, FBI received nearly 2,500 ransomware complaints in 2020. And that was up about 20%, from 2019, according to be an IC three report. And then more recently, we heard from the US Department of Treasury that in 2020, ransomware, payments reached over $400 million, by their estimate, which was more than four times the level in 2019. And, really, that's just a fraction of the overall economic harm caused by the cyber-attacks, because of course, there's costs, you know, associated with recovery and the like. And that's a pretty significant increase over prior estimates. For example, the National Cyber investigative Joint Task Force previously estimated that from 2013 to 2019, the payments were around 144 million in Bitcoin paid out as ransom. So I wanted to provide a few of those statistics, you know, just to kind of level set, but, of course, you know, the statistics are gonna vary depending on which source you're looking at. And, you know, there's not really one, one source that sort of provides the true number, because what really is the true number, it's likely that we're not capturing the full extent of the threat, in part, because some victims, you know, may be reluctant to report their incident. And so, you know, we have these broad measures of the number of attacks. But what's clear, is ransomware continues to be a major threat to government agencies, especially at the state and local level.
James M. T. Morrison: He brought up a great point there. And I think we always felt like in the bureau that the numbers were like dramatically underreported. And I think there's, I think there's sort of a, I mean, for example, we had here in Texas, we had 14 municipalities get hit, because of a managed service provider that was hit by an attack. And a lot of those groups never even reported it, we found out only from the MSP, that they had been hit by ransomware. So that's a concern, especially, you know, from the stakeholder standpoint is, how do I validate that that kind of attack hasn't occurred against my, you know, my city or my municipality? And then last, did I lose any of my data? And that's where I think privacy laws are starting to push into that area now.
John Breeden: No, that's really interesting. James, and I was kind of curious about beyond just the volume of the attacks, you sort of were mentioning this a little bit, but a ransomware, the thread itself is actually also evolving. And so can you talk a little bit about kind of the way that ransomware is, is in some ways getting worse? And also, what about the delivery method? The delivery method is usually email. But has that changed or evolved some as well?
James M. T. Morrison: Well, yeah, from a large enterprise, most of the time, they're coming at you through a phishing attack. That seems to be the kind of the attack that works, what we've started to see we've seen in small and medium businesses and in municipalities and state and local governments, is they're more likely to get attack through their managed service provider. So we have to be very careful of who we have contracts with, and who we grant the right of, you know, system administrator access, for example. And that's why a lot we've seen is at the state level, a lot of the state governments are starting to try to set up sort of an I hate to say an approved vendors list, but a validation process for MSP providers. We actually even saw CISA recently came out with a some guidance around MSP and how to choose the right MSP. So when you're when you're making that decision of, you know, who am I going to allow to remotely administer my network, it has to be a very outdated decision, and not a decision that sort of based on lowest, lowest bidder, and then that's, that's become really the attack surface. I think the numbers I saw were in small and medium businesses and state and local governments, is 60 to 65% of the attacks are actually coming through remote access, and system administrators being compromised.
John Breeden: Really interesting stuff. And it's interesting, you bring up phishing, phishing seems to be kind of the scourge these days. We actually have an entire show this week, as part of our headlines event on Friday, uh, just about fishing. So we'll definitely get into that. So thank you for bringing it up, James. Josh, I know that you are actually working on a GAO report in this area, which I'll ask you about in just a bit. But based on what you're seeing, I know you've sort of gone over some of the some of the statistics. Are there are certain areas where the ransomware criminals are? I mean, we realize they'll target anyone but are there certain areas where they're really kind of putting people in the crosshairs these days?
Josh Leiling: Yeah, thanks, John, for that question. So unfortunately, the threat actors have really focused in much of their attention on institutions like hospitals, school districts, utilities and local governments, in addition to the millions of dollars that can be paid out in ransoms and recovery, the disruption to these critical sectors can really cause severe damage to our health, safety and economy, and can really disrupt the services that constituents require, I think, the ransomware threat and you know, honestly, all cybersecurity threats for that matter, becomes a tougher challenge for smaller counties, jurisdictions and municipalities that just may not have the same staffing capacity and resources as larger counties or state agencies, I've talked with several that may have only one IT person if they're lucky. What's interesting is that some of the local government agencies and utilities that have been attacked, reported that they thought that no one would be interested in exploiting them for ransom. But in truth, these can be prime targets for the bad actors who may be looking for a quick payout, you know, because there's going to be a lot of pressure to restore service, for example, if a 911 system or a police dispatch system goes down, in some cases, local government agencies, you know, may not fully understand or have an appreciation for the cyber threat and protections that are required, which could make them more vulnerable. But also we see cases where there is an understanding of the threat and what needs to be done. But, you know, they may just not have the resources have the capacity to address the vulnerabilities.
John Breeden: No, that makes a lot of sense.
James M. T. Morrison: You're talking about backups. And it's absolutely true that the first thing you're going to go after is your backup. You know, I think we may touch on that a little bit. But also, we've also seen cases where they knew how much to ask for on ransom, because they knew how much your insurance policy was for that it's an interesting thing that they some of the ransomware groups have done a little research ahead of their, their attack. And they knew that you have an insurance policy for a million dollars, so therefore, they asked for a million dollars. So be very careful about that those little elements of information that we think aren't security related, but they give up very critical information to a determined attacker.
John Breeden: Excellent. And thank you, James, for jumping in. So thank you both for helping to level set the situation in government. Now let's talk a little bit about how state and local governments can help to mitigate this, this terrible threat. It's been said on previous FedInsider webinars that fighting ransomware is at least a twofold process. First, you have the prevention angle, but then part two is Recovery and Resiliency. So let's talk a little bit about that. And starting with prevention side of things. Josh, I know you've been waiting to talk about this. I know this is a big issue for you. The federal government offers a good deal of assistance to state and local governments in the prevention side of ransomware, including some programs that they may not be aware of. I know that the GAO is working on a report right now about that, but what can you share with our audience today about some of the help that may be available to them in this area?
Josh Leiling: Certainly, I'd be glad to do so the federal government can provide a variety of assistance to SLTT governments, often at no cost. So to start with DHS is cybersecurity and infrastructure security agency, or CISA manages a central federal website, stopransomware.gov, which contains a ransomware guide and links to other guidance and resources from various federal agencies. So I think there was a question about, you know, where can I find certain NIST guidance, there are links to NIST and FBI and Secret Service guidance as well on this website. Now, in terms of direct assistance, DISA and the multi-state Information Sharing and Analysis Center also offer a variety of technical tools like scanning and testing services. That includes vulnerability scanning, which can identify externally accessible assets and services that are vulnerable to common attacks. They also have web application scanning, which can identify website weaknesses and poor configurations that attackers might exploit. And then one of the popular ones and we were just talking about this a moment ago. They also offer a phishing campaign assessment, which can help to determine whether personnel are susceptible to those malicious emails. And we mentioned this that that is definitely a leading cause of ransomware. There's also a remote penetration test that a lot of state and local and other entities are taking advantage of which you know, can assess your perimeter defenses by mimicking techniques of the adversaries. I also want to mention a service known as the malicious domain blocking service that cis and MSI sack offer and so this is designed to prevent your IT systems from connecting to harmful web domains and can potentially blocked the vast majority of ransomware infections. Finally, in terms of support from CISA, I wanted to note that they also, in some cases can help, you know, with a personal and customized support through their regional cybersecurity advisors. Cisco does have a number of regions throughout the country. And there are several cybersecurity advisors who can provide state and local officials input on how to reduce their cyber-attacks. One of the goals of this program is really to just have a trusted advisor and a point of contact available, especially if something goes wrong, so that they can help guide you through the situation. But from the prevention side, they can also run you through a number of their services and assessments that can help you prepare and test out, for example, your incident response plan or other measures. I should also know that the DHS has the nationwide cybersecurity review. That is a free and anonymous self-assessment that helps to measure gaps and capabilities of SLTT cybersecurity programs. And then CISA has developed a self-paced review option through a desktop application called the ransomware readiness assessment. So all these services are discussed on the website, stopransomware.gov. And then real quickly, John, I just wanted to mention, you know, from the law enforcement side of things, and certainly James might be able to speak to this as well. But one program that I personally wasn't aware of, until only recently is a free cybercrime investigation training that the Secret Service offers to state and local government as well as legal and judicial professionals. And that's through the National Computer forensics Institute. In addition to classroom training, there can be a good deal of technical equipment that can be provided after the course. And then of course, DOJ and FBI share a lot of information through those services that I mentioned previously. And they have also been working to help disrupt the infrastructure of the bad actors.
John Breeden: Great. Well, thank you, Josh, that that's some really good information, it's nice to know that there's a lot of help out there. And our state and local government people can reach out and start to take advantage of some of that. So appreciate you bringing that up. James, looking at the technical side of things, since I know you're an expert in this area, what are some of the things that state and local governments should be considering in terms of trying to prevent their ransomware attacks? What are some good steps that you would definitely recommend that they should be looking at?
James M. T. Morrison: Well, again, we've kind of talked about, we've hit on the idea of backups, and I definitely evaluate my disaster recovery plan. There's a statistic out there that says 75% of backups fail when you need them. So I've actually heard some of my backup, people say that backup should be considered recoveries only if they're successful, you know, testing your backup plan to make sure that you're actually backing up the critical data that you need to restore operation. The second thing is, is really looking at a good data loss prevention type technologies, encryption of data at rest. You know, we often I talked to one of the hackers from years ago, and he had he had reformed his way done his time. And, and what he said was, one of the biggest things that people could do to prevent, you know, a data breach is to make sure that the data is useless to the to the criminal when they steal it, you know, looking at, you know, encryption technologies for data at rest. And then lastly, and this is sort of a non-technical in having an incident response plan and making sure that you test it before something goes wrong. And, you know, my example was, I went out to one of the municipalities, they got it in that ransomware attack, and I think it was 2019. And I walked in the building, and this two days after the attack, and there was two judges, a sheriff and two part time it people sitting around a table still trying to decide what to do. And they asked me, you know, what do we do, and I was like, of course, the FBI weren't really allowed to provide too many options of what to do next. But I was surprised that they just didn't have a plan in place. You've got to have a plan in place prior to ransomware. And it has to be tested successfully. I wish I could reinforce that more than enough. But you have to make sure you test your plan and know what everybody's was role is. And I guess one thing I would add is, I think somebody put this in as a point, you know, making sure that anybody who has system administrator access to your network, there needs to be an a solid, iron clad agreement in place of what they're allowed to do and what they are not allowed to do. So you have to make sure you have a third party. I think they call it a third party remote access agreement, and no as those rights so those would be the kind of the most solid semi-technical responses I would say.
John Breeden: Excellent. Well, thank you so much. No, that's very interesting. Josh, it occurs to me, you know, Some of the programs that you described and the help that's available from the federal government, it almost sounds like the same kinds of things that were offered to state and local governments leading up to the previous presidential election. I remember a guest from CISA back then talking about the ways that they could help local municipalities secure their election systems. So are these pretty much the same kind of programs but aimed at ransomware instead of protecting election security around like a certain day or a certain event, this is more the same type of things, but applied to continuality of operations?
Josh Leiling: Yeah, I think that's a good way of framing it, John. In a sense, many of the services from systems catalog to have been used in the push to enhance election security can also be used to address a variety of other cybersecurity concerns as well, in part because they're targeting essential cyber hygiene practices, for example, CISA and then OSI stack used the malicious domain blocking service, and another service called tabletop in a box to help enhance the security of elections, and have also seen their value as key tools in the fight against ransomware. A lot of the services in the catalog address a broad range of cyber threats. And so in essence, they can have value in addressing issues regarding elections, ransomware, or other areas. I recall when we were doing our election security work that the phishing campaign assessments was the number one requested service for a while ahead of you know, election season. In some cases, CISA has tailored a few of its services to address ransomware specifically, such as the ransomware guide, and, you know, scans of ransomware related indicators of compromise and scenarios that they've built into the National Cybersecurity review and the ransomware readiness assessment. And then CISA has also been working to customize the scope of certain services for smaller or you know, localized jurisdictions recognizing that their needs and capacity may differ, compared to perhaps a statewide IT shop. One example of this is the remote penetration test service that doesn't require quite as much preparation or burden from the local agency is the full on site penetration test.
John Breeden: That's really good. Thank you, Josh, appreciate you explaining that for us and making those connections. So James backup is kind of a little bit more on the recovery side of things. But I wanted to ask you about the importance of it in terms of mitigating a ransomware attack, and how should agencies deploy their backup in order to keep them from being infected with ransomware, as well. And we've heard examples where people have backups, and then the backups get encrypted as well. So what are the kind of things that you should think about when setting up a backup program?
James M. T. Morrison: Yeah, so and that's a good, I mean, a good question around that. Because, you know, it's not just about having a backup, but it's making sure that that backup is either not accessible or are not changeable. And so the old military rule was what he said, three to one, three backups, two mediums, and one of those in an off-site. But there's more technologies out there that actually don't require you to have the backup in an off-site. And it's called immutable backup. And what an immutable backup does is, you know, and we, of course, he has some technologies around this as well is it takes the backup, but then make that backup, unchangeable, for a period of time, usually 30, or, you know, 60 days. And then what happened then so in the criminal pops in there and tries to, you know, either delete or encrypt the backups, the immutability actually prevents that backup from being changed. Now that that means that what they'll try to do is they'll try to change its location. But they found that it's very, very tough for them to actually make any changes to those backups, backup files. In addition, the old mindset of having, you know, if you're in a cloud, if you're big enough to be in a cloud space, we acquired a company called Zurdo, which has a 100% ransomware recovery mechanism. Because what it does is it does, it keeps two copies of all of your data in real time up in in cloud instances. So that if you lose one, you know, one cloud, for example, to ransomware, the backup clouds and automatically recover to the last the last node staff snapshot. So there are a number of technologies around that. But even at a lower tech level, and I'll talk in particular about this one. This one city, I went to this one city had a single backup drive that backed up their data every day. And that was all they had. And so when that backup drive got encrypted, they didn't have another copy. And what was interesting was that in that in that city, the tax office was maintaining its own backups, and the tax office was rotating reback up drives on a weekly basis. So what happened was they only lost their last week of data, they were then able to pull a week ago data and recover to that one. And in this case, the ransomware had not infected that backup. But that's sort of a challenge sometimes, because, you know, you don't know how long that ransomware has been in the network before it's exploded. So sometimes we'll have people call us and HP and say, Hey, can you help me find a backup that, you know, hasn't been modified or isn't been infected? So it's a that's a kind of a guess, another service that a lot of the vendors will provide for their product?
John Breeden: Yeah, very interesting story. So in that particular city, those people that live there, their services stopped working. But luckily, they were still able to pay their taxes.
James M. T. Morrison
Because the tax office didn't want to turn their stuff back on until the firewall was figured until the firewall was set up. The records were safe, they just weren't able to really turn on online services. But that said he did lose 911 services for a period of a while they were recovering. They were 911 had to go back to being paper, you know, they had to go back to a paper system.
John Breeden: Wow. Great, great story. And good, good to look at. So I wanted to I know both of you have real world experience with fighting ransomware. So I wanted to tap into that a little bit for this next section. Josh, I know you previously worked on an election security report, as you mentioned, I wanted to ask you if in doing these research for these different projects, and the one you're working on right now for ransomware. Have you learned anything that might be helpful for state governments to know about when combating ransomware beyond? You know, the help that's available? Are there any like, you know, key insights that you've gained for going into this dark world?
Josh Leiling: Yeah, sure. So a few things. I think one of the things we learned in our election security review is that the resources and capacity to deal with all cybersecurity threats, including ransomware is a real challenge for local counties and jurisdictions. The amount of dedicated IP support for local counties and schools is often limited and cybersecurity may not be considered a top priority for officials who have to deal with a wide variety of issues. So as you might expect, oftentimes the issue was not necessarily implementing a sophisticated cybersecurity program or software. But there was a more fundamental need to build a culture of cyber awareness and address some of the basic cyber hygiene issues. To that end, one of the more popular services that the election community took advantage of I mentioned previously was the phishing campaign assessment. In some cases, the services and assessments you know, helped to raise awareness of cybersecurity concerns and address some of the low hanging fruit. Another thing that we learned is that local flush officials can be flooded with information and guidance about cyber threats. And they may not know what is the most critical and relevant information that needs to be addressed. Some of the local officials we spoke with found that it was absolutely essential to have a trusted cyber advisor from either their state, the private sector, or the federal level, who can help them navigate all this information and take the necessary steps. And while the services that CISA, MSI SEC provided can be helpful, one thing we learned is that some of those reviews and assessments can take time to arrange. So you know, when planning things out on your calendar, it's important to consult with a regional advisor about how long that may take. One of the key themes from our review is that sometimes state and local officials, you know, simply we're just not aware of this assistance that was available to them. And while CISA, and other agencies have, you know, attempted to market their services, you know, some are still not aware of the full extent of the support that they can receive. And some are already obtaining that those things from a third party. So I guess our main takeaway in that election work was that when local jurisdictions did reach out and obtain those services from the federal agencies, they were generally quite pleased with the assistance, and mostly the fact that it was available at no cost.
John Breeden: Yeah, definitely. Thank you. One of the things that we hear a lot about when talking about ransomware, is that especially for a public facing enterprise setting, preventing 100% of the ransomware threat is practically impossible. Eventually, something's going to get through. So the concept now seems to be shifting towards, obviously prevention is important, and you want to do that. But you also have to, and I believe James talked about this, you also have to have a plan, and you need to be able to recover quickly. James, I just want to ask you, I know you've spoken this a little bit previously, but how important is that resilient side of the ransomware plan? Could you maybe just reiterate for our audience how important it is to be resilient as well as preventative?
James M. T. Morrison: Well, yeah, and part of this comes down to the idea of having your data properly classified right, and having an understanding of what data It is important to your to your state or local government, you know, to restore operations. Because since we know that, you know, I can't spend money equally across my entire network, I can't protect every endpoint, I can't protect every server. So, you know, what may happen is we may have more security on one particular data location than I might have on another location. And so, it's really important to have that understanding, know where that data is stored, make sure that our network is segmented. So that, you know, unfortunately, a lot of our networks are very flat and open. So that if I lose, if I if I get, for example, a phishing attack or, or an attack in one portion of my network, I can at a minimum, you know, based upon my incident response plan, I can shut down that portion of the network and limit the damage to a portion of my network versus the entire thing. I think part of that that preparation ahead of the time, you know, and that was like seven, this particular town, the city I went into the Tax Office kind of had their own plan. And they were able to avoid the damage of the ransomware attack, because they had, they had sort of a different methodology of how to protect their data and backup their data. So it's crucially important to know your data, know where that data is stored, and then prioritize your backups, your recovery and your security around what data is critically important.
John Breeden: Makes a lot of sense. And Josh, I wanted to ask you about this, that we appreciate you telling us about all the programs that are available in terms of prevention. But on the recovery side, there seems to be a little bit of a misconception that the federal government will come in, you know, riding on a white horse and rescue after a ransomware attack. But that's not really the case is it?
Josh Leiling: Unfortunately, it's not. I'm glad you mentioned this topic, John. So federal agencies like CISA, FBI, and secret servers can be very helpful in a variety of ways. And I don't want to diminish the importance of what they do. For example, they can help provide checklists and walk you through what to do to prevent the attack and or to prevent the spread of the attack and help to isolate your systems. They can also do some forensic analysis by examining system images and identify the attack vector, they can provide, you know, some immediate and longer term recommended solutions. And in very rare instances, although I wouldn't count on this, they may be able to provide a decryption key. So all those things can be really helpful, especially when you're in a state of panic and unsure about what to do next. But at the same time, it's really important, and James mentioned this earlier, when he said, it's important to have that agreement on, you know, what folks are allowed to do and not do. But it's important to understand the limitations of any federal assistance that, you know, may come on the recovery side, a federal agency is generally not going to put hands on keyboards to mitigate the threat and restore operations for you, you will still be on the hook to carry out the necessary steps to isolate affected networks, recover backups, restore systems, and any of those longer term concerns like policy updates. And typically the complexity and sheer volume of that work to restore and recover. Even if you have the capability to do it. It's going to demand some outside help oftentimes, and this can potentially be provided by a statewide CIO or CISO shop or an IT managed service provider. If you have one or a consultant that you are, perhaps your cyber insurer hires to deal with the issue, the State National Guard may be able to assist with recovery and restoration. I just wanted to know that, you know, they're out there. But it would be similar to a hired contractor. And it's really going to depend on the criteria for calling them into action. For example, it may require declaring a state of emergency to activate them. And it really depends on whether your state has already funded them for this activity.
John Breeden: Wow, lots to think about.
James M. T. Morrison: I was gonna say, because that's what happened in this particular case in Texas when they had the 14 municipalities was that they activated the Cyber National Guard unit, you're in Texas and had them go out and assist with the recovery. But it did it required. I think the governor had to declare a state of emergency in order to activate that. So that's a good point.
John Breeden: Yeah, definitely something you don't really think about. So that's good to keep that in mind as a resource. Josh, let's start with you on this one. A lot of the programs you talked about today, like malicious domain blocking, they sound really good in terms of the kinds of help that the federal government can offer to state and local agencies, for our state and local listeners that are with us today. How can they get started with those programs? Who should they reach out to in order to start to get that help that they so desperately need in terms of ransomware?
Josh Leiling: Sure. So I actually think a great place to start would be to touch base with it officials within your state or locality or even a vendor or contractor who you work with, because they're likely already aware of a lot of those services that are available. And there may be some that you could receive, you know, within state, they may also be able to help coordinate certain services for you and put you in touch with others that have received similar report or support. Sorry, that said, you can certainly check out the stop ransomware.gov webpage, which is that central webpage that system manages for more information about the services offered and they have some information of key contacts as well. I would also recommend just in general, reaching out to a year regional, either the FBI regional field office or secret service field office or your regional Cisco cybersecurity advisor, who can also provide more information about services and gets you scheduled for a variety of assessments.
John Breeden: Well, thank you, Josh. Very good advice. And James, it looks like you get the last word today on our show. We covered a lot of ground today from level setting the situation to talking about prevention, and then resilience. So to kind of wrap things up today for our audience, what are some of the key takeaways that you want to make sure they really know about and take home with them, as they try to mitigate the terrible ransomware threat?
James M. T. Morrison: Well, for an example, you know, he talks about Incident Response Plans, there are good examples of them out at NIST or SAN you, you know, look for some templates out there. It doesn't matter which one you use, it just matters that you use one. I mean, I've actually seen some incident response plans that were just written down notes. And as long as you're doing something, and you're making a plan to go forward, you know, make sure you're testing your people on phishing, you know, you know how to recognize phishing as a, you know, as a threat, and maybe have a plan for recording phishing. Now, that's very common. These are all non-technical issues. But I would tell you, you know, even before you have before you sit down and have a technical conversation about, you know, what should I buy it? And let's say we get to that point, you need to make sure you have all of your ducks in a row about what are my you know, my key data points, you know, what is my data classification policy? You know, where is my data stored? Where do I want it stored? I mean, these are all things that need to be done before you ever sit down with me at HPE or anyone else, but have a security conversation with someone, okay? Find someone in another town, find someone at the state level, you know, you can talk to, you know, advisors, but talk about security, come up with a plan, and then and then follow that plan as best you can. Every step you take away from being insecure, makes you know, less of a target or at least less of an easy target.
John Breeden: Now, that makes sense. I wanted to thank both James and Josh very much for being with us today. All of their insights made this an amazingly productive session, I learned an awful lot I'm sure our audience did as well. Ransomware is a serious threat, but it sounds like with good cybersecurity practices, and a focus on resiliency. And as James was saying, a having a good plan in place to deal with it should the worst happen. It sounds like it can at least be kept in check. So thank you to our guest today for explaining all that.
Speaker 1: Thanks for listening. If you'd like more information on how Carahsoft or HPE can assist your state or local government agency, please visit www.carahsoft.com or email us at HPEGroup@carahsoft.com. Thanks again for listening and have a great day.