How to Scale up Your Data Initiatives

As soon as agencies started thinking in terms of enterprise-level data initiatives, their existing data solutions became legacy systems.

Let’s say an agency’s security operations center wants to use analytics to improve its ability to detect and respond to threats throughout its environment. The problem is the necessary data is scattered across the organization and stored in siloed systems both on premises and in the cloud or, more likely, in a multi-cloud environment.

Legacy data management solutions simply were not designed with that kind of scenario in mind, said Chris Townsend, Vice President for Public Sector with Elastic, which helps agencies find the answers they need from data in real time and at scale.

“The way that we have many of our systems today, making that data available to our government analysts can be very cumbersome,” he said.

A Data Lake Reality Check

Some organizations try to alleviate their data complexity by dumping all their data into a central repository, such as a data lake. But with enterprise-level initiatives, too much data is spread too far to make a central repository a practical solution.

But even if it’s doable, a data lake doesn’t fit with how most agencies want to operate, Townsend said. They will continue to maintain some of their data on premises and some on their various cloud platforms.

“We need solutions that are more flexible in allowing access to all that information in a very fast, efficient and cost-effective manner that’s operationally sound,” he said. “That’s where things are going.”

A Different Tack

An increasingly popular alternative is to leave that data in place, but to index it using a common platform that makes the data readily accessible and findable. This is the approach that Elastic takes.

This decentralized architecture — sometimes referred to as a data mesh — makes it possible to handle growing volumes of data without running into performance problems.

That is what made Elasticsearch a good fit forthe Continuous Diagnostics and Mitigation (CDM) program at the Department of Homeland Security. Elasticsearch is the foundation for the CDM Dashboard II, which enables cyber teams at the Cybersecurity and Infrastructure Security Agency to search cyber data in hundreds of federal agencies as part of threat-hunting efforts.

“We’re deployed in some of the largest government agencies at scale because this is a very flexible, efficient way to index and ingest the data,” Townsend said.

Elasticsearch as a Service

To help agencies pivot quickly to this approach, Elastic offers Elasticsearch as a managed service. The service ensures one-click upgrades to the latest software, as well as immediate access to new features and fixes.

Additionally, Elastic will take responsibility for maintaining snapshots for backup and recovery, monitor network and hardware performance, and provide engineering support as needed.

Agencies can purchase Elasticsearch Service within the Elastic Cloud console or through the cloud marketplaces offered by Amazon Web Services, Google or Microsoft.

To see more of our experts’ ideas on using cloud and cloud-based services to help your agency adapt to change, download our guide. “How to be a Pivot-Ready Organization.”

Photo by Felicity Tai at pexels.com