This post was written by Dave Wilson (Security Operations Engineer at Anitian) with contributions from Joshua Barott (Security Operations Manager at Anitian) and Ryan Farris (Vice President of Product & Engineering at Anitian).
The role of a security engineer never stops, never gets easier, and anything which increases manual labor is a non-starter. Bad actors constantly find new ways of exposing vulnerabilities, pushing SecOps teams to increase security posture through higher and deeper levels of observability. The ever-changing, dynamic landscape of new and evolving vulnerabilities and threats creates the ongoing challenge of building defense postures against cyber-attacks. Considering the multi-dimensional responsibilities of DevSecOps, they must be diligent with their time investments and must choose the best technology and tooling within budget to cast a wide enough net across the surface area for which they are responsible.
Leveraging Cybersecurity Anomaly Detection with Machine Learning
To enhance our threat-hunting capabilities, the Anitian SecOps team has invested in anomaly detection as a capability to surface subtle security risks and/or obvious outliers across a sea of data signaling. Our SecOps team uses Elastic’s X-Pack technology to build out unsupervised machine learning (ML) models. We are leveraging the application of ML to include other classification detections which are challenging for analysts to parse manually. This allows our Security Analysts to spend more time performing meaningful security analysis and dispatching those signals to our customers.
Anitian uses machine learning to perform ongoing anomaly detection across multiple sources of time-series data. This allows us to correlate data by classifying unstructured log messages into comprehensible strings, which can then be analyzed as components. We classify these correlations by host type to ensure we can differentiate between infrastructure elements like a Domain Controller (DC) and a load balancer as detections on these host types will detect distinct types of anomalous behavior.
Anitian ingests and aggregates many different log types from each client’s infrastructure, such as OS-level logging, VPC flow logs, endpoint detection and remediation (EDR) events, and console API calls. With each security component sending event data to an Anitian-hosted SIEM, a new time series index is added to the set of indices used in the machine learning anomaly detection algorithm. Performing anomaly detection over N indices covers more surface area and provides an opportunity to correlate anomalies from one index to another, which over time increases the confidence score of the ML job results. When a new, unique environment enters Anitian SecOps, we standardize the model training by use of consistent procedures across environments and log types. We continue developing methods to standardize detections and disseminate that work to relevant clients.
By feeding these time series indices into our machine learning tool, we can then enable alert thresholds in the SIEM which detect anomalous behavior, unlocking use cases such as:
- Login attempts during unusual or off-hours, both successful and unsuccessful
- Logins from improbable geographical locations
- More robust network intrusion detection
- Tracking of abnormal internal and external network traffic surges
- Visualization and analysis of correlations across multiple log sources, such as network events preceding certain OS-level events
- Detection of unusual processes by argument in addition to process executable.
- Behavioral detections with greater dimensionality.
- Expanding our coverage for the MITRE ATT&CK framework into domains previously very difficult to detect by conventional means.
- Numerous specific detections including the ones outlined in Figure 1 below.
Threat Hunting in Action using Machine Learning Anomaly Detection
After the initial model training period, the output of the machine learning detections begins to surface valuable insights that identify anomalous patterns of a bad actor’s attempts to breach or, in the worst case, identify the indicators that a breach has already occurred.
In the screenshots below [Figure 2], we see a sample of notable events detected, including an instance of unauthorized network enumeration using Nmap, and an unexpected registry modification. Nmap is a network scanning and security testing tool that can map and discover networks and identify listening services and operating systems. It is sometimes used by hackers to gather information in support of exploitation, execution, or lateral movement within an environment. An incident involving dozens to hundreds of discrete events, such as Nmap scanning, can be difficult to write detection logic for using traditional tools.
Isolated registry modifications are interesting because if there is a single change in one machine among many, it flags a lack of uniformity across the host fleet, which suggests a human has made a single change [Figure 3]. That action is indicative of suspicious activity if all hosts are meant to be managed through infrastructure-as-code. These alerts are easily consumable visually and come from different log sources and were made easily available through a machine learning-generated dashboard.
Any solution that consumes and makes decisions on large quantities of data can have drawbacks. An immature model can assign severity inaccurately and even cause false alerts. Analysts need to consistently investigate and validate the results over time to ensure confidence is high for the signals that surface as a potential anomaly. In this way, our machine learning capabilities act as an augmentation to the toolset available to our analysts to perform threat determinations.
Evolving Value with a Growing Infrastructure
Anitian is continuously evolving its approach to using machine learning as a domain-agnostic tool. This allows us to collect data from many disparate sources and output actionable security signals based on anomaly detection. Our models start with an initial supervised data set collected from OSINT sources and Anitian generated threat intelligence for compliance automation and enterprise cloud security, then are trained for each client’s individual environment for the highest level of customization. The longer our models run, the more easily they can distinguish interesting events that occur within a longer period. As a result, our analysts save time by allowing machine learning to winnow down the noise into interesting events. This time is then reinvested towards making the learning models smarter, creating a positive feedback loop of increased hunting efficiency and overall, a more productive SecOps team.
