What logs to collect in a SIEM?
When embarking on a Security Information and Event Management (SIEM) project, one of the most common dilemmas is determining which data sources to collect. Maximize the effectiveness of your SIEM implementation with these simple key considerations.
Technologies and domain specific data sources
A very common approach when it comes to determining what logs to send to a SIEM, is to consider both the technology-specific logs and the domain-specific logs.
For example, the following types of logs could be considered for inclusion in a SIEM:
Technology-Specific Logs:
- Firewall logs: they provide information about network traffic, including allowed and blocked connections.
- Intrusion Detection System (IDS) logs: capture events related to potential intrusions or malicious activities.
- Active Directory logs: they contain information about user authentication, authorization, and group policy changes.
- etc
Domain-Specific Logs:
- Authentication logs, record user login attempts, successful logins, and authentication failures.
- Access logs, track user activities, including file access, system commands executed, and resource interactions.
- etc
However, this is not a "natural" way of representing our IT systems and it feels overcomplicated, confusing, and it's easy to miss some data sources along the road.
Instead of relying on a complex and potentially confusing list of specific log types, it is more effective to adopt a holistic approach. In fact, all data is relevant to cybersecurity. So, we need to collect everything!
Do we really need to collect everything?
Yes, but no.
Over the last few years, Log Management and SIEM markets have been converging to a point where Log Management commoditized SIEM, and this is creating confusion when one starts such a project.
In fact, thinking of Log Management and SIEM as two separate services in a given organization will help foster the best solution. So, yes, we need to collect everything for Log Management, but no, we don't need everything for SIEM. We'll explore this hereafter.
What is Log Management?
Log Management refers to the process of collecting, storing, analyzing, and managing log data generated by various systems, applications, devices, and networks within an organization's IT infrastructure.
The primary purpose of Log Management is to centralize and organize log data to facilitate effective monitoring, analysis and troubleshooting.
Common use cases are:
- Troubleshooting
- Post incident analysis (Forensic, Postmortem analysis)
- Threat Hunting / Use cases development
- Dashboards such as network load monitoring for example
- Near real time view for debugging (tail -f)
- Compliance / Auditing
Ideally, we want to collect everything in a Log Management system because we don’t know in advance what incident — security related or not — we will be facing in the future so we don't know in advance what data will be relevant to the investigation.
INFO
Technologies evolved in such a way over the last decade that performances are no more a deal breaker as it used to be. We could mention for example technologies such as Splunk, Devo or Elastic. Also, from a cost ownership point of view, combining those technologies may also make sense depending on your context and use cases.
What is a SIEM?
SIEM systems collect, analyze, and correlate data from various sources, including logs, network devices, security appliances, applications, and more. The goal is to provide organizations with a centralized and real-time view of their security posture, enabling effective threat detection, incident response, and compliance management.
A SIEM, by oversimplifying, is a thing looking for threats by analyzing ingested logs, with techniques such as pattern matching or IOCs matching for example, and will ultimately raise an alert in near real-time when such a threat is identified.
Logs sent to the SIEM needs to be carefully selected because not all logs are created equal. Some logs are way more valuable to your Security Operations than others, so, you want to start with them (see volume-value matrix below).
Moreover, SIEMs are usually a separate software license and require a separate architecture to run. Avoiding to throw all logs to the SIEM will save money...money that can be used to hire detection engineers to develop threat detection use cases :)
In short, we don’t need to collect everything in a SIEM, that's a Sales speech (most vendors sell on a volume based license).
WARNING
SIEM also often comes with promises around case management and response capabilities (automation & orchestration) but they simply aren't there yet and we commonly have to select an appropriate SOAR or SIRP solution to complement a SIEM. The easiest path to get started here is using the company's existing ticketing system.
How to decide which logs to collect in a SIEM?
Think big, start small
Willing to build a tens of terabytes a day architecture is great, but new entrants should start more modestly to succeed (i.e. demonstrate value of such investment).
Starting with one single data source and fulfilling the whole loop from collecting the logs, ingesting them, indexing them, up to raising an alert for a specific log event and responding to it is the approach that will require the least effort and provide the maximum learning.
This approach is very similar to how one would write a prototype in Software Engineering. The learning from this exercise is huge and the effort, so cost, is relatively low so it can be quickly executed.
Another benefit of this approach is that it forces thinking about the response to be given to each detected threat. Too often a SIEM project is started and then alerts are fired by thousands, indicators are blinking like a Christmas tree, and because no effort has been put on the response phase, the SIEM is shut down or alerts are just ignored. This is how dramas start!
Volume / Value ratio
Data sources come with different volumes and values. For example, Proxy logs are known to be very verbose, so high volume, but also to be very helpful in crafting security detection scenarios, so high value.
With Proxy logs for example, we can detect C2 channels, SSH Tunnels, randomized domain names used by Malware (and CDN!), and much more.
The following matrix illustrates this concept of high/low value relative to volume.
| Low value | High value | |
|---|---|---|
| Low volume | Bash history | DNS logs |
| High volume | Netflow logs | Proxy logs |
It is important to understand that this is dependent on your very own context, but also it can change depending on if you are thinking in terms of Log Management or SIEM. For example, and in general, the Bash history will have little value in real-time monitoring but a lot of value in post-incident analysis.
This matrix is interesting because it forces to:
- Understand the volumes per data source (it’s not always easy!)
- Understand the inner structure of each data source (is this log format supported? What can we detect with it? Do we need to increase, or lower, the verbosity? Can we tweak it and add a custom field? etc)
Use-case driven detections
Another key in starting a SIEM project is to think of the threats to detect and what will be the response once detected. From there, it is possible to understand the associated footprint of these threats in your organization (i.e. traces produced).
For example, many SIEM come with default rules such as Port Scanning detection, but:
- Is it illegal to do a port scan? By law, or per your company policies? If not, do you still want to send an alert to your analysts for this?
- What can you possibly do when the alert is fired? Block the source IP address on the firewall? Then the IP will change and you are back at the starting point... so was it worth the effort?
- Do you detect port scan on your edges — from the Internet — or into a very sensitive internal zone?
Ultimately the goal is to build correlation rules that will trigger with no-to-little false positives, or your SOC will be flooded by irrelevant alerts, and thus will not be able to inspect everything. Recent history has proven such a situation can lead to heavy financial losses.