The Math of SIEM Comparison

There are many comparisons and scoring reports like Gartner. But a small part of their scoring is technical capacity. Other comparisons available on the web or magazines are marketing, sales, and presales documents. They do not include extensive technical analysis.

In today’s ever-evolving cybersecurity climate, businesses face more threats than ever before. Finding the right SIEM is crucial in protecting against the latest risks and equipping your organization with a robust security strategy.

A SIEM’s power is in its correlation. 80% of SIEM is a correlation. if you are spending 80 percent of your time within a SIEM tool doing alert review and analysis, then you are on the right track. [SANS Your SIEM Questions Answered]

A detailed comparison of the correlation capacity of SIEM products technically will be given. The comparison based on the most critical correlation and detection capabilities:

1. Rule Chain (Multi-Stage Rules)
2. Correlation Logic
3. List/Watchlist Management
4. Real-Time Correlation
5. Cross-Correlation
6. Correlation Operators
7. Correlation Field Operators
8. Correlation Field Restrictions
9. Machine Learning

Rule Chain (Multi-Stage Rules):

Rule chain is the ability to combine multiple steps (rules) of a use case without any restrictions. This type of rule detects a sequence of events that occur.

Most of the SIEM tools like Micro Focus ArcSight, LogRhythm, Qradar, Securonix, and SureLog support multi-sage rules.

AlienVault, McAfee, FireEye, FortiSIEM, Solarwinds LEM, ManageEngine SIEM are the other SIEM tools that support multi-sage rules with some limits.

McAfee has a restriction on the rule chain capability. For example: "if a firewall admin login has occurred and after this login action there is no configuration change immediately (wait for 15 minutes) but if there is a change in the firewall after this 15 minutes within 12 hours, notify", is not possible with McAfee. It is not possible to develop this type of rule chain because it is not possible to define "wait 15 minutes" then check for "later 12 hours".

McAfee Rule Chain Editor

Since there are two or more actions that require time windows, the 10 minutes must be divided between them. For this example, five minutes is the period for each action. 

Once the unsuccessful attempts have occurred in five minutes, the system begins to listen for a successful login from the same IP source in the next five minutes. .so there is no chance to implement wait logic between actions (rules)

FireEye has the same restrictions as McAfee.

Solarwinds LEM has the same restrictions as McAfee.

ManageEngine SIEM has the same restrictions as Solarwinds LEM. In ManageEngine SIEM, there is no chance to define a new rule type to chain. Also, there are schema fields restrictions to link rule chains.

AlienVault has the same restrictions as Solarwinds LEM. Also, when chaining rules, Alienvault only uses None, Plugin_sid, SRC_IP, DST_IP, SRC_Port, DST_Port, Protocol, and Sendor. There is no way for other schema fields to link rule chains.

FortiSIEM also does not have "wait for 15 minutes" kind of capability to chain rules.

Logpoint does not have this kind of correlation capability.

Rapid7 does not have this kind of correlation capability.

One another requirement when chaining rules, is cross-linking of rule fields. As an example: If a device is the destination of a brute force attack and then this destination device is the source of the port scan, detect this device.

Alienvault only uses SRC_IP, DST_IP, SRC_Port, DST_Port, Protocol rule fields.

McAfee does not support cross-linking of rule fields.

Logpoint does not have this kind of correlation capability.

Rapid7 does not have this kind of correlation capability.

Exabeam and Securonix are UEBA tools. They are not correlation-based solutions.

Correlation Logic:

Rules are discriminators used to find a certain behavior. If their designer knows what it's searching for, they will be invaluable tools. To design a rule without any limits or barriers, the correlation logic of the rule engine must be very powerful and flexible. It is hard to test the correlation logic of the SIEM tools. One of the simplest ways is to try to implement a discriminator use case (correlation rule). For example :

"Detects more than three authentication failures from the same user within five minutes without any successful login in-between."

Micro Focus ArcSight also can detect similar use cases.

If you want to detect this use case with Splunk, it might be possible to do with "transaction" events. But those searches are very taxing in the search head.

Rapid7 and Logpoint have the same issues with Splunk.

AlienVault, FortiSIEM, ManageEngine SIEM, McAfee, Solarwinds LEM could not detect the above use case.

Another test use case is detecting changes. Rapid7 has a change detection capability.

Rapid7 Change Detection Wizard

Qradar also has a change detection capability.

Another example is "Never Seen Before Type Of Rules" [2,3]. While Micro Focus ArcSight, Exabeam, Qradar, Rapid7, Securonix, and SureLog have this capability, AlienVault, McAfee, FireEye, Solarwinds LEM, and ManageEngine do not.

If we continue with the decisive scenarios that are not available in every SIEM product, we can use the below rules:

1. If a machine bandwidth usage is > 200Mbytes within 5 minutes or If a user to DSTIP bandwidth usage is > 500Mbytes within 10 minutes.
2. If an account has not been used at least in the last 30 days, notify/lock/delete this account. (This use case is mandatory for FedRAMP Moderate; Control AC-2(3) nd NIST 800-53; Control AC-2(3))

McAfee does not support those types of rules. Please check the below blogs.

• https://community.mcafee.com/t5/Security-Information-and-Event/Total-Byte-Accumulator/td-p/583883
• https://community.mcafee.com/t5/Security-Information-and-Event/accumulator-field-in-correlation-rule/m-p/634698

• https://community.mcafee.com/t5/Security-Information-and-Event/Not-working-Accumulator-Value-Bytes-from-Client/m-p/437949
• https://community.mcafee.com/t5/Security-Information-and-Event/Calculate-fields-like-SUM-bytes-in-or-SUM-bytes-out/m-p/437295

The second rule is also not supported by many SIEM products like McAfee, FortiSIEM, LogRhythm, AlienVault, Solarwinds, ManageEngine, and RSA.

Maybe you can develop a query with Splunk and Logpoint. But if you want to detect this immediately, you should consider system resource usage.

List/Watchlist Management:

Micro Focus ArcSight, LogRhythm, Qradar, Securonix, and SureLog - all have a list management feature. 

Both products support simple lists, multi-dimensional lists, complex lists, lists with 20 columns. Also, those products add, delete, modify, list items dynamically, or manually. 

AlienVault:

Dynamic list usage in correlation rules is not supported in AlienVault. It is not possible to develop a rule like If a VPN user connected after business hours and the user is not in VPN white list, alert.

The only way to implement a simple Active Lists is to develop a code.

https://www.alienvault.com/blogs/security-essentials/how-to-use-ossim-usm-active-lists-with-python-scripts

But even if you can develop Python Scripts, there is no key: value, reference set, reference map, multi-dimensional type of lists. 

AlienVault SIEM does not support updating multiple lists at the same time (more than one list) by a single rule (only one rule). Also, AlienVault does not support list operators like count, sum, compare, check case sensitivity.

FortiSIEM:

Dynamic list usage in correlation rules is limited to one dimension.

There is no key: value, reference set, reference map, multi-dimensional type of lists. The only available operators are “IN, NOT IN”. Also, the only way of removing items from a watchlist is time-based. Also, FortiSIEM does not support updating multiple lists at the same time (more than one list) by a single rule (only one rule). Also, FortiSIEM does not support list operators like count, sum, compare, check case sensitivity.

McAfee:

There is no key: value, reference set, reference map, multi-dimensional, type of lists. McAfee SIEM does not support updating multiple lists at the same time (more than one list) by a single rule (only one rule). Also, McAfee SIEM does not support list operators like count, sum, compare, check case sensitivity.

LogPoint:

LogPoint supports two kinds of lists; Static List and Dynamic List. Also LogPoint supports tables, but there is no reference set, reference map, multi-dimensional type of lists.

Also, if you are looking for a GUI for list/watchlist management, LogPoints works over queries. Dynamic lists and table updates are the only query-based. Also, LogPoint SIEM does not support updating multiple lists at the same time (more than one list) by a query. Also, LogPoint SIEM does not support list operators like count, sum, compare, check case sensitivity.

RSA NetWitness Platform:

RSA has a limited list management capability. There is no key: value, reference set, reference map, multi-dimensional, type of lists. Also, RSA SIEM does not support updating multiple lists at the same time (more than one list) by a single rule (only one rule). Also, RSA SIEM does not support list operators like count, sum, compare, check case sensitivity.

There are many other correlation features to check [1] . But without an advanced list/watchlist management, it is not possible to detect advanced attacks.

Real-Time Correlation:

AlienVault, Micro Focus ArcSight, Fireye, FortiSIEM, LogRhythm, ManageEngine SIEM, McAfee, Qradar, RSA NetWitness, Solarwinds LEM has a real-time correlation capability. if you use Splunk ES for real-time detection, you have to consider "Each real-time search unpreemptively locks 1 core on EVERY INDEXER and on your Search Head”.

Elastic also has no real-time correlation feature.

Cross Correlation:

Micro Focus ArcSight, FortiSIEM, LogRhythm, McAfee, Qradar has a cross-correlation capability.

AlienVault cross-correlation can only run on (just) IPS and Vulnerability Scanner logs and the combining on just IP addresses.

RSA NetWitness utilizes ESPER CEP, and there is no GUI for cross-correlation rule development.

Logpoint does not have this kind of correlation capability. Mainly it is a search-based tool.

Rapid7 does not have this kind of correlation capability. Mainly it is a search-based tool.

Exabeam and Securonix are UEBA tools. They are not correlation-based solutions.

Correlation Operators:

Micro Focus ArcSight, LogRhythm, Qradar, SureLog has a correlation operator support feature, such as:

• And
• Or
• Fallowed by Within
• Not Fallowed by Within

SureLog also has some additional correlation operators like:

• At the Same Time
• Before
• After

McAfee has some missing operators like "At the Same Time", "Before", "Not Fallowed by Within"

Solarwinds LEM documents mention some other correlation limits.. For example you cannot create a rule using “NOT FALLOWED BY” operator."At the Same Time", "Before" are an example of ther missing correlation operators.

Only "AND", "OR" Operator supported. "NOT" Operator is not supported. Also, other operators listed above are not supported.

ManageEngine Eventlog analyzer correlation has only one operator “Fallowed by Within”. Many operators are missing like ”Not Fallowed by Within”.

Also, other operators listed above are not supported like "At the Same Time", "Before".

FortiSIEM does not support "At the Same Time", "Before".

RSA NetWitness does not support "At the Same Time", "Before".

Logpoint does not have this kind of correlation capability. Mainly it is a search-based tool.

Rapid7 does not have this kind of correlation capability. Mainly it is a search-based tool.

Exabeam and Securonix are UEBA tools. They are not correlation-based solutions.

Correlation Field Operators:

Required correlation operators for powerful correlation:

• Link Fields
• Check Base64
• Count Characters
• In List
• Not In List
• Count
• Sum
• Regex Matches
• Matches,
• Not Maches,
• Entropy Bigger Than
• Entropy SmallerThan
• Is null,
• Is not null,
• IP Range Equals,
• IP Range Not Equals,
• In list,
• Not in list,
• Starts with in list,
• Starts with in list case insensitive,
• Not starts with in list,
• Not starts with in list case insensitive,
• Contains list key in data,
• Not contains string in list,
• Not contains string in list case insensitive,
• Is contained in string,
• Regex in list,
• Check data in regex list,
• Contains in list,
• Not contains in list,
• Contains credit card number

McAfee has some of those correlation operators but less than the above list.

McAfee Operators

AlienVault, FortiSIEM, ManageEngine and Solarwinds LEM does not support most of the above list.

RSA NetWitness utilizes ESPER CEP, and there is no GUI for rule development. ESPER language does not support all of the operators.

Logpoint does not have this kind of correlation capability. Mainly it is a search-based tool.

Rapid7 does not have this kind of correlation capability. Mainly it is a search-based tool.

Exabeam and Securonix are UEBA tools. They are not correlation-based solutions.

Correlation Field Restrictions:

Micro Focus ArcSight, LogRhythm, Qradar has no restrictions on fields. All the available fields on search and report schema will be available for correlation.

McAfee has also some limitations. if I see a user attempt to login to our VPN from two different "regions" within a three-hour window. I have the logic built but in the correlation rules "Advanced Options" I try to set a 'Distinct values' of 2 but the monitored fields only seem to provide a 'Source Geo location' option, and not the ability to select state, region, country, etc.

AlienVault correlation engine has sticky diff restrictions.

Solarwinds LEM does not use all the report fields on correlation. Also, correlation cannot fire on raw log data that is received

ManageEngine SIEM has correlation field restrictions. It is not possible to use all the available reports and search schema in correlation.

Machine Learning:

SureLog, IBM QRadar, Microfocus, LogRhythm, Exabeam, Securonix, NetWitness Platform has NLP/ML/AI features like DGA detection, outlier detections, rarity detection, similarity detection. 

LogPoint uses 3rd party UEBA tool Fortscale (RSA Now).

References:

1. https://drertugrulakbas.medium.com/detecting-unusual-activities-using-a-next-generation-siem-use-cases-d91f4e24b0f2
2. https://drertugrulakbas.medium.com/at-the-same-time-siem-operator-be8d6598b7b8
   
3. https://www.itcentralstation.com/articles/what-really-matters-when-selecting-a-siem-and-how-to-choose-a-siem-looking-into-the-correlation https://answers.splunk.com/answers/663659/need-help-writing-query-to-alert-if-an-account-has.html
4. https://techcommunity.microsoft.com/t5/azure-sentinel/azure-sentinel-correlation-rules-the-join-kql-operator/ba-p/1041500
5. https://gosplunk.com/accounts-deleted-within-24-hours-of-creation/
6. https://drertugrulakbas.medium.com/siem-for-smb-in-2020-a04e3fe8e98d
   
7. https://www.itcentralstation.com/articles/how-to-select-the-right-siem-solution
8. https://drertugrulakbas.medium.com/ml-ai-is-a-feature-not-a-silver-bullet-and-ueba-questions-d504a6926c4e
   
9. https://www.linkedin.com/pulse/ai-cybersecurity-closing-steve-king/
10. https://towardsdatascience.com/the-limitations-of-machine-learning-a00e0c3040c6
11. https://www.computerworld.com/article/3466508/the-impact-of-machine-learning-on-security.html
12. https://drertugrulakbas.medium.com/detecting-unusual-activities-using-a-next-generation-siem-use-cases-part-2-27b201bcc127