What's The Best Way to Trial SIEM Solutions?

1. Understand your environment: Segments, microsegments etc. Know where everything is.

2. Understand what your trying to do: Why are you monitoring, regulations? Compliance?

3. Understand your retention requirements: Storage Cost!!! Your capturing events per minute, and it gets expensive.

4. Understand how you want to use the SIEM: Is it part of your SOC or NOC? How will your Security Analyst use it? Will it be monitored 24/7? Have a game plan on who and what to do with alerts.

5. There are two basic ways you will pay for it: Either by the amount of traffic, or by the # of employee’s in the company. Splunk uses the amount of traffic across the wire, Exabeam is by # of Employees.

6. Should you use VM’s or buy hardware. Hardware is cheaper in the short run, but in the long run, VM’s are cheaper and more versatile with storage.

7. Do you have C level buy in? This will cost, so if you don’t have that level of buy in you will not get what you want.

8. Narrow your choices down to three vendor/solutions and ask each to do a pilot program with no promise to purchase for 90 days if possible, shorter if needed. This will give you an idea of the amount of data you will be monitoring and give you a better idea of the cost. Set each solution on a different subnet of the network and then review the success or failure of the solution with those that have to use it. Don’t forget to get management to give their two cents worth. They will give you honest feedback on reports required etc. also, include your Auditing Dept. to make sure the solutions will meet their requirements.

9. After the test, evaluate the solution with the same criteria for each solution: Make a list of requirements and grade them all with the same criteria.

10. Check the cost against what you can afford, and remember, the cost will go up 10-20% each year due to the newer technology will give you more visibility into the network.

11. After running the system for a year, re-evaluate the solution: Did it do what you thought it would? Does it meet your needs? Do you need to enhance it?(buy more modules) etc. or do you need more training.

When speaking SIEM it should be (probably) one of the last solutions that with you will reinforce your Cyber Security Defense. Remember SIEM is not EQUAL to secure.
I don't think that a PoC is something that will help you with a purchase: All SIEM today are intelligent, with powerful correlation engines, rules, and options. You need to find documents on the internet showing differences:
www.esecurityplanet.com

One important question you should ask and find the answer: budget (sizing)? Do you really need a SIEM solution on-premises? What about the cost? What size of IT infrastructure you have: small, medium, big? You will need staff for management, incident handling, etc., it should be 24X7. I will not buy a SIEM just for 8 hours per day. What about SIEM as a service or CO-Managed?

If you already have those answers and need a SIEM on-premises, the following factors will make sure your SIM project is a success:
-Events Per Second (EPS)
-Storage
-Parsing
-Filtering
-Management
-Accountability
-Automation
-Forensics
-Threat intelligence
-Easy to use: friendly user
-Compliance

I suggest that you try an open-source SIEM like Graylog (not a lot of intelligence) but it will help you even for sizing, and options.

Hi Rhea,

When it comes to evaluating a SIEM solution, there is a bit of research and evaluation required from the customer or your end as well - these mostly includes answering questions like: What is the business objective that you want the SIEM to fulfill? Is it compliance? or threat hunting? Do you have enough resources to man the SIEM? and many more....there are few things that you need to evaluate on your end before going all out on vendors as to what there solution is capable of.

Here are some resources that will help you plan or evaluate a SIEM vendor in the most effective manner and help you answer the Why, What and How for your SIEM deployment:
- How much does a SIEM Cost: dnif.it
- Why you need a next gen SIEM: dnif.it

I agree with Chris and would like to elaborate even more. Understanding your own use cases before the POC is key to then generate the test cases you would like to evaluate.
1) What data sources are required to collect from to support this use case? Does the SIEM support collecting from these data sources? Does the SIEM only present raw log data or generate additional contextual information from these data sources?
2) What built in analytics are available in the SIEM to support my use cases leveraging these data sources? How easily can I customize the analytics aligned to my specific needs or environmental/organizational nuances?
3) How easy is it to interpret results and to differentiate from other observations/alarms generated by the SIEM?
4) How actionable are results? Meaning, how quick/easy is it to advance the investigation to the next step? How easy it it to pivot on search results and/or lookup additional contextual data (what is the reputation of the external IP? what is the role of the host and its vulnerability state? who is the user? etc)
5) What guidance/capabilities does the SIEM have to lead or even automate steps of the investigative process for my use case?
6) How can I perform a retrospective on how the use case was fulfilled? Does the SIEM capture the details of the investigative process to be able to self-assess and improve?

There is a lot of prep-work that can be done. I worked on a document about 10 years ago and have posted it here: www.dropbox.com for a while if anyone wants to get a copy, feel free. It is 10 years old but should get you off to a good start in prepping for a POC.

If you need a SIEM for compliance, connect as much log sources as possible from your production environment, and pay attention to storage architecture, parsing non-standard/non-typical sources, licensing moments for network devices and hosts.
If you need a SIEM for threat hunting or risk assessment, then pick up a small environment where you can generate a lot of logs/events with all possible severities. Then you can check system for ease of administration, tuning, creation of correlation rules, event/alarm handling, orchestration. Check for usability and preferrable behavior, analysts will spend a lot of time with a system after all.

Some options for doing a POC. These are some common things that come up from customers that we deal with. But there can also be a lot of others based on specific business needs.

* It helps if they have a clear objective on what it is they are wanting. So review questions like the following:
* Is its just logs from a select few systems or all systems like servers, databases, applications and desktops
* Are all the users and systems internal or are they also mobile and could be working from home or over the Internet from a café.
* What operating systems need to be covered ie Windows, Linux, Solaris, Mac OSX etc
* Do they want to collect syslogs from other devices like firewalls, routers, switches, wireless APs etc.
* There can be some discussions on Agents vs Agentless so there can be discussions on the procs and cons of these needs
* Do they have compliance issues they need to manage ie like PCI DSS, ISO27001, HIPAA etc
* What is it they are wanting from the SIEM in reporting, there can be a lot of options on static reports, dynamic dashboards, PDF reports, correlation of log data with other systems like ticketing systems.
* Do they want to run a SOC or just get reports if and when they want to look at something, do they have the resources to monitor things or do they need to also work with an MSSP.
* What sort of alerting and threshold reporting do they want to get
* Do they have complex network segments with multiple zones to collect and aggregate logs that they need to centralise to keep the logs away from the systems generating them and away from potential hackers.
* In general for those that are starting out for the first time, just stick to the core critical systems and collect logs from those systems until they understand more of what they are wanting to do with their SIEM collection and reporting. This helps to keep the project scope more controlled and confined so its easier to manage. As they learn more then they can grow later on.
* So once they have a clearer idea on what they are wanting then its looking to the vendors to download the software and see how well it works in their environment
* How easy was it to get a eval license, did the sales and presales support help you get going quickly
* How easy and quickly can you install the software and start to collect logs and then start to get some reports and visualisations on the data
* How easy was it to identify problems and security issues, and what sort of value is that to the business.
* How easy is it to rollout, many large corporates can have complex change control processes and can the software easily fit within these processes.
* Cost is always a component to any solution so how well does it scale for their business, does it have known costs or are there variable costs like per GB of storage which often bites customers as there is always more data than they expect.
* How well can the solution scale out to hundreds or 10s of thousands systems as the business needs change or the business grows.
* Can upgrades and license changes be done with minimal effort.
* What are the futures of the company do they invest in R&D to keep enhancing the product as there is always something new to do or OS version to support.
* How well does the vendor do support, do they only do internet only or do they allow you to talk to real person that can understand you.
* Does the vendor play nicely with others, almost all customers have a mixed environment so being able to integrate and work other SIEM vendors always helps.

So having a clearer understanding of what they are wanting makes it easier to see “ yes this was a success” or ”no this was a failure” and did not meet the business objectives.
Some use a scoring system in a spreadsheet to rank various areas from a scale of 1-10 with 1 being poor and 10 meets all needs. By doing this in a matrix it often helps to sort the good and bad more easily and the good from the very good as part of the review process. So having a bit of structure to the evaluation process helps with finding the right fit for the business.

Yes, SIEM solutions are intended to have two types of values:
Provide a comprehensive and accurate view of an organization’s security posture
Correlate and intelligently analyze security events to find (and act on!) threats that would otherwise go unnoticed

So, integration and ease of automation are every bit as important (more important) then a
big list of features.

There is no easy way to “trial” an SIEM solution and get any idea other than how hard the
initial integration into security event feeds is going to be. Unless the solution has built-in
automation/learning (like LogRhythm and QRadar), you just won’t get that insight for a long
time. Weeks to months.

Biggest mistake is to believe a short-term trial equates to long-term experience. My clients
for whom I’ve done SIEM solutions have opted for efficient and automated - LogRhythm and QRadar
for every one of them. They are all delighted. My clients who chose their own path for SIEM
ended up with complex solutions from ArcSight, AT&T, NetWitness, etc… Splunk is becoming
a full SIEM solution and also deserves some careful consideration.

I recommend NOT trialing the different SIEM solutions in a bake-off. You will be comparing
several unbaked cakes, although I pushed that analogy to its absolute limit. Instead, find
a solution that is affordable and offers ease of integration and correlation/detection automation
that your organization values. Do a POC with that solution. If successful, do a fun deployment.

Or, just buy LogRhythm or QRadar ;-)

The SIEM field is changing very quickly. Do a thorough research before you start and only use fresh sources. 2 years ago User Behavior Analytics was a separate field, now it is part of SIEM. Decide what are the most important features for you - traditional log management and reporting, Analytical abilities, Automation and orchestration? Select vendor candidates based on your choices.
Identify what data sources shall be feeding information into your SIEM. Firewall logs, remote access gateways, proxies, financial applications, authentication systems, IDS - what have you. Select environment that is small enough so you can wrap your hands around it but representative enough to have different types of sources.
Setup a POC environment to specifications of vendor you are testing, provide feeds from one device of each source type (one firewall, one remote access gateway, one proxy, etc.) Review data as it comes to SIEM, configure some alerts, evaluate how many false positives you get. If you have capabilities - setup POC with 2 competing vendors at the same time, compare them side by side. Expect that one vendor will better at one thing, another will be better at something else. Then select a winner.

Doing a decent POC for a SIEM is a pretty easy thing to do from a technical perspective .. but it can be dang hard from a Security Management perspective if the customers' risk assessment processes do not exist or just arent't mature enough. Then scoping the POC is almost impossible as the customer really needs to identify their crown jewels.

After you've gotten down and dirty with your customer and found out what they really want (SIEM can stand for a multitude of things, from a light and fluffy 'Security Information and Event Management' to a much more substantial, AI driven 'Security Intelligence and Event Management') you'll have to decide on a couple of things like your sources (logs, flows, scans) and on a decent scope of hosts involved, ideally based on the customers current risk assessment.

After that, just let her rip for a week, tune a little and review the results. Fairly simple, but then simple things can be very hard.

p.s. If you have to talk to the customer a lot about 'use cases' the odds are that the customer does not really do much information security based risk assessment yet. In that case you should rather sell them a risk assessment workshop than a SIEM POC.

p.p.s. 'Don't try to boil an ocean' is a very good piece of Advice, too.

I cannot agree more with what Chris said, however your first question should be:
Do I have enough money to afford the number of HC and level of expertise (min. 2 senior security researchers and 8 senior security admins) which is needed to run successfully a SOC with SIEM?
Or it is not better/cheaper to receive those services from an MSSP instead?

To set yourself up for success you will need a set of Use Cases defined for your organization (in reality this statement applies for any product POC). Without Use Cases, a SIEM will likely end up languishing in your environment. These products are far from fire-and-forget, however I see this all to often in the industry.

I'd also recommend having a Logging Standard (if you don't already) built prior as this will help you define how products interface and extract data from pertinent business systems.

In the case of SIEM products, understanding how easy or difficult it was to implement the Use Cases will help you determine the best product for your organization. For help with Use Case development I highly recommend searching for talks by "Ryan Voloch" on YouTube.

To help the POC be successful, don't try and boil the ocean with each product. Pick a small environment that you understand thoroughly and can wrap your arms around for the POC. Also make sure you have buy-in from any other teams responsible for managing systems in that environment.

As a side-note: vendors will appreciate you having the above complete prior to approaching them for a POC. A good vendor will want a level playing field against competing products.

Lite & quick tip:

1. Transcribe the goal that made you think about acquiring a SIEM.

2. Transcribe or transform this objective into activities that the platform should serve (usually these are the most basic).

3. Start by testing "your process" using an open-source or trial solution.

4. Mapping both positive and negative experiences (they will naturally tell you what should be important or unnecessary at the time/maturity of your company)

5. After that, just get the pros/cons and see which market tool (free or comm) fits your need.

Ensure objectives and goals are defined and agreed to with all involved
Understand the log sources and systems/hosts you want to integrate
Understand what event types you are most interested in the SIEM correlating
Start small and tune

Work with your SIEM vendor to define success criteria for your organization. Only you can define what is important to you.

First, check Gartner's magic quadrant and choose (if you had not done that already) the SIEM products of your interest.
Second, check the key features you need to test, like logs, network flows, vulnerability scan integration, etc.
third, define a brief list of your most critical/relevant devices to analize. Start with your corporate F/W.
Check the compatibility of your devices with the SIEM solutions you are planning to test. For the trial you must choose preferentially the SIEM's that support all the devices you choose in the third point.
Check if a CLOUD version of the tool is available. Some SIEM's offer a free trial period for their cloud-based solutions.
If not, you could try to download a all-in-one installation image, preferentially as a virtual appliance, if you have a virtual infrastructure for the deploy. This will make the deploy much easier.
Try to find a basic installation guide for the deploy. Don't forget to consider minimal resources required, network connectivity and space requirements.
Active Directory integration is strongly recommended.
After basic installation, try to redirect the logs to the siem repository.
IF you consider flows, integrate them to the analysis.
Create a tennant and a basic dashboard.
Check any additional modules/options you consider relevant for the test
Try the reporting capabilities
Review the default events/threats detected by the tool
try to set alerts
try to define your own correlations

Good luck !

*What's The Best Way to Trial SIEM Solutions?*

There is no silver bullet for the implementation of the SIEM, it depends on
the objective and priorities you are looking for.
However, a good point of start is to identify what are your critical assets
and see what are the chances you have to do a PoC with similar technology.

*There are so many SIEM solutions out there and so much vendor hype in the
market. Conducting an effective trial is really important!*

Definitively conducting a trial is important; however, it is highly
probable we do not have all the time and resources to test many products.
I suggest doing a paper-analysis as the first step to get a short-list.

A number of community members are currently evaluating solutions.

*Do you have any advice for them about the best way to conduct a trial or
POC? *

- Set the goals of the PoC.
- Identify what are the critical assets
- Identify our capacities (our technical skills and expertise of the
partner)

How do you conduct a trial effectively?

- Set the goals of the PoC.
- Identify what are the critical assets
- Identify our capacities (our technical skills and expertise of the
partner)

Are there any mistakes to avoid?

- Start a PoC without the support and approval of managers; the PoC
should be treated as a project because if not, we would not have the
resources.

Identify your requirements and budget first in order to narrow the field of SIEM's.

Read Gartner's Magic Quadrant report as it provides reviews of industry leading SIEM's at various price points outlining feature / functionality.

Understand what systems are going to be integrated with the SIEM including open API integration for future system integrations.

Issue an RFI to a short list.

Most suppliers will perform limited POC at no coat as part of an RFI.

If really serious in deploying SIEM solution, need to evaluate the SIEM Quadrant leaders. Also, check reviews on these solutions.

Best way to conduct POC is create a Test Environment and use your own use cases for better result.

Greetings, IT Security is becoming a Board discussion and CxO broadly. I think a fine assessment is required with all stake holders and departments. USE Cases and other Security controls including mitigations and Incident Response, should be taken into consideration, and once must have a clear roadmap especially with the presence of any Initiative such as Digital Transformation, embracing cloud (Which Cloud Fabrics !?) What are the medium and long terms objectives and risks. SIEM is becoming more and more complex; it is essential to ask any vendor about their Roadmap, upcoming new features and enhancements. In addition, to enriching the SIEM platform with threat Intelligence

To be honest, SIEMs are difficult to evaluate and it will eat up a lot of time and resources during the trial period. There are also different types of offerings that you can consider like cloud-based or appliance-based SIEMs, which may be a good way to filter the SIEM that’s right for your environment and start your evaluation. Also think about shared SIEMs or Managed Services as alternative options if you have some budget constraints.

You can create a checklist of features that you need, from the basic (i.e. interactive dashboards, ease of integration, Threat Intelligence), to the more advanced (i.e. Automated response, Behavior Analytics, etc.). Give each item on your checklist a score so that you can weigh in on each item as a measure of your decision. Don’t forget to factor in usability and support.

If you do your homework well and stick to a plan of action for the trial, you are all set to go.

Different SIEM solutions can be evaluated on trial bases by subscribing to
the services provided by OEM. Here I can guide about two SIEM solutions,
i.e. IBM QRadar and FortiSIEM.

For IBM Security QRadar, logged in to ibm.com with a valid IBM ID and
browse through the IBM Security QRadar web page. There you can subscribe
for a free 14 days trial. This will provide you a product's GUI with basic
functionality.

For FortiSIEM, with a valid ID, subscribe the demo on your registered
email for one month trial.

The first thing you need to do it begin figuring out what to truly want to accomplish with a SIEM. Every SIEM had different strengths and weaknesses so you need to know what is most important to you in terms of goals do you don't waste time looking at something that can't do the thing you need it to do.

It depends upon the location and project size, but you can launch a POC of our SIEM at no cost providing
you have a project that is currently scoped and budgeted within your SecOps team.

1. If course get an evaluation copy
2. Define the end goal
3. Script out the requirements first and vet them with NW and security engineers
4. Make sure you have constant vendor support
5. Give yourself a time limit. Sometimes vendors offer evaluations for too short a time
6. Review a finite number if products, otherwise you’ll never finish
7. Include both the guys that have been around awhile as well as new up comers, even products that have been around but not well known (they don’t tend to spend as much money advertising but are really good products)

Weigh your requirements. If two or more are head to head, then look at purchase, support, maintenance, training, and personnel costs

Yes, I would like to suggest you to start evaluate 2 to 3 SIEM products, needs to do the granular POC to test the Use Cases and verifying log method limitations or if there is any compatibility issues in regards to your log sources ( Logical domains).

I would also suggest you to select the following SIEM products for your assessment.