What is our primary use case?
We use it for IT security and observability.
How has it helped my organization?
We did not have anything prior to this that could perform the same function. Previously, if we needed to trace a security event, we had to search across logs on multiple systems to figure it out. Since Splunk, we have got it all in one place, and we can dashboard that out and save searches.
It has reduced the time for root cause analysis. It gets us to the logs quicker, so it has reduced our mean time to resolve (MTTR). The time saved is entirely dependent on what the problem is, but it shaves a good hour or two off the initial investigation per incident.
It would improve our company's resilience if it was used effectively. It has helped the technology teams that do use it improve their business resiliency. It needs either evangelizing or being made more accessible to the front-end teams or departments that do not use it today. That is largely on us. We can do that in Splunk, but there is a never-ending list of things to do, and a part of that is building Splunk outs so that we can provide that centralized logging, and then give users access to it while maintaining the privacy of their data within our organization.
We have probably not seen any cost efficiencies. The benefit of any cloud platform such as Splunk, AWS, or Azure is that you do not have to look after it, but you pay a premium for that. For example, for VMware, you pay a premium for vCenter, vSphere, etc. You can do the exact same thing with OpenStack, but you need to hire five people to look after it versus two people for VMware. You pay for Splunk Cloud, but you run into other challenges. You do not own your data anymore because it is now stuck there, and you have to export to AWS, and then rehydrate into a different Splunk instance if you want to get access to it, or you pay through the nose for the data or retention history. It is horses for courses.
Do you want to host it yourself and save money on the OpEx but spend more on headcount and CapEx, or give it Splunk Cloud and spend more CapEx, but save money on CapEx and headcount? I prefer to have it on-prem. I prefer to go down the CapEx and headcount route because it gives me more control over my data, and it gives me more flexibility of my data. It gives me easier access to troubleshooting when something is wrong. It gives me easier access to scaling when we are seeing performance issues. I can bulk my hardware. It does not lock me into Splunk Cloud Platform. I know that Victoria promises some improvements around that with being able to manage my own applications and being able to have auto-scaling on search heads, but I will believe that when I see it, and I have not seen that yet, so I would personally prefer to put money in somebody pocket and food on their table than to give money out to a cloud provider.
What is most valuable?
I do not really like it, but being able to correlate events across platforms in a single place is valuable. I can trace an event back to its root cause. I can find the root cause instead of just looking at the symptoms across different things.
What needs improvement?
Its stability and performance can be better. Very rarely does a day go by when we do not see an error in the console, such as a health check error. Because it is cloud-hosted, we do not have access to the backend to figure it out ourselves. We are reliant on their support to figure it out, and a couple of days later, the error comes back or it is a different error. It is a never-ending cycle of support tickets. Their support is also not great.
In terms of performance, we are on the classic version of Splunk. We are not yet on Victoria or the new version, so we do not get auto-scaling. Therefore, we are limited. 90% of the time, Splunk is not doing anything. It is just reading logs, and 10% of the time is when we need to use it, but when we actually need to use it, there are five or six different teams trying to use it at the same time, and there are speed issues with search.
For how long have I used the solution?
I have been using this solution for about eight years.
How are customer service and support?
I could not interact with them very much, but I have people who do. It is not often a pretty experience. From what I understand or from the complaints that I hear, you are often told that this is not a problem or you have done something wrong, and then magically, it manages to fix itself an hour later.
Which solution did I use previously and why did I switch?
Before Splunk, we used distributed instances of Elasticsearch, Logstash, Grafana, and Graphite. This was ten years ago. Splunk was in its early days. Everybody had heard of it, but it had not become apparent why people need something like Splunk, so people had been building their own little instances. A lot of that still exists today in the organization because of the Splunk pricing model, the performance issues that we have on Splunk Cloud, and the stability. People want access to their data, but they also want to own their data. They do not want it to go into the black hole that is Splunk Cloud, so they keep it on-premises. They keep it in their own systems, such as Elasticsearch or Logstash, mostly because they can maintain sovereignty over data.
What was our ROI?
When compared to not having anything, we have seen an ROI. If we were going into it today, and that today was ten years ago, I do not think I would be at this Splunk conference. I would probably be at an Elastic conference and an Open Compute conference.
The value is definitely there, but it needs more performance around it. It needs to be more responsive. The value is definitely there in terms of a centralized point of visibility, but this value is provided by Splunk, as well as all of its competitors. Splunk potentially suffers from the same problems as ServiceNow, which is, if you want to do something clever with your data, you need a Ph.D. in data sciences to figure out how it works. It is hard to put in front of end-users who do not necessarily want to do something clever with their data. They want to be able to link it to the tools that they are familiar with.
What's my experience with pricing, setup cost, and licensing?
It is a touchy subject because we are locked into it. That goes back to the rehydrating data. We cannot have the retention that we want to store for legal and compliance purposes because that is seven years' worth of data for some of the indexes, so we ship them off into S3 buckets and install them there, at which point they are invisible to Splunk, so we have to rehydrate them, but we cannot rehydrate those pockets into Splunk Cloud. We have to rehydrate them into a self-hosted version of Splunk, which can take days to set up and get going. I would not call Splunk's licensing and pricing predatory, but they have made it very difficult to maintain the independence of your own data.
Which other solutions did I evaluate?
There are a few solutions out there that are similar to Splunk. You can get something similar with CloudWatch, BigQuery, Azure Monitor, and Azure Sentinel. In the cloud, Azure Monitor for the analytics platform and Azure Sentinel for the SIEM platform are the biggest competitors of Splunk. When you put dollars next to them, they all cost about the same at the end of the day. I probably would not trade Splunk for another cloud provider or another cloud-hosted solution.
We are heavily AWS compared to every other cloud. If that was not true and we were heavily Azure, I would probably move everything to Azure Monitor and Azure Sentinel to get that single ecosystem, but we are not going to live in that world. I also do not like AWS CloudWatch, so we are not doing that. On the cloud-hosted side of things, Splunk does not really have a competitor out there. Despite being very mature, Grafana is not as convenient as Splunk, but Splunk definitely has on-prem competition. Ten years ago, everybody was itching to get to the cloud. Everybody was pushing everything to AWS. It was like, "We have got to go to the cloud. We have got to be the first. We have got to be hybrid." Now, everyone is like, "I can do this cheaper in my own data center and have more control over it and not go offline every Friday when AWS East goes down." The competition for Splunk Cloud is with Splunk on-prem and probably Elastic on-prem, which is significantly cheaper and offers 99% of the same functionality.
What other advice do I have?
In terms of Splunk's ability to predict, identify, and solve problems in real time, if this capability exists, I have not seen it.
We monitor multiple cloud environments with it. We also have the on-prem environment and a lot of SaaS providers. We are largely dependent on the people who are deploying to the cloud. They are configuring their services and their platforms to talk to Splunk. We provide Splunk as a centralized service, but it is largely up to them whether they consume it or not. Some departments are eager to get in there so they can get visibility. Some want to build their own little greenfield internally, and some have not reached the maturity of realizing why they want it.
I would rate it a six out of ten. We have frequently run into many performance problems with it. The search is slow. We cannot scale it. We cannot troubleshoot it. We cannot get access to some of the functionality that we wanted, which is changing because we are moving to the new version. We also want to be able to manage our own applications. We are just locked into this parted sandbox, and we send our data off to it, and all of a sudden, it is no longer our data because it is trapped in the Splunk cloud. If we wanna get it out, it is going to cost us money. Their support is also not great, but it does provide single-pane access to data from a whole bunch of different places.
Disclosure: I am a real user, and this review is based on my own experience and opinions.