What needs improvement with Splunk ITSI (IT Service Intelligence)?

Currently, Glass tables in ITSI only display metrics related to KPIs. I proposed adding an option to show metrics related to entities. This would eliminate the need for custom SPL to achieve this functionality. Since KPIs already have an entity split feature, extending this capability to dashboards makes sense.

When we check the service analyzer, and we have custom inputs, there are issues. Sometimes our inputs are not taken or recognized. Alerts are not being automatically generated. Also, if someone comes and creates a maintenance window, we can't properly identify who created it. We have to create our own queries before we can identify anything.

One thing ITSI could improve on is the maintenance windows. I have a huge case where I had to implement something related to the maintenance window. If you try to look up the issues in ITSI, you have to check the incidents individually, and putting hundreds of hosts in maintenance can be a hindrance.

The UI could be updated. Some elements of the KPI section aren't where you'd expect. It looks like a website from 2010 or maybe older. You can't change some things, like if it doesn't word-wrap well. For example, if you have a long list of KPIs that exceed a character limit, you need to hover over them and wait for the HTML text to pop up to see which KPI it is. Packaging synthetic monitoring in ITSI would be good. I'd also like a complete package for doing health checks. It would also be nice if Splunk standardized the add-ons. Splunk relies on these add-ons that users build. It's like the App Store. People put time and effort into these custom things, and if they get big enough, Splunk will purchase them and take them over. For example, we have a custom Slack output. It'd be good if they put some effort into stuff like that because it's useful. Instead, we're putting custom wrappers around stuff, but why isn't this a thing produced by this massive platform that costs so much? They recently partnered with Cisco and don't have any plans to improve ITSI in that area. It feels like they could do more.

Some developers struggle to write accurate queries, often inputting incorrect text or using asterisks in the source or index, which can significantly degrade search performance and overwhelm the queues. To prevent this, I suggest implementing a system that warns users about incorrect syntax or automatically corrects errors, particularly for complex queries like regular expressions. While Splunk has existing add-ons, they are unreliable and do not provide accurate results. Improving query autocorrection and regular expression handling would be beneficial.

There should be entity conflict resolution, specifically regarding duplicate entities. There should be case sensitivity for various keys amongst entities, specifically host names. We need IT metrics-based indexes and more content packs. I know they are coming out with these features.

They should make it easier to use. Many people are new to it. It is hard and has a steep learning curve.

We're getting alerts with delays of maybe five minutes, however, we'd like to see real-time alerting in the future. From a predictive analysis point of view, we'd like to see emails corresponding to the alerts we get. That would be an added benefit.

Since ITSI is primarily used for monitoring-related services, it would be beneficial if Splunk offered pre-built dashboards or a drag-and-drop interface for creating custom dashboards. This would simplify the process for users, especially for monitoring basic services like Windows and Linux servers. Currently, Splunk doesn't provide this functionality, requiring users to write queries and build dashboards manually. Including pre-built panels would significantly enhance the value of Splunk for ITSI users. The end-to-end visibility in Splunk ITSI is limited and has room for improvement.

After upgrading Splunk ITSI from version 4.11 to 4.13, the analyzer stopped finding values for KPS and services. We had to manually deploy a script to resolve this issue.

In Splunk ITSI, thresholds automatically trigger incidents when a service value falls below the threshold. This prevents us from automatically triggering alerts for situations where the service value is within the acceptable range. We've identified this as an issue with the ITSI product and are working with Splunk for guidance on how to implement the desired behavior. While the overall Splunk documentation is detailed, the documentation for specific premium apps, like Splunk ITSI, is more brief. The technical support has room for improvement.

It would be advantageous to enhance the dashboard by incorporating sections for monitoring, service health, and a filter for the KPIs.

Predictive analytics, in terms of preventing incidents before they occur, still needs time to mature. I am not very, I would say, convinced of the prediction feature's capabilities. It does not have a release comparison on the server comparison feature. For example, if you have an application, and you introduce a new feature, and you're going to deploy it, then the release comparisons should show automatically or generate a report to show the impact of the feature on the overall application. It should show what you can do to optimize it.

One issue we have with Splunk Cloud is that the service team is sometimes not very helpful. This is because the team is outsourced, and they often cannot provide us with the information we need. This is a major complaint of mine, and it is unacceptable given the large amount of money we pay for the service. Splunk Cloud outsources its support team, and the people who are supposed to be helping us are not very knowledgeable. They often give us unhelpful or incorrect answers. The UI needs improvement. With real-time monitoring, we can have a service structure, but we cannot easily adjust the graphical interface. For example, if we have a long name or a 2005 feature, we cannot easily move it slightly to the right on the web page. This can be a real pain. Our large-scale system is noisy, making it difficult to pinpoint the exact cause. This is a trade-off for using Splunk as a central monitoring tool, as we cannot give everyone access to everyone else's AWS environment. We are investigating ways to reduce the noise, but I am not sure if it is a specific ITSI problem. Quality-of-life features have room for improvement. The search function and other features are fine, but there are a few UI changes I would make. For example, I would like to be able to extend the graphical user interface so that we can see the full name by moving the window around. It is currently difficult to work with. We can create a correlation search, but when we save the page, it redirects us to the search system. We should be able to save the page and stay on the page, which is a bit annoying. We have a lookup file, but it doesn't work very well. In fact, it doesn't work at all. I hope Splunk fixes this at some point. When we make a change, it completely wipes out the change. It also says to type in the search bar, click on what we need, and if we make a slight adjustment, it will completely wipe out the search bar and we have to start over. This is very annoying.

It could be a little easier to use with the thresholding. We've struggled a little bit with thresholding.

The solution is okay. I am not sure whether the current release has already moved to the new framework where instead of the glass tables, we can directly use the Dashboard Studio. It would be nice to have that integrated into the same framework.

If the product had some prebuilt machine learning features, it would add value to our use cases. It would be very good if the product had some in-built predictive analytics and future forecasting features.

Splunk ITSI could function even better, particularly when it comes to refreshing the service infrastructure. If we could have the option to go back not just sixty minutes, but also one or five minutes, it would enhance our capabilities. The service analyzer component is excellent, particularly the default analyzer. However, I believe the refresh time should be faster. If it also takes five minutes to complete, as suggested by the KPI requirements, then the refresh time should be significantly reduced. If the data doesn't load within five minutes, our service and KPI will not function properly. Therefore, it is crucial to make it faster. I would appreciate having more customizable dashboards to assist with in-depth analyses.

Microservices is the only area where Splunk ITSI can be improved. When things come from one EC2 instance to another, there's a lack of exposure to microservices, so we can't know what's happening. Apart from that, it's doing pretty well.

The data recovery has room for improvement.

Splunk ITSI lacks out-of-the-box solutions for enterprise users. Currently, everything needs to be created from scratch. In their next release, Splunk should offer API integrations with products like ThousandEyes, and AppDynamics, or some other network monitoring tools or dashboards.

There are no areas I can pinpoint that need improvement because the product is working well. It would be good if an interface was included in the next release.

We haven't come across any shortcomings. We'd like them to show more inputs on the dashboard. The Wizard should be easier to use.

The dashboard queries should be improved. More queries should be suggested in order to produce better dashboards.

Some of our customers occasionally require the development of the connectors when there are no native connectors so that we can develop in Python or for customer slash comments as well. If they could adjust that, it would be ideal.

Something that we did find with the product (they may have resolved since then), had to do with the ability to contextualize the data sources. For example, we might bring in data for 50 applications from one source, but for each one of those applications, we would have to set up a different data source connection. Because of this, I had to set up one connection each for application A and then B and then C, rather than being able to set up one connection and then segregate the data coming in for those dashboards. That was probably the biggest challenge that we faced. We also faced challenges relating to UI development — being able to get the UI the way we wanted it to look performance-wise. Some of the customization levels of the UI just weren't there.

ITSI could benefit from a security model that would allow operations team members to get involved in model building, KPI implementation, and model maintenance while maintaining appropriate segregation of duties. To date, all of our ITSI development is being done by our Splunk Admins, while our KPIs and much of the modeling work are managed by our Splunk developers. Future development of templates and ready to use add-ons could facilitate faster time to value, as many IT infra and even Packaged Application data models are consistent across organizations and could be plugged in easily.

The problem becomes the price, as Splunk is an expensive product. In some regards, it's not a large issue because when you compare apples to apples and not look only into the price tag, but, look at the infrastructure, the platform, office time, and the people that you need to operate the other products, you will see that it's not necessarily an expensive product. It may even be cheaper than the others when looking at the bigger picture.