What needs improvement with ELK Logstash?

The SIEM modules in Logstash, need more improvement. In the future, the modules could be more advanced as right now there are only very basic modules. We are facing an issue with the engineers. In the region, there are not many available. Only a few people might be available in our particular region, which is a problem. There isn't really a very good user experience. You need a lot of training. There is an interface with limited options. You need to work with the coding and you need to work with the scripts. It's not simple. If you want to configure something, for example, you need to be a proper programmer. It would ideal if they had a dashboard. Right now, the only way to see what you need to see is to go through all of the logs.

The price of this product could be improved, especially the additional costs. I would also like to see better-quality graphics.

We are paying dearly for the guy who is working on the ELK Stack. That knowledge is quite rare and hard to come by. For difficulty and availability of resources, I would rate it a five out of 10.

The troubleshooting or diagnostic tool can be improved to provide a better understanding of internal behavior and how data is stored. It would also be helpful if they were to release the next version as a plugin or an extension, or as a JAR file, for the latest features. When releasing a new version they currently provide a new stack which means everything needs to be removed before the new version is installed.

Using ELK the first time there was a lack of security. We had to buy the paid version due to the fact that we needed to secure access to Kubernetes. The problem with ELK is it's difficult to administer. When you have a problem, it can be very, very difficult to rebuild indexes. In fact, you have to monitor the stack and it's very, very difficult. Sometimes we lose indexes or we have nothing on the dashboard.

The solution does not have a UI and this is one of the reasons we are looking for another solution. When setting up some of the pipelines we are receiving different types of log messages with different patterns. When I try to force a certain pattern I need to restart the solution causing a huge inconvenience for us.

Sometimes, the solution isn't the easiest to use. The solution probably doesn't have all of the advanced machine learning like some other SIEM providers have right now. It's something that could be improved upon.

This type of monitoring is not very mature just yet. We need more real-time information in a way that's easier to manage. We need to be able to monitor from any location in the world and any location in the company. We find that solutions such as Dynatrace and Datadog offer much more functionality, perhaps due to the fact that they are more mature. The solution needs to integrate more AI capabilities, specifically to assist in anomaly detection. The instrumentation of APM can be enhanced; can be better. It's not automated. It's a very manual process. This ends up being more costly for us. Dynatrace and Datadog are better in this area. The support on offer could be much better.

We're using the open-source edition, for now, I think maybe they can allow their OLED plugin to be open source, as at the moment it is commercialised. We are planning to go into the production to use the enterprise edition, we just wanted to check how this one works first. I think maybe on the last exercise part, I think the index rotation can be improved. It's something that they need to work on. It can be complex on how the index, all the logs that have been ingested, the index rotation can be challenging, so if they can work on that. In terms of ingestion, I think they should look at incorporating all operating systems. It should be easy to collect logs from different sources without a workaround to push the logs into the system. For example, in AIX, there's no direct log shipper so you do need to do a bit of tweaking there.

The process of designing dashboards is a little cumbersome in Kibana. Unless you are an expert, you will not be able to use it. The process should be pretty straightforward. The authentication feature is what we are looking for. We would love to have a central authentication system in the open-source edition without the need for a license or an enterprise license. If they can give at least a simple authentication system within a company. In a large organization, authentication is very essential for security because logs can contain a lot of confidential data. Therefore, an authentication feature for who accesses it should be there.

The solution is lacking some features of AI and machine learning. There may be a feature out there we are not using or maybe it's on a different solution, however, having more AI would be so helpful for us. The solution needs to be more reactive to investigations. We need to be able to detect and prevent any attacks before it can damage our infrastructure. Currently, this solution doesn't offer that. I know there are some features which are coming, and which is already available. To be honest, I haven't had any time to play around and check what could be the advantages of them. Compared to other products, already the features available - and there are lots of things which are provided - are quite useful. We are not managing it. We're only using it. For us, if we had the technical skills to manage the solution, we might be able to see and understand a few features that we're not already taking advantage of.

I would like the process of retrieving archived data and viewing it in Kibana to be simplified. We ran into trouble once or twice regarding problems with timestamps that came about because of issues with memory. Consequently, the correct data was not logged and it had to be done again.

Our system architect has noticed a slowdown of the solution, but I don't see a slowdown. One thing they could add is a quick step to enable users who don't have a solid background to build a dashboard and quickly search, without difficulty.

In terms of what could be improved with Elastic, in some use cases, especially on the advanced level, they are not ready-made, so you'll have to write some scripts. This is the case, especially with a trade. If you are comparing it with a SIEM tool, you don't have ready-made use cases. I would say that to have a better place in the market they should have more built-in use cases so that rather than people creating them, the prime uses had inbuilt use cases. It could even include more templates or automation.

Configuring the server is difficult and can be improved. I would like to have a high availability set up that is easy to configure. Anything that supports high availability or ease of deployment in a highly available environment would help to improve this solution.

This solution cannot do predictive maintenance, so we have to build our own modules for doing it. It doesn't do advanced analytics. They should have some advance analytics in this solution. With Kibana, we wanted it to be easier to use. The data visualization is there but it should be easier to use. Also, they should start proving APIs for doing ML and AI.

There are connectors to gather logs for Windows PCs and Linux PCs, but if we have to get the logs from Syslog then we have to do it manually, and this should be automated. It would be good if I could get technical support for specific devices. I think that Windows should have some specific connectors. When we implemented a new product, we had to create it manually.

The documentation for this solution is very important, and more needs to be developed. It was not as good as we expected, and because of that, we prefer to work on commercial solutions such as Splunk or ArcSight. If the documentation were improved and made more clear for beginners, or even professionals, then we would be more attracted to this solution. As you gather more and more data, and the data continues to grow, I think it is difficult to handle, administer, and perform declustering. I would like to see support for machine learning, where it can make predictions based on the data that it has learned from our environment.