We are developing a SIEM application that is similar to QRadar, ArcSight, or Splunk. This application uses Elasticsearch as its search engine because we want to retrieve information fast. We are just using the basic search engine part of Elasticsearch. We have developed lots of things on top of Elasticsearch, such as security, correlation, reporting, etc.
The search speed is most valuable and important.
Its licensing needs to be improved. They don't offer a perpetual license. They want to know how many nodes you will be using, and they ask for an annual subscription. Otherwise, they don't give you permission to use it. Our customers are generally military or police departments or customers without connection to the internet. Therefore, this model is not suitable for us. This subscription-based model is not the best for OEM vendors.
Another annoying thing about Elasticsearch is its roadmap. We are developing something, and then they say, "Okay. We have removed that feature in this release," and when we are adapting to that release, they say, "Okay. We have removed that one as well." We don't know what they will remove in the next version. They are not looking for backward compatibility from the customers' perspective. They just remove a feature and say, "Okay. We've removed this one."
In terms of new features, it should have an ODBC driver so that you can search and integrate this product with existing BI tools and reporting tools. Currently, you need to go for third parties, such as CData, in order to achieve this. ODBC driver is the most important feature required.
Its Community Edition does not have security features. For example, you cannot authenticate with a username and password. It should have security features. They might have put it in the latest release.
I have been using this solution since version 1.0.
For a one-node installation, it is easy. You can do it and retrieve information fast, but when you are trying to scale up, everything becomes complicated. If you want to deal with several terabytes of data, you should read whitepapers or case studies or get proper consultancy from Elasticsearch. Otherwise, you will lose data. I know many customers who lost their data and could not recover it. It is not like you store everything and search for everything, and it is just instant. It is not like that. You should do your homework very intensively. It looks easy, but when you scale up, it gets complicated.
We got 60 days of development consultancy with them. Until we sign the agreement, they were quick and prompt. After the signature it changed. Overall experience, we are not satisfied with the development consultancy.
We switched from SQL Server to Elasticsearch. For our application, we wanted the information very fast without locking everything. In SQL Server or Oracle, that would not have been possible. Deleting is also very difficult in SQL Server.
Its initial setup is straightforward. There were no problems.
We are using the Community Edition because Elasticsearch's licensing model is not flexible or suitable for us. They ask for an annual subscription. We also got the development consultancy from Elasticsearch for 60 days or something like that, but they were just trying to do the same trick. That's why we didn't purchase it. We are just using the Community Edition.
We evaluated other products and chose Elasticsearch because the data that we are collecting is unstructured. Every log has a different structure.
The most important thing to keep in mind is that it is not as they advertise on their site. If you want to scale up and are looking for a big deployment, you must read everything. You also need support from the company itself.
I would rate ELK Elasticsearch a seven out of ten.