Elastic Search Reviews

The solution is a dashboarding tool that's useful for DevOps engineers for monitoring. The solution is like a log database. You can ingest into it anything you want and then find the value of the things you ingest. The solution can also be used to make reports.

The most valuable feature of the solution is its utility and usefulness. I use the solution to see the logs better or the error explained. The solution allows us to be more on top of the alerts for the logs. The solution makes passing of the logs easier and faster.

I would like to see more integration for the solution with different platforms. Sometimes, it's hard to understand what you need to send to Elastic Search.

I have been using the solution for two to three years.

Elastic Search is a stable solution.

More than 50 users are using the solution in our organization.

We use the solution's live data analysis for operations purposes. The solution also has a monitoring aspect. ElasticSearch is like a middleman between the PRTG and ITSM tools. It is easier to pass the information about the metrics or the full logs of the cloud platform you are ingesting in the solution instead of giving the output to PRTG.

The solution is deployed on the cloud in our organization. Elastic Search is something that comes after the projects are done. After implementing the project, we use the solution to have that project monitored. I would recommend the solution to other users.

Overall, I rate the solution an eight out of ten.

I'm involved in architecting and implementing Elasticsearch-based solutions, catering to various use cases including IIoT, cybersecurity, IT Ops, and general logging and monitoring.

The intention of this article is not to compare AWS Elasticsearch with Elastic ELK Elasticsearch and at the end declare the winner. Elasticsearch by itself is one of the coolest and versatile Big Data stacks out there. If you are planning to use it in your organization or trying to evaluate if it is the right stack for your product/ solution, this article offers some insights from an architect's perspective.

I'm not the right person to answer this question as I'm the service provider. My clients are the right people to answer.

The Spaces feature in Kibana is really useful. I can ingest all data and then offer multi-tenancy on a single stack to various departments (internal) or customers (external). This feature isn't available in AWS Elasticsearch, and Machine Learning isn't available either.

Other useful features such as Canvas (used to create live infographics) and Lens (used to explore and create visualisations using a drag-and-drop feature) are available only in Elastic's ELK Elasticsearch.

In the last 18 months Elastic has really caught up and also gone way beyond AWS by putting together all the missing components that make ELK Elasticsearch the most comprehensive stack in the entire Big Data ecosystem. Comprehensive because one stack addresses all of the three essential technical components of an end-to-end system: collect, store and visualise terabytes (and even petabytes) of structured or semi-structured data at ease.

Enhance the Spaces feature to make it fully multi-tenant by enabling role-based access control (RBAC) at a Space level rather than overall Kibana or stack level like it is currently.

Elastic needs to work on their Machine Learning offering because currently they have been trying to make it a black box which doesn't work for a serious user (a Data Scientist) as it doesn't give any control over the underlying algorithm. It's like a point-and-click camera vs a DSLR. The offering started with a single/ univariate anomaly detection on time-series data. Now, they have a multivariate which is good, but beyond this, we cannot build any other Machine Learning models, like traditional supervised models. Anomaly detection uses mostly unsupervised algorithms and also it is a very broad problem space for a black box to solve it fully.

Make index’s metadata searchable (or referenceable in search queries).

5 years

Elastic ELK Elasticsearch is one of the most stable Big Data engines and the simplest to maintain and scale. Redundancy is built into the design so there is no single point of failure. We can configure a DR easily and if something goes wrong, we can restore the system into a brand new cluster in hours.

Elasticsearch by itself is 100% scalable as scalability is built into the design like any Big Data system. We just have to add more nodes, and it scales horizontally and then redistributes the data into the new nodes, and the cluster becomes faster and agile automatically. Cross-cluster replication comes with a Platinum license. But this feature is highly exceptional and not a common need.

I have worked with all the flavours of Elasticsearch viz. Elastic.co's ELK which is popularly known as the ELK stack (pronounced as 'yelk'), AWS Elasticsearch and Open Distro plugins for Elasticsearch.

All (including Solr that comes with Hadoop) are built on a common underlying technology, Apache Lucene. The difference is the added features that I call 'batteries included'. To be precise, Elastic's ELK Elasticsearch, unlike others, comes with free enterprise-grade apps (called plugins in Kibana) and a bunch of cool and useful Kibana features. It also features a good deal of engineering automation conveniences built into the stack.

Did you know that the original founders of Elasticsearch are the folks at Elastic.co, the company that has recently transitioned to an open-core philosophy by design. But since AWS took the initial lead and started offering the stack as AWS Elasticsearch service it became more popular and a preferred option for the uninformed. Elastic, on the other hand, was busy innovating and adding more muscle to the stack that it is no more limited to being just the fastest search engine on the planet. In fact, the keyword ‘search’ in Elasticsearch is not relevant anymore and, moreover, it is misleading.

Initial setup is indeed straightforward and fast because it will mostly be a single-node cluster. But as the data volume grows and we start seeing a performance lag, the stack requires scaling (by adding more nodes) and a professional intervention for doing the right capacity design and configuration fine tuning.

It is always a good idea to engage a professional vendor to implement it right the first time and save yourself a lot of time in experimenting and trying to figure out the optimisation hacks and how-to’s all by yourself.

A stack like Elasticsearch that enables heavy lifting of the data effortlessly comes with its intrinsic yet obvious ROI. If one is not able to realise the ROI it means either the data is bad (garbage in, garbage out) or the stack is not implemented properly.

The basic license is free, and it comes with a lot of features that aren't supposed to be free! With a Gold license, we get Alerting (called Watcher) and some modest enterprise features. Note that if alerting is a must feature for you, you can install open-source alerting plugins like Open Distro Alerting or ElastAlert and avoid the Gold license cost. Active Directory integration, SAML, SSO, Machine Learning etc. come with Platinum license. The licensing is per-node and per-annum basis for an on-premise installation and for Cloud Elastic-managed service the cost is baked into the hourly pay-as-you-go fee. Kibana does not have a license, so it's free.

If you don't want alerting, Active Directory or LDAP integration and are good with native authentication, the basic license will suffice. The basic license also comes with many internal stack features, which are free. For example, data segregation into hot and warm storage, automatic configuration, and rolling over the index after achieving a certain size limit. 

SIEM (Security Information and Event Management) app is free. Also is another cool app called Uptime that helps us monitor the uptime of servers and web services. We can do this without any third-party licensing cost. Just turn on the apps, ingest data using Beats and the apps will start thriving. Over time they become mission critical to your business.

For example, the SIEM app will automatically populate the dashboards and allow us to monitor network traffic, successful logins, unsuccessful login attempts, and anomalous security events. All that comes off the shelf and is free. You'll pay a lot, on the other hand, for a traditional SIEM like ArcSight or LogRhythm.

Another free app called Infrastructure (formerly known as Metrics) helps monitor the server infrastructure by configuring light-weight data collectors called MetricBeats (for Windows systems) and AuditBeats (for Linux systems). The Beats will start pumping in all the system performance metrics into the stack and help monitor the memory, CPU and disk utilization.

I have worked with all the flavours of Elasticsearch viz. Elastic.co's ELK which is popularly known as the ELK stack (pronounced as 'yelk'), AWS Elasticsearch and Open Distro plugins for Elasticsearch.

All (including Solr that comes with Hadoop) are built on a common underlying technology- Apache Lucene. The difference is the added features that I call 'batteries included'. To be precise, Elastic's ELK, unlike the others, comes with free enterprise-grade apps (called plugins in Kibana), a bunch of cool and useful Kibana features, and a good deal of engineering automation built into the stack.

Moreover, the original founders of Elasticsearch are the folks at Elastic.co, the company that's built on open-core philosophy. But AWS took the initial lead and offered the stack as AWS Elasticsearch service catering mostly to search-engine use cases. But ELK, with all its goodness, is much more than a search engine! In fact, the keyword search in Elasticsearch is very misleading.

You can spin up Elastic ELK Elasticsearch fully-managed service either on AWS, GCP, or Azure, or have your own on-premises installation and dockerize it. Whereas the AWS Elasticsearch is available only on AWS. That's the hosting difference.

Elastic ELK Elasticsearch comes with a support-only subscription, and there are a lot of updates happening. Kibana is constantly improved and there’s a new release every two weeks.

I am an end user, and we use Elasticsearch for our logs. Specifically, we use it for security logs for our enterprise, including machines, networks, and endpoints, as part of our IT infrastructure.

We have been able to collect our live logs, which helps us run security operations more effectively. It has enabled us to identify false positives and detect real-time malicious activities in the network.

The security portion of Elasticsearch is particularly beneficial, allowing me to view and analyze security alerts. It serves as a query engine for the database, enabling us to analyze logs for potential threats.

An improvement would be to have an interface that allows easier navigation and tracing of logs. The current system requires manually inputting dates to verify alerts. A visual timeline that pinpoints possible anomalies would be beneficial.

I have been using Elasticsearch for approximately one year.

I would rate the stability of the solution as nine out of ten. It is very robust.

I would rate the scalability as either nine out of ten. It's a very robust solution.

I do not interface directly with technical support from Elastic. Another colleague manages that aspect.

Positive

We did not use any different solution before Elasticsearch.

I was not involved in the setup process. Our architects and technical officer managed it.

I am not directly involved with pricing or setup costs. While I know a portion is open-source, a paid version might be necessary.

It was not my duty to evaluate other options. The architects and chief technical officer handled those decisions.

For someone wanting to be a security analyst, Elasticsearch is a valuable tool. It helps organizations collect large amounts of logs from various platforms like Windows, Ubuntu, and Palo Alto Networks.

I'd rate the solution eight out of ten.

All my use cases have been based more on observability for IT operations. We deal with it in terms of metrics, logs, transactions, traces, and so on. 

In terms of enterprise, most of the use cases are based on search capacity within the company to find documents and relevant information. That is the main use case.

The most relevant feature for me is the platform capacity. I consider the capacity high-performance with a distributed model that can support it, and recently we are growing. 

Search is really powerful. All the search engines and the rules that complement them allow the users to create different kinds of administration for the platform. YOu can create synonyms or rules to better understand or to better detect partial search criteria. It's like an AI that boosts searchability.  

The platform has a powerful tool to correlate and create rules that understand what people will be searching for. 

All the community support that we have available from different users in the open source community is great. Everyone shares and publishes all of these different use cases. That makes the platform and the platform understanding really powerful for anyone who wants to implement a different case.

It is easy to set up.

The solution scales well. 

They have great integrations on offer. 

Maybe Elastic Search could improve the analytics part of the search so it can be more powerful to the user. It could help provide more understanding of what people are searching for. 

We'd like more user-friendly integrations. It should be easier for non-technical people to understand how to handle them. 

I've used the solution for the last four years or so. 

It's stable. We have on-premise and on-cloud deployments. It's stable on both. I prefer the cloud as I avoid the time it takes to manage the platform. However, both cases are stable.

It is a product that can scale well. It's not a problem. 

We have maybe 200 people on the product right now. 

I have experience working with technical support. They are good at responding to incidents. I have not had too many incidents, however, sometimes for probably technical questions in terms of platform performance, search, cluster distribution, and so on, I might reach out. 

My point of view is that the technical support is awesome. They are very responsive and they have a really high understanding. The team has a lot of people with a lot of technical skills and technical knowledge.

The initial setup is very straightforward. It's not difficult as well. 

As I use the cloud, all of the costs for me are based on customer needs. There is a fascinating calculator published in Elastic. That there is not a specific starting cost. It can move from $10,000 US Dollars per year to any price based on how powerful you need the searches to be and the capacity in terms of storage and process. That said, you can start with a small budget, implement the use cases, and start growing slowly.

I'd rate the solution nine out of ten. 

I'm a customer and end-user. 

In terms of use case, we combine a lot of things with Elastic. It's two platforms, so with Elasticsearch, we're using the Beats, Kibana, and Suricata. It's a query engine and we use the information from our sensors. It gets ingested into that and we use the resources to get everything put on our dashboards. If something is detected, alerts come up right away and it's very, very accurate. The more ingest it receives, the better we can respond to threats. It's not just Elastic or Logstash, it's a combination of those and other tools that we would apply towards our threat detection and prevention. We have a partnership with ELK.

The company provides excellent technical support and wonderful engineers, even their sales engineers are great. The dashboard is a valuable feature - it's awesome and very customizable. 

I would like to see more open source tools and testing as well as a signature analysis in the solution. I think that a lot of times when we go into a corporate environment where it becomes more add on features or an additional service fee, it typically draws away from that product. 

I think it would be cool if they could provide a couple of licenses that would be test bed licenses so that engineers and people with have their hands on the keyboard could test any new development. 

I've been using this solution for three or four years. 

It is a very scalable soluton. It is very easy and I would recommend it to anyone. In terms of users it's all tiered. Most things are from tier zero at egress point of any major large-scale network all the way down to the customer. We have roughly 200 users. And those would include analysts and real time threat analysts. 

I'm very satisfied with the technical support and would rate it highly. Sometimes there are issues because we are overseas and there is a six hour time difference which creates a lag. It's hard to get around that but they're very responsive. 

We had issues when we first did the initial setup, because our resources were limited because it was a test that it was a proof of concept. It meant the initial setup was somewhat resource intensive. The data NGS itself was an issue when we were trying to filter and pull that information. Again, a signature analysis would have been helpful here.

For anyone considering implementing this solution, I would say take a good hard look at your own infrastructure resources and scalability as you have to future proof everything. Whether it's scale or increase in customers building up through your actual hardware and your network infrastructure. You need to know it's capable of performing the tasks needed, because sometimes you outgrow yourself. So, I would say look at your resources and how it can be scaled.

I would rate this solution a nine out of 10. 

We use the product for log analytics and metrics features. 

We can easily collect all the data and view historical trends using the product. We can view the applications and identify the issues effectively.

They could improve some of the platform's infrastructure management capabilities. There should be better visualization and insights about the cost of the SaaS services, which are not effective. Additionally, there needs to be more native integrations to merge the data.

We have been using Elastic Search for about a year.

I rate the stability a ten out of ten.

It is a highly scalable application. We have 15 users in our management team. I rate the scalability an eight out of ten.

I have experience working with Splunk in the past.

The initial setup for the SaaS platform is quite easy. We took assistance from an engineer for the onboarding. Thus, it was straightforward for us. However, there could be a better integration with AWS.

I rate the process a seven out of ten.

I rate Elastic Search's pricing an eight out of ten.

By integrating Deepgram insights with the product, we've gained visibility into logging, service behavior, and cost optimization.

I rate Elastic Search a nine out of ten.

It's a cloud-based service. At that time, we were using AWS, so we could get the same Elasticsearch capabilities from AWS. It was mostly a PaaS service that we could access. We had the Elasticsearch specific server and database hosted on an AWS instance, and then we fed the data to it and tried to fine-tune the algorithm to give the necessary search intelligence that we needed.

We're not using the latest version. We're using a version that was released one year ago.

The whole organization has about half a million users, but at any point of time, a hundred users might be using it.

The AI-based attribute tagging is a valuable feature. It passes through text data and identifies the tag-words and keywords and connects them to various attributes in the whole system. The system was supposed to run through a lot of existing data in terms of which tag-words would reflect which keywords. There was a model built on top of that. We were building a machine-learning model, which passed through all of the data and did the necessary attribute tagging. We couldn't find attribute tagging in other services.

We initially tried to do it in-house, but we couldn't get the accuracy that we wanted. Elasticsearch was quite efficient in terms of getting accuracy with the limited amount of data that we had. We had 10,000 to 20,000 records. Based on that, we had a good amount of accuracy, which we were happy with. There's a lot we can do with customization.

The documentation regarding customization could be better. Other than that, Elasticsearch has very good documentation. We can get a lot of information from forums.

I have worked with this solution for six months. 

The solution is stable.

As far as what we could accomplish, it was scalable, but we didn't have a lot of data that needed to be processed. We had 10,000 records and it was scalable.

We have reached out to tech support when we have had queries, and they respond in time. We didn't have an escalation process, but we had a lot of queries.

We chose Elasticsearch because we could build a model in a short amount of time. It allows us to build a whole setup in one month and get 93% accuracy. Even if you look at the complex AI-based features that we built within a shorter span, we could build that model with high accuracy, which wasn't possible with other search enterprise vendors that we used.

Setup was a little complex, but we had in-house expertise.

The solution needs regular fine-tuning in terms of the data model. As we get more and more data into the system, the predictability and accuracy of the output keeps changing. On the application and DB side, it was fine. Not a lot of maintenance was required.

Deployment was done in-house.

The solution is affordable. Previously, we wasted a lot of time by building our own system, which we could have avoided by moving to Elasticsearch earlier.

I would rate Elasticsearch as eight out of ten. 

Elasticsearch provides a lot of possibilities. You need to understand your requirements and how Elasticsearch can fulfill them. Somebody might be looking at a simple keyword service or attribute tagging. If you don't understand exactly what you're looking for, you'll get lost in their options and waste a lot of time.

We are developing a SIEM application that is similar to QRadar, ArcSight, or Splunk. This application uses Elasticsearch as its search engine because we want to retrieve information fast. We are just using the basic search engine part of Elasticsearch. We have developed lots of things on top of Elasticsearch, such as security, correlation, reporting, etc.

The search speed is most valuable and important.

Its licensing needs to be improved. They don't offer a perpetual license. They want to know how many nodes you will be using, and they ask for an annual subscription. Otherwise, they don't give you permission to use it. Our customers are generally military or police departments or customers without connection to the internet. Therefore, this model is not suitable for us. This subscription-based model is not the best for OEM vendors. 

Another annoying thing about Elasticsearch is its roadmap. We are developing something, and then they say, "Okay. We have removed that feature in this release," and when we are adapting to that release, they say, "Okay. We have removed that one as well." We don't know what they will remove in the next version. They are not looking for backward compatibility from the customers' perspective. They just remove a feature and say, "Okay. We've removed this one."

In terms of new features, it should have an ODBC driver so that you can search and integrate this product with existing BI tools and reporting tools. Currently, you need to go for third parties, such as CData, in order to achieve this. ODBC driver is the most important feature required. 

Its Community Edition does not have security features. For example, you cannot authenticate with a username and password. It should have security features. They might have put it in the latest release.

I have been using this solution since version 1.0.

For a one-node installation, it is easy. You can do it and retrieve information fast, but when you are trying to scale up, everything becomes complicated. If you want to deal with several terabytes of data, you should read whitepapers or case studies or get proper consultancy from Elasticsearch. Otherwise, you will lose data. I know many customers who lost their data and could not recover it. It is not like you store everything and search for everything, and it is just instant. It is not like that. You should do your homework very intensively. It looks easy, but when you scale up, it gets complicated.

We got 60 days of development consultancy with them. Until we sign the agreement, they were quick and prompt. After the signature it changed. Overall experience, we are not satisfied with the development consultancy.

We switched from SQL Server to Elasticsearch. For our application, we wanted the information very fast without locking everything. In SQL Server or Oracle, that would not have been possible. Deleting is also very difficult in SQL Server.

Its initial setup is straightforward. There were no problems.

We are using the Community Edition because Elasticsearch's licensing model is not flexible or suitable for us. They ask for an annual subscription. We also got the development consultancy from Elasticsearch for 60 days or something like that, but they were just trying to do the same trick. That's why we didn't purchase it. We are just using the Community Edition.

We evaluated other products and chose Elasticsearch because the data that we are collecting is unstructured. Every log has a different structure.

The most important thing to keep in mind is that it is not as they advertise on their site. If you want to scale up and are looking for a big deployment, you must read everything. You also need support from the company itself. 

I would rate ELK Elasticsearch a seven out of ten.