What needs improvement with Vectra AI?

Neither Vectra nor Darktrace have a function like a status health check on my log sources and traffic sources.

The detection algorithms can be improved at the sensor level rather than doing all the things at the brain. For example, if the sensor has some directional algorithm or detects repeating traffic, it can drop those packets at the beginning itself. There is no need to send that traffic to the brain in order to reduce the bandwidth. AI is picking up a lot now. There is no manual intervention needed. Whenever a detection happens, it can automatically summarize and give it to you. But Vectra doesn't have those kinds of capabilities. It still needs manual intervention to analyze, and they don't have a summarized kind of output. So that can be improved. But apart from that, the detection models and all the other categories have good support for that. In future releases, I would like to see Vectra AI to generate a summary of the instance.

It would be commercially beneficial if Vectra AI had something like Darktrace's Antigena Email or something similar to email protection.

Other alternatives, like Darktrace, have a fancier UI.

One area where there's room for improvement is the absence of a comprehensive TCP recording and replay feature. While there is an alternative method available, it doesn't provide the same functionality in a graphical interface.

The solution's marketing is not good. It probably needs to refresh its branding because a lot of it is confusing. People see it as an expensive tool for what it actually does.

We offer two solutions, Vectra and ExtraHop in the Qatar market. However, ExtraHop has better features that seem more advantageous when compared to Vectra. During demos, I encountered challenges with Vectra when demonstrating its capabilities, such as dealing with expired SSL certificates. Vectra AI is capable but ExtraHop is able to provide comprehensive insights and easier data querying. It excels in data query capabilities which is helpful for customers to access and manipulate their data effortlessly. This is where Vectra needs to enhance its capabilities. Customer support and handling high network traffic are additional areas that it needs to work on. There should be more flexible options to handle customers’ needs. Also, customers desire performance enhancements and integration capabilities with a single solution and cyber security.

A blind spot that I have is around the ease with which you can automate threat intervention.

When it comes to Vectra AI, some possible areas for improvement could be enhancing the accuracy and performance of the algorithms, providing more granular control over the detection rules and policies, expanding the coverage of supported network environments and security devices, and improving the user interface and visualization of the detected threats. 

Additionally, users may want to see improvements in the AI development process (https://omisoft.net/service/ar...) itself, such as increasing the transparency and interpretability of the machine learning models, ensuring the fairness and ethical considerations in the data used for training and testing, and enhancing the ability to customize and fine-tune the AI capabilities according to the specific needs of the organization.

We have had a few issues with the integration of Vectra AI with EDR. Some filters have not been working. We've also had issues with the brain not being powerful enough. In the next release, I would like to see more triage choices. From my point of view, Vectra is missing a lot of choices. This is an area that they could focus on. Vectra is also moving to a full cloud model, and I am not sure if going full cloud and leaving the on-premises environment is the way to go. We are not sure whether we'll move to the cloud with Vectra because it's hosted by AWS, which is one of our competitors. We don't like to work with anything that works on AWS.

We have a lot of system solutions and integrations with system solutions. Vectra is a type of black box. It implements AI-informed detection mechanisms, but we cannot create system detections. I understand that the product is designed this way, but it would be great if we could create our own detections as well.

The rules for threats are not always precise and Vectra AI should improve this.

What is most important for us is to have one place where we can manage a few brains because we are based on a zero-trust network. As a result, each customer needs to have a separate brain. For the SOC team, we need to have one place where the SOC analyst can go to visit the website and from that site manage all of the customers. Right now, Vectra AI doesn't have this capability, and I would really like to have this feature.

There is room for improvement in the documentation. We would like to have more details on how it detects what we see.

I think Vectra AI's automation, reporting, and integration could be improved.

In education as a sector, we are looking at AI a lot in terms of how it can be used as part of the teaching and learning side of things. It would be great to have Vectra AI look at a better way to enhance the security posture related to the AI tools in our portfolio.

The UI/UX and detection could be improved. More detections of specific security events could be useful. We've had a few incidents that were not detected by Vectra. The teams are working on it right now, but more detection is always better. Vectra AI is quite good at threat detection, however, it cannot respond to threats and attacks in real time by itself. It has to have plugins with other components, such as EDR or other software, to be able to respond properly. By itself, Vectra AI cannot do much, but it's powerful enough to pilot other software.

For S&D account scans, it would be easier if Vectra AI could triage with users. If a client uses a lot of accounts, then it could indicate that these accounts are benign, for example. That would help a lot.

Vectra Recall could be utilized much more, and I'm seeing some indications of that today with the investigative components. I use the visualize feature to visualize components and dashboards a lot. I'm interested in new ways to build automated searches or having them leveraged already from Vectra.

The solution needs to become more proactive. When Vectra AI is the primary solution in an environment - like it is in our case - we must work on response time. We have a small team so response time at the endpoint level is vital. At the network level, response time actually works with Vectra AI.

One of the things I am not so happy about when it comes to Vectra is the scoring board. In Darktrace, you can point or click on any client and see any connections that have been made directly in the dashboard. You don't have to go to recall. This is likely why Darktrace isn't as fast as Vectra, but it would still be nice to see this feature in Vectra. In addition, Darktrace has an advanced mode, but you are also able to see it directly in the main dashboard. This would be great to see in Vectra as well.

We got two problems that couldn't be solved because of the philosophy of the product. We are using SMB 3.0, which is an encrypted protocol. When we get some alerts or something, we cannot go deep into the protocol to see what's wrong because it's encrypted. We need to decrypt the protocol in another way, which is quite difficult. We might go back to SMB 2.0 just for this reason, but that's not a good solution. We did some penetration tests and tried to get some hashes or encrypted passwords from Active Directory. Those hashes didn't provide alerts into Vectra. Vectra doesn't survey them, which is quite problematic because it's a very common attack. They said that it's not the only aspect that would come with that kind of attack, but when somebody tries to get a lot of hashes, we would like that there is an alert because that seems like the start of an attack. For the hashes issue, it could be very easy for them to make the improvement. They can just change a rule, and that's it, but for encrypted protocols, it could be trickier.

One of the things that we are missing a bit is the capability to add our own rules to it. At the moment, the tech engine does its thing, but we have some cool ideas to make additional rules. There should be an option in the platform to add custom rules, or there should be some kind of user group where we can suggest them for the roadmap and see if they get evaluated and get transparent communication on whether they will be implemented in the product or not. I understand that not everything can be implemented in the product, but if everyone presses the plus one button, then you know that there's a need for it. There is the concept of groups within Vectra. You have IP groups, host groups, and domain groups. Wild cards would be very handy there, or side ranges would be a good one to start with. One of the big things that some of our operational people complain about is that if it's an IP and it has reverse look-ups, why do they need to make two groups—an IP group and a hostname group—just to get the same feature set?

The solution's ability to reduce false positives wasn't very good, initially, because it was picking up so much information. It took the investment of some time and effort on our part to get the triage filters in place in such a fashion that it was filtering out the noise. Once we got to that point, then there was definitely value in time-savings and in percolating up the high-risk events that we need to be paying attention to. I'd like to be able to get granular reports and to be able to output them into formats that are customizable and more useful. The reporting GUI is lacking.

We would like to see more information with the syslogs. The syslogs that they send to our SIEM are a bit short compared to what you can see. It would be helpful if they send us more data that we can incorporate into our SIEM, then can correlate with other events. We have mentioned this to Vectra. It does some things that I find strange, which might be the artificial intelligence. E.g., sometimes you have a username for a device, then it makes another. It detects the same device with another name, and that's strange behavior. This is one of the things that we have with Vectra support at the moment, because the solution is seeing the device twice.

An area for improvement in Vectra AI is reporting because it currently lacks some details. For example, when you download a report from Vectra AI, you won't see complete information about the alerts or triggers. Another area for improvement in the tool is that sometimes, an alert has high severity, yet it's marked as low severity. Vectra AI should have a mechanism to change the severity level from low to high or critical.

The reporting from Cognito Detect is very limited and doesn't give you too many options. If I want to prepare a customized report on a particular host, even though I see the data, I have to manually prepare the report. The reporting features that are built into the tool are not very helpful. They are very generic and broad. That's one main area that I keep telling Vectra they need to improve. Also, whenever there's a software upgrade and new detections are introduced and the intelligence improves, there is a short period at the beginning where there's a lot of noise. Suddenly, you will get a burst of detections because it's a new detection. It's a new type of intelligence they've introduced and it takes some time to learn. We get worried and we always check whether an upgrade has happened. Then we say, "Okay, that must be the reason." I would like to see an improvement wherein, whenever they do an upgrade, that transition is a bit smoother. It doesn't happen all the time, but sometimes an upgrade triggers noise for some time until it settles down.

If you hit a certain number of rules, triage filters, or groups, the UX responds more slowly. However, we have a complex network and a lot of rules. So, our setup might not be a typical implementation example. We even had UX engineers onsite, and they looked at issues, improvements, and user feedback. Since then, it has gotten a lot better, they even built in features that we specifically requested for our company. We know that Vectra AI sensors for cloud IaaS deployments have been released and we are planning to deploy those shortly.

They use a proprietary logging format that is probably 90% similar to Bro Logs. Their biggest area of improvement is finishing out the remaining 10%. That 10% might not be beneficial to their ML engine, but that's fine. The industry standard is Zeek Logs or Bro Logs, or Bro or Zeek, depending on how old you are. While they have 90% of those fields, they're still missing some fields. In very rare instances, some community rules do not have the fields that they need, and we had to modify community rules for our logs. So, their biggest area of improvement would be to just finish their matching of the Zeek standard. They could provide distributed endpoint logging capability. We have a lot of remote workers nowadays in the day of the pandemic. If they're not connected to our VPN, then we're not capturing that traffic. So, the ability to do the traffic analysis for endpoints that are distributed would be cool. I have no idea how they would do that. I'm not aware of a single vendor that does that, but it would be cool if they could do that. To my knowledge, that's not really possible with the amount of compute power it would take on endpoints. It would be ridiculous. They'd have to really invent something new and novel that doesn't exist today in order to accomplish that. If they do, that would be great. Because I'm a customer already, I would use it. Cost-wise, they're not cheap. They were definitely the most expensive option. Their licensing model is antiquated. We have to pay for licensing based on four different things. They need to simplify their licensing down to just one thing.

Integration with other security components needs improvement. It should have true integration as opposed to just being a separate pane of glass.

The solution’s ability to reduce false positives and help you focus on the highest-risk threats is mostly good. It is still a bit of work in process, but I can give feedback to the company from the help desk. There is follow-up from the Vectra team who follows it closely. We can also give a lot of inputs to make it still a better product. It's already a very good product, but in comparison with a lot of systems I used in the past, the false positives are really a burden because they are taking a lot of time at this moment. The Office 365 integration is still a pretty new feature. I also have seen some improvements, and they email us with every step in the improvement process. I think that this integration will grow. Every area has room from improvement. Security is an ongoing process. It is important for Vectra to keep updating their system based on new behaviors. We would like to see the combination of the cloud with on-premise, e.g., what's happening in the cloud versus what's happening in the on-premise situation. If there is a phishing mail in the cloud, then the phishing mail comes in and a colleague clicks on that mail. Normally, it would be blocked by the system. However, when it's not blocked, then there can be malware on the system locally. We think it's important to get the integration of what's happening on Office 365 with phishing mails. Sometimes, it is a bit noisy on the dashboard because all the systems are on one field. On the dashboard, we have a complete overview of high, medium, and low risks. However, it would be more interesting for us if they could split that dashboard into high, medium, and low devices. For example, there is a dashboard on a device with a complete overview specifically for high-risk.

Vectra is still limited to packet management. It's only monitoring packet exchanges. While it can see a lot of things, it can't see everything, depending on where it's deployed. It has its limits and that's why I still have my SIEM. I am in contact with the Vectra team, if not weekly then on a monthly basis, to propose improvements. For the time being, the main improvement I can see would be to integrate with more external solutions. Since Vectra provides an API, that should be quite easy to handle. For example, we're using an open source ticketing system within our team and I want to have it handled properly by Vectra. We'll go forward on that with the API. Another area for improvement that I have pinpointed is that the Office 365 solution and the Detect solution cannot match the same users. That means we have two "different worlds" currently, the world from Office 365, which is bringing alerts based on users' emails and email addresses. And we have the network world, which is bringing an Active Directory view. On the one hand we are seeing emails or email addresses, and on the other hand we are seeing things like logons on to the domain controller. From time to time, it does not match and the tool cannot currently cross-check this info and consolidate everything. I would like to be able to see that detection related to one workstation and covering a user: what he is using, what services he is using, and what he did with his Office 365 and configuration. That would help. Another major feature would be to have all logs pushed to Cognito Detect, and all these logs should be also pushed to Recall. Currently, within Recall, I can't call up the Office 365 detections and I would love to do so. The last point would be an automated IoT threat feed consumption by the tool.

The false positives and the tuning side of it are some things that could use improvement but that could be from our side. I don't want to criticize the product for performance with our role out of it. It does what it says it's going to do very well. We've got issues with the way we've deployed it in some places, but the support we've had in that is very good as well, so I'm very happy with the support we get.

I would like to see a bit more strategic metrics instead of technical data. Information that I could show to my executive management team or board would be valuable. I would like to see some improvements on the integration aspects of it. They are getting better in this. However, most organizations have a plethora of cybersecurity solutions that they run, and I think that there is a bit more that could be done on the integration side.

It does a little bit of packet capture on alert so you can look at the packet capture activity going on, but it doesn't collect a whole lot of data. Sometimes it's only one or two frames, sometimes it does collect more. That's why they have the addition of their Recall platform, because that really does help expand the capability. I would also like to see more documentation or user guides about using the product.

Some of their integrations with other sources of data, like external threat feeds, took a bit more work than I had hoped to get integrated. I think the company has been very responsive, willing to take our feedback, and look at addressing our concerns. I have asked that they give direct packets capabilities.

Some of the customization could be improved. Everything is provided for you as an easy solution to use, but working with it and doing specific development could be worked on a bit more in the scope of an incident response team. In my opinion, it's built as a solution for everything, instead of it being part of a bunch of other tools. For example, we have a source solution which will orchestrate the ability for us to use a host EDR and the ability for us to use Vectra. We see Vectra from a purely network standpoint. Therefore, we don't want it to be the incident manager where we have to fill in specific things to be fixed. We think the integration with source solutions could be better. It tries to treat itself as an incident resolution platform.

One thing which I have found where there could be improvement is with regard to the architecture, a little bit: how the brains and sensors function. It needs more flexibility with regard to the brain. If there were some flexibility in that regard, that would be helpful, because changing the mode of the brain is complex. In some cases, the change is permanent. You cannot revert it. I would like to see greater flexibility in doing HA without having to buy more boxes just to do it. Another area they could, perhaps, look at is with OT (operational technology) specifically. Vectra is very specific to IT-related threats. It really doesn't have OT in its focus. We are using another tool for that, but maybe that is another area they can consider venturing into. It's being used by my team of four or five people. Once we hand it over to operations, then the team size will increase significantly. It will grow to about 10 to 15 people.

You are always limited with visibility on the host due to the fact that it is a network based tool. It gives you visibility on certain elements of the attack path, but it doesn't necessarily give you visibility on everything. Specifically, the initial intrusion side of things that doesn't necessarily see the initial compromise. It doesn't see stuff that goes on the host, such as where scripts are run. Even though you are seeing traffic, it doesn't necessarily see the malicious payload. Therefore, it's very difficult for it to identify these type of host-driven complex attacks. It only shows us a view of suspicious behaviours. It doesn't show us a view of key or regularly attacked company targets. This could be because we don't have one of the other tools or products that Vectra provides, such as Stream or Recall. My challenge with the detection alerting platform, Cognito, is it tells us this host is behaving suspiciously and is targeting these other machines, but it won't give you a view when a host is the target of multiple attacks. This because you may have a key assets, such as domain controllers or configuration management servers. These are key assets which may get targeted. If you're a savvy attacker, you spread out your attack across multiple sources to try and hide them across the network. That is where the solution falls a bit short. It is trying to build that chain of relationships across detections and also trying to show detections from a perspective of a victim rather than the perspective of an attacker. I have expressed these concerns to Vectra and they are currently in as feature requests. There is another feature in place which takes additional data feeds, such as DHCP IP allocation data. Their inputs are taken from Windows event logs, and that's the format they have in place. They use that to provide them with a more accurate view of host identities. If you are only relying on IP addresses, and IP addresses change over time, it's sometimes very difficult to show a consistent view of a system behaviour over time, as the IP can change per month. Unfortunately, because their DHCP data is taken from Windows host events and our DHCP data is taken from a Palo Alto system that generates the IP leasing, the formats are incompatible. I think taking different formats for that type of data is something else we have a feature request in for. At the moment, we don't have an accurate view, or confidence, that they are resolving when an IP address changes from host to host. So, we may be missing an accurate view of risk on some of those hosts. We also have the same problem with VPN and Citrix. E.g., if you're on the network and on IP address A, then you come in via the VPN, you're now on IP address B. Thus, if you're spreading your suspicious behaviour across both the internal network and VPN, then across Citrix, we don't get to join all that information up. They are seen as three different systems, so it causes a bit of a problem trying to correlate that type of event data.

Room for improvement depends on how their strategy and roadmap develops, as they have a lot of third-parties that they integrate with, e.g., more orchestration around what alerts and what to do with afterwards. They don't pretend to be working in that space. That is a third-party type activity. There are always the little things that they could do a bit better, like grouping or triage filters. Clearly, they've taken that onboard and developed those over the course of the last 18 months to two years to put these additional functions in. My guys are constantly saying, "Oh, it'd be useful to do this and useful to do that." The solution has not reduced the security analyst workload in our organization because we still need to SIEM. Unfortunately, while Vectra, for us, is a brilliant tool for network investigations, giving wonderful visibility, it doesn't go the whole way to replace our SIEM that is needed for compliance. So, I still have the same amount of alerting and logging that I did before. It gives us more defined ability to see incidents, but it doesn't give us enough information to satisfy a PCI or 27001 audit.