The challenges we were looking to address were mainly around making sure that my team wasn't overloaded with alerts and that we could tune out things we don't care about or that aren't important to us at that particular time. That was really what I was trying to accomplish, since I knew I wasn't going to be able to build out a team large enough to be 24 by seven.
Before we really had anything in place, or when we didn't use them in a managed way, we felt overloaded with issues like: "How do we deal with all these alerts?" "Is this a problem? Is this not a problem?" "Are there other customers that are also experiencing this?" It was pretty easy for us to justify paying a little bit to get some help on those things and get the benefit of their experience with the other customers they have on their platform.
In terms of the transparency of data on the platform, what comes to mind is that I've asked them a few times, "Hey, we've got this weird alert that you've escalated to us and we don't really know what to think about it. Have you had any other customers that have experienced it?" Obviously they're not quick to say, "Oh, well, Company XYZ had the same experience," and for good reason. But when asked, they're usually pretty good about saying, "Yeah, we've had some other customers that found this, or we've worked with them to determine it was this or that." Some of that you get upfront, but there are times when you do have to prod to get more information about something. Once we learn more about it, it affects our security operations because we're pretty small. So if I know that a large organization has spent time on this and had other analysts looking at it, analysts who have determined it's this and that, I'm going to lean toward what they found. I often just don't have the resources to do that myself, or it may be because I have respect for the security organization of that company. It's definitely valuable.
Using CRITICALSTART has increased our analysts' efficiency to the point where they can focus on other areas of businesses. That's definitely been a benefit of the whole thing. Instead of worrying about every little alert coming in, we really only pay attention to the ones that we need to pay attention to, the ones that are escalated to us. Otherwise, we would just be thumbing through thousands of things that likely don't really matter that much.
We have different groups throughout our company that use the equipment that we give them in different ways. So we've reached out to CRITICALSTART to build out groups and we can update those groups ourselves with different peoples' usernames. That way we can say, "All right, Nmap for the engineering group is always allowed. Don't ever alert us about that," or perhaps we make it a low alert as opposed to high. But if it's any other group, or if a user falls outside of those groups, we want to know. And that's been really useful for us in bringing down the number of escalations to us, things that would pop up as "high" at 8:00 at night, because some guy's running Nmap or something similar.
CRITICALSTART also takes care of Tier-1 and Tier-2 triage. In terms of time saved, I've always assumed that if we did this ourselves, I'd have to have at least a minimum 24/7 staff, or at least a few shifts throughout the day to cover the amount of things that would have to be researched and looked at.
Outside of using the platform to manage alerts, the feature of the service that we get the most value from is being able to reach out to them and say, "Hey, we might go buy a SIEM," for example. They give us their overview of what's out there, what they've dealt with, what they integrate with, and what that looks like. That's been pretty powerful over the years for us.
And when it comes to the alerts, they get the number of them down and only alert us about what we really need to know about. We get about a dozen or so things escalated in a day. Most of those are low alerts.
We chat with CRITICALSTART's analysts back and forth with comments or when we escalate things back to them. Occasionally we'll open a support request for a feature or we'll have a question about something and we may converse with them over that. Their availability has always been pretty good, especially when it comes to escalating to the SOC directly. We get responses pretty quickly.
I've used the updated user interface about a half-a-dozen times. I felt like it was going to take a little bit of getting used to it, but it did seem like it was pretty quick. It had more of the data right in front of me that I usually want, as opposed to clicking around to go find it. So far I have nothing but positive things to say about it.
We've had a little bit of frustration with some of the alerts that we receive because they're not as high-priority for our type of organization, as we are very engineering-heavy. But I can understand from their perspective, if a bank were a customer, or some other organization that doesn't have a lot of heavy engineering folks who are in a command-line and running all kinds of tools, the service would be much more valuable to them. But that's one of the main frustrations we've had: Trying to find ways to tune that out so that we can say, "Look, for this group it's normal for them to run a ping or Nmap or the like, but if accounting does it that's a problem.
Also, it has frustrated us that they don't have a native Slack integration, because most things do now. That's something we've asked for, for years, and it just doesn't really seem like it's a priority. The workaround is that we just have it sent to an email and you can email into Slack. Of course, email through Slack is not very good, but that's our workaround. We set that up ourselves.
Where CRITICALSTART could potentially grow is on its internal compliance, and maybe how they disclose how they secure data. All of that could be a little stronger. I pushed them on that early on, and they did provide some information, but like I'm doing with us — we're ramping up our compliance efforts too — that's where I'm likely going to have to push them in the future to make sure that they're at least meeting the minimums that we have, because they are seeing data from our employees.
I've been using CRITICALSTART for four years now.
It's definitely not utilized as much as I would like because of other priorities that have come up. My team is pretty small, so we can only do so much. But we are ramping it up because some of those other priorities are no longer as much of a priority. We should have some more time to do it.
I want my team to get in there and just clean up a lot of the low alerts that are sitting out there, alerts that we looked at and just didn't care about or that weren't important to us. We just need to go in, close those out, and get them to update filters about that stuff so we don't get alerted in the future. There's a fair bit of that.
I lean towards evaluating their support as good. Occasionally, we have spoken with them about something we have had open for a while and have had to look for an update. But they're generally quick to respond, initially.
From a project management standpoint, I have always felt that CRITIALSTART was pretty good. When we first brought them on, and when we switched to different products, and even when we tried out some of their other products, they were pretty good on that score. We had weekly calls and it seemed like we were getting moving on things. I really don't have any issues or complaints there.
Their overall customer support is pretty good. I can only compare it to our company and the support we provide, which I feel works pretty decently. They're on par with our organization.
Prior to using CRITICALSTART we were just managing things ourselves completely, without any help. But we brought them on pretty early after the company's creation, so it wasn't too painful from that perspective.
From the time that we entered into an agreement to use CRITICALSTART until we were able to start actually using it — I don't remember it taking too terribly long. We used them for a different endpoint service for a little while. When we switched to the new one, I do remember thinking that it took a little bit longer than I would have liked, but when they came back and technically explained it, it made sense to me.
Initially there were some calls where they were just getting an understanding of the environment and the types of users we have. We voluntarily provided them usernames of folks who were more high-priority or the groups that we needed to really focus on.
But the setup was definitely straightforward. It was a couple months before we were really comfortable with the setup, from our perspective, and felt that it was complete.
There were four of us, from our organization, involved. My architect was leading the effort and then I and one or two other analysts were the ones who were looking at the alerts and providing feedback to them so they would know we didn't care about this or that issue and that they should filter it.
When I start thinking about if I were to try to light up a SOC, which I've done before and I have no interest in doing, it could be a million dollars a year or more to do that. For what I am paying them for the managed fees, it's a steal. What I can get from them costs less than one body that I would hire. I've always felt like that it's a really good deal.
I've told CRITICALSTART that I think the managed service they provide is cheaper than it should be. It's a really good deal.
As far as using them to purchase software and other things that they don't necessarily manage for us, they seem to be pretty on-point with pricing. We've looked at them and put them up against Myriad or some others to see if we are getting good value, and they've always been pretty aggressive. In some cases, I feel they have been able to get us a bit more than another VAR would have been able to get us, because of the relationships they have. I feel pretty good about the value there.
Our expectations have been met when it comes to their services being delivered on time, on budget, and on spec.
We didn't evaluate other options. I have worked with the architect that I have for a long time now, and I know that he had evaluated options when he was at IBM. I didn't feel the need to, since he had just done it before he came on board with us.
The biggest lesson I've learned from using CRITICALSTART is that you don't necessarily need an internal SOC to make your customers happy. We get asked all the time on questionnaires, "Do you have a SOC?" We're able to say, "No, we use an external SOC to manage alerts for us." I've really only been pushed on that a couple of times. And at other times I've had companies that are larger than you would think come back and say, "Hey, we do the same thing." They may have an internal SOC too, but they still leverage a similar company to triage stuff before it even gets to their SOC.
I use CRITICALSTART's mobile app occasionally, although not as much as I did when I didn't have a dedicated person really looking through the alerts. It's mostly good. I don't have any major complaints about it. There are a few things here or there that need to be polished, but I think it's come a long way. The rest of the team is like me. They use it occasionally to pull up an incident that may be a higher risk, when they're running around doing things. But for the most part, we use the web browser.
On a daily basis there is only one person using CRITICALSTART. He's a security analyst for me. I'll occasionally jump in and my architect will as well, to help on the more advanced things or to adjust the filters and to do things that the analyst doesn't really do.
I would rate CRITICALSTART at eight out of 10. There's room for them to improve, but overall it's a good value and we're happy with them.
