Splunk On-Call Reviews

We used it for on-call rotations. We used it to send alerts for monitoring. We also used it for escalation, so when we actually had an issue, it would find out who to call and call that person.

It's deployed on-premises.

About 200 people were using this solution in my organization.

The flexible schedule is the most valuable feature. It was very easy to set out a rotation. If somebody went on a holiday, it was easy to take over for those particular dates. In many other systems, you either have to take over for good and then remember when the person is able to take it back. But here, you just say that on a particular day, somebody else is covering for you. It was very flexible. The feature was called Override.

Our company used the solution to provide infrastructure monitoring, IT alerts, and incident management for one of our customers. The project ended about six months ago. 

The alert calling feature is the best because notifications are delivered via phone calls or text messages. We have 150 users who receive mobile alerts with no issues. We like the solution's calling feature better than competitive products. 

The dashboards are easy to use. 

The solution is used to track, communicate, and escalate issues.

The most valuable feature of the solution is helpdesk escalation. VictorOps has a feature that acknowledges the incident and escalates to the next level with a simple click of a button. Each level can continue to escalate the issue by clicking on the button.

The on-call scheduling and the alerting. It completely changed our organization as far as recognizing system-generated alerts and responding to them.

We have a hundred licenses. I have 93 users in the system right now. All of our teams that have on-call resources are using VictorOps with rotations, schedules and escalations set up. This way, our monitoring systems can use route keys to route alerts to the proper groups for escalations alerting. Typically most alerts go to my Tier Two SysOps team for initial triage, and then if they need to escalate to a Tier Three enterprise resource provider, the database, Windows OS, Linux OS, or network people can then reroute the VictorOps ticket to them. The value of that is that obviously once that reroute happens the alert starts paging the on-call guys within the appropriate group. This has helped us to drastically reduce our acknowledgement time.

My VP of Operations is ecstatic about the VictorOps product. He has visibility into what's going on, and he can tell that folks are responding to alerts. It drives a level of responsibility. We used to have a problem of folks who would be an hour from their computer when they were on-call. We don't have that problem anymore, because our escalation schedules work, you have the on-call user, the next user in the rotation, their manager, and if they don't all acknowledge, it goes to the VP of operations, and you really don't want him to get your page.

It's driven a level of accountability. When folks are on call, they're responsive, they know that the mean time to acknowledgement timeframe is being looked at and monitored. That's a hugely important aspect of the product. It drives accountability and visibility of mean time to acknowledgement, which is the first piece. The sooner you acknowledgement you have a problem, the sooner you get to the mean time to repair part.

The most valuable aspects of this product are the integration pieces. It is pretty much limitless, and I also enjoy the reporting aspects as I use this product to assist in RCA’s.

The most valuable feature of the product is how they allow you to do the scheduling compared to the other companies. I switched from PagerDuty to VictorOps because the time that it would take for me to create a full-on schedule for a single item would be anywhere from 45 minutes to an hour and a half in PagerDuty. That same schedule takes me no more than ten minutes in VictorOps.

That’s because with VictorOps, you’re in a single-pane window to create that calendar or that schedule, compared to PagerDuty where, once you set up a single session, you have to go back and set up another session, and that's your escalation policy. The escalation policies in VictorOps are: you have a drop-down menu where you add a task in that same window where you see all of your people and the times you're going to roll over the on-call; you're in the same screen the entire time. That's not the same as it is in PagerDuty, where you have to go to three separate screens to do the same function; it just takes so much longer.

Also, the transmogrification feature was really awesome. We use that quite heavily, so we can make sure the messages coming in are getting properly formatted. We're able to add whatever little customization that we want to for that type of message, so that the drops can accept it, and then give us valuable feedback based upon that.

Then, another great feature, because we were part of their beta system, is being able to do the calls and start a call within the timeline. That was awesome. Once you start a conference bridge, it would show the users and you notified the users within VictorOps, which was awesome. You could see in the timeline, for RCA purposes, who joined the call, when they joined the call, and if they typed out anything into the timeline concerning that incident that took place. We use it, I currently use it, for RCA tracking purposes. Any incident that takes place, we tag an incident number to it, and then we tell everybody that's associated with that incident to make sure that they put all of their findings related to that incident number and then we print out a report at the end as our RCA.

We are using this product to streamline RCA processes. We make all our engineers enter in notes to a specific monitor we have designated as the issue and then do cross-team collaboration to add notes to that problem. Later we can run reports on the entire timeline of the event so we aren’t having to do it twice. I can upload the logs to management for them to review what took place and who were involved in troubleshooting.

Obviously Net TGA and Net TGR are huge in the DevOps community, so one thing that we like to do is see what our NTGA and NTGR are per route, and try to figure out where we have gaps and why we have those types of gaps, whether we have an employee that simply just doesn't answer his on-call, which goes against your NTPA, and then ultimately against your NTPR. Then the reports, obviously for the RCA, our upper management expects to see those media files when issues arise.

It captures the incidents from New Relic and sends out notification when there is a critical alert.

Integration with Amazon CloudWatch and New Relic. Most of the AWS applications are monitored using CloudWatch but it cannot notify using sms/call. Integrating with VictorOps has helped the notification across global telephone numbers.

Transmogrifier and automatic solution report gives me a report with the solution and the way to solve issues when an error occurred. This reduces communication and error time significantly. It sends a report with the solution for you to perform.

It reduces the communication around CI/CD and production errors in about 90% of the cases and made our support tasks much easier.

We are very satisfied with the tool. We were using a similar tool, but the features were not enough for us. I did some research and found VictorOps, which we then tested. It was perfect, so we decided to go for it.

My team as well as other teams are now using the tool. As for me and my team, we use it mainly for our phone alerting. If there are a few non-critical incidents, a notification is sent to VictorOps which converts it to an email notification and then to a phone alert.

One thing which we would like is an improvement to the history log on the main page. There is some history, and I think you can filter it, but the filtering options are not sufficient for us. We would like to have more options. You can find some history records for the previous two or three days, but you cannot dig into them. For example, you cannot find logs that are a week old. Another example is that you cannot check the incidents created in VictorOps from the previous Friday. This is one limitation that they need to fix.

When I plan according to the schedule in the platform, for some reason I'm sometimes not able to change the on-call duty person. For example, if I want to remove a person who is currently on-call and assign the on-call duty to someone else from my team, it's not possible. I need to log out and log back in first. Maybe it is browser-related, I'm not sure, but this is something that causes me difficulties from time to time.