Cisco Provider Connectivity Assurance Reviews

We had issues for a number of years with our carrier. I had dropped our bonded T1, 3 Mbps, to go to 100 Mbps fibre, minimum, to every location. We have 26 branches. Because of the scale and the magnitude of the capacity, it's fraught with not noticing whether we're getting our subscribed bandwidth, and we're paying a lot of money annually for our MPLS network. When I figured out the players like AT&T, and others in the mix, can have NNI, even though you think you have a 10 Gb connection—my backbone and my data centers are now connected to 10 Gb lines, my backbone is pretty much limitless—I wasn't sure what we were getting.

The scale of improvement from going to fibre, with this institution so used to having 3 Mbps, made everything look better. Except, I knew that when I need to put hundreds of terabytes of data across data centers and other components, I would have very specific expectations, and my intuition was that it wasn't even close. I was pretty much correct. We had video that was having trouble, and that was the "tell." When we started to look deeper, we had massive amounts of packet loss at higher capacities and smaller packet sizes. I've got a year or two worth of research at the highest levels. We needed a product such as the Accedian, that could knock it out—exactly where the issue is—in a matter of moments, and they were right. It's an extraordinary endeavor. I had an intuition that things needed correcting, and I was spot-on.

Accedian is a combination of SaaS and hardware devices. I have eight in my data centers and I have 26 branches, so the total we have is close to 85.

It has application and Layer 7 support, so that's going to be the next evolution. I had built an institution that I wanted to be number-one in Southern California, and I got bored and wanted it to be number-one in all of the US. When I talk to folks that used to work for, or who work for, Citigroup, they want to come work with me because we're doing things like NVMe capabilities of my sub-infrastructure that operates with 100 Gbps port speeds across state-of-the-art Aristas in a way that people can only dream of. 

As we prepare for the next five to 10 years, everything is based on network performance, latency, and application support. We have it set up in a way where we test the capabilities of the telco, the main provider, the local ex, the firewalls, and the switches right into our virtualized infrastructure in every single one of our locations. We virtually eliminate any finger-pointing. We know that if anything goes into maintenance due to an outage, and the failover is supposed to occur, we can instantly test it.

We called out AT&T on a 1-Gbps NNI that was supposed to support 60 folks—which in and of itself was a problem, because we're supposed to have 700 Mbps, guaranteed. When they failed over to a 100 Mbps, they completely saturated it and ran out of bandwidth. We were able to call them on all of this in real time. When we thought there should be 10 Gbps per 60 clients, we identified that they had unnaturally put all 60 through a 1-Gbps line.

The value is for networks on which hundreds and hundreds of thousands of dollars are being spent, where we have this level of intel. And we don't actually have to move around the network at a point in time, the way the world operated for the first 20 or 30 years of networking. We literally send traffic during the day and off-hours. We schedule these tests for repetitive performance, and the results are delivered in a digestible PDF report so that a non-technical person can see what passes and what fails.

It has saved hundreds and hundreds of thousands of dollars. You can pay that money, but there's no guarantee you're getting that capability. 

We also can tell instantaneously whether or not we need to consider justifying more throughput or if we don't need what we have. There are probably several ways to look at what that is worth to an organization. However, if you have a system that's out in the cloud world, whether it's budgeting or mortgages, try to put a dollar figure on the capability to ensure the systems are working beautifully at all times.

In addition, Skylight has improved the interaction between my network, data center, and application teams, 100 percent. The layer side of it will be the icing on the cake. I will tell you the user experience before the user can tell you the user experience. We're fixing things for the applications team, and they don't know how or why. We're taking things that, in the mortgage world, historically had epic low rates in production, and doubling them. Just this week, I did something which changed a process that used to take between 30 seconds and a minute and a half and made it real-time, instantaneous. It's extraordinary.

Skylight brings the firewall and security teams to be very in-tune with the network teams, the telco teams, and the carriers that do the local. Any one of those could be finger-pointing at the other. We pinpoint exactly what the issue is, we tell them where it is, and it's pretty irrefutable. There are a time savings and there is collaboration.

You start buzzing away with NetFlow out of SolarWinds, and they'll tell you 17 reasons why it's something else. It's like playing a good game of chess. You think about every move two years in advance and, when it comes down to it, you end up with $50,000 worth of free gear, including, but not limited to, removing traffic shaping and policies that you know are strangling the network. I liken it to having the very best medicine for bronchitis or asthma or any type of breathing condition. This would be the medicine you would want. It opens up the complete "breathing" of the network, right down to every component, whether you're streaming video or doing other things.

You can simulate VoIP issues. As we move the organization to enterprise SIP, my biggest concern in my data centers is that if we ever had one-direction audio, we would never be able to figure it out. It would just take an extraordinary amount of time. Now we can simulate it, understand it, and resolve it in really short order. In fact, I have an engineer who came from LA Fitness, with 720 locations, and he said he took over a year and a half to figure out a problem, and he could have done it in under a month with this software.

When we moved small packets over certain routers, they would absolutely fail, and the world would tell me, "Oh, well, that's the way it is," and I said, "That's not possible." I proved that with higher caliber gear. We were able to move packets flawlessly down to 0.1 and 0 percent packet loss, which is unheard of, and maintained it. I proved it and showed the result. And then I did that again to ensure that the memory and the processing capabilities of the next-gen firewalls that are in every single one of my locations, are capable. And that was right through my primary firewall, to make sure they're capable of sending and adhering to certain traffic loads.

When I joined the company four-and-a-half years ago, I changed everything on a massive scale, to the point where if a vendor calls up and says something may not be running as expected, I politely explain to them that it's not even possible and that they need to take a look at their coding. I took apart every single piece of the puzzle, including four brand-new core switches that run the entire institution, where before we had three that were 10 years out of date; 120 switches, all Cisco, all brand-new. There was no stone unturned. When we run Accedian, if something isn't right, I can tell you exactly what it is and why.

In terms of reducing mean time to respond, it's done so by years. I don't even know how to estimate it. There are things that could never have been solved. When they tell you that they have put Quality of Service on your network, it could take you a lifetime to figure out and identify that the packet traffic and the tagging is converted because you have poor policy that converts everything into a particular tag. No one will believe what I'm saying in terms of saving "years to a lifetime," but you just have to see what this stuff can do. When used properly, there's no telling what you can do. I can say "from a year-and-a-half down to a month" and that's a reasonable metric.

We are a company who specializes in analytics, both testing and analysis. We get involved in DDI: DNS-DHCP-IPAM. We do a lot of stress testing, protocol testing, and protocol analysis, so we use a number of different tools. We typically try to promote to our clients best of breed type products.

At the moment, we have a demo system with the whole suite. That would be VCX and PVX. We are using the Skylight centres to do link testing as well as analytics for multi-office simulations. So, we have set up a demo system within our lab to talk to a couple of home offices as well as look at traffic coming into our own office. Therefore, we have some setups that we can demo to our clients who want to do similar sorts of things, e.g., people who want to be able to text links or look at jitter and latency from one site to another.

Typically, people who have multi-office/multi-sites want to be able to link test and understand the latencies between each site. First, they want to be able to understand if there are any throughput issues. Second, from an analytics perspective, they look at application network performance from a centralised perspective.

We use it for demonstration purposes to provide solutions to our clients looking for this level of analytics and level of testing. That is our primary goal. While we use it to look at our own networks, it is mainly there as a demonstration type of tool. We have a number of tools that we use for demoes. This is just one of them. However, the fact that we have a demo system to show people is also the way we look at analysing our own network. We can look where things are being held up or being impacted, such as cloud apps, on our own in-house environment as well as at our connection to our public service provider.

We have a public cloud instance for all our other probes or SFPs. Then, everything else is on-premise, VM deployment. For example, even our PVX analyser is sitting as a VM. So, we have one big server that is partitioned out with different VMs to handle all different parts of the Accedian suite.

We are dealing with a couple of organizations who are using Skylight sensors, so the SFP based analytics to look at multi-sites. 

There is a large supermarket chain looking to invest and put into deployment a lot of sensors to look at different store performance, trying to understand latencies between stores, packet loss, and why certain applications are having issues. They are really looking into the link metrics of latency jitter, packet loss, and response times for multiple sites and stores to give them some insight into performance of networks, since one of the things they have got to work out is, "Do their internal networks have a problem? Is it their external networks? If it is their external networks, does it have anything to do with them, their edge devices, or is it the carrier who has the issue?" It is about the finding the demarcation between internal edge and carrier problems to understand who is at fault. This is sort of an example of where our customers get some value out of the solution.

It is about finding operational problems. When sites go down, we try to determine who is at fault. While there is not much finger-pointing, the solution is just trying to analyse when there is an outage and where do we start looking to fix it. The very nature of why organization chooses to use the solution is to accelerate the meantime to resolution and find where problems lie to get them rectified as quickly as possible.

It helps optimize productivity and downtime. Before our clients used this sort of stuff, the downtime was sort of an unknown problem. People were doing silly things with Wireshark traces trying to work stuff out. The hard part was finding out what the problem was. There was meantime trying to understand what the issue was, then trying to solve it. We have cut down that scenario of trying to work out where the problem is, which is obviously a benefit to our clients who use Accedian sensors.

We are an IT consulting system integration and managed service provider. We provide service to different companies and focus on IT infrastructure, licenses, and security network. We also provide IPs, data center, virtualization, cloud solutions, etc.

Our customer had a performance issue. So, I introduced them to enterprise side of the Accedian solution.

They have both on-premise and SaaS, but the primary component is deployed on-premise. We gave them two components to the solution: One is called active monitoring and another is called passive monitoring. 

The active monitoring is part of their box appliance into the circuits and used to collect real-time information. That information will be sent over to their SaaS platform for analytics. We collect the information locally, installing a sensor control unit, which is a virtual server. Then, information gets collected from all the locations via active monitoring. The information will then be sent over to a the SaaS platform, which Accedian has for analysis. 

• The ant Module is an appliance monitor, like a black box appliance. That's controlled by VMware. It is active monitoring and appliance-based. It plugs in data to the local VMware virtual server for collection, which is sent over to the cloud staff for analytics. 
• The latest version of PVX that we are using is 6.6. That's for the application performance monitoring, which is passive. I know they are also trying to get the PVX into the cloud for analytics as well. I think that will happen later this year, but right now, it's being analyzed locally on a separate VMware server. 

They have a bunch of VMware servers using the full control for orchestrator and to do analysis.

For this particular deployment, we monitor the Internet, but we don't have Amazon Web Services or cloud deployment. We don't use the cloud for our service application. We have a local hosted service data center and our own private cloud. We built our own hosted data center at a third-party. Therefore, we do monitor the data side of the performance, including the connection, network, and application, but it's not a public cloud. We deploy the same PVX and ant Modules in our private cloud environment. These collect the application data for me so we can see how the application performs. It has achieved what I wanted to do, so it works great.

One major piece of what we are doing is called active monitoring (ant Module). Active monitoring is to monitor all circuits we have. For example, from Miami to New Jersey, we have a circuit. We're going to measure the performance package: loss, delay, jitter, and slew codes. We measure those KPI metrics, then we know how our network is doing. That's one side of the monitoring.

Another component is called a passive monitoring. At every location and between locations, we have a lot of application data. So, we are collecting all that data to the Accedian on-premise enterprise appliance (PVX). The PVX will collect all the application data locally and analyze it based on the different network segmentation of the traffic and its different applications, such as the Internet abroad in traffic, email traffic, database traffic, web traffic, or other production application traffic. It will separate them and analyze that traffic for each flow. If there are some issue with some application, it will be able to detect them.

For example, if people say, "Oh, my email is very slow." It can tell me, "Because the DNS resolution took 10 seconds. It should it be maybe 0.5 seconds. That's why there is a problem." Then, we can look into why it's taking so long to resolve a DNS thing. So, that helps us troubleshoot. Otherwise, when people complain about the performance, we don't know where to look into that issue. Therefore, this is a very good tool. We can look at the real-time data. We can understand why maybe a server is overutilized, then we can troubleshoot it from there. So, it's very powerful. We like it.

We have had poor network performance, and when we complained to our service provider, whomever they were, we didn't have data to back up our claim. We needed to show them something, such as, "Over the last three minutes, we haven't been collecting data between the two endpoints of your circuit." That's not good, and we needed to show them something.

Accedian can help me find the root cause of this issue. In general, we need to measure the service provider's performance. The service provider, when they sell you your circuit, will give you an SLA. Therefore, we can use the information that we collected to compare it to their SLA, whether it's package loss or latency. It can help me pinpoint the root cause of the network problem quickly. That's definitely one good thing.

With the passive monitoring for the application, they can pinpoint an application issue quickly to avoid finger-pointing. Application people might say it's a network issue. Network people might say, "No, it's not my issue. It's an application issue." We can get into those ping pong games all day long. It will not get us anywhere. With this tool, it can help us intelligently troubleshoot the root cause of the issue, which is great.

I use Accedian Skylight to diagnose performance or slowness issues on networks.

The solution has undergone a change - it used to be on-premise, but now it has transitioned to a SaaS solution. Despite this change, it still requires the purchase of some internal probes. The SaaS environment is currently not widely adopted and thus, not much is known about it.

The feature I used to like the most was its ability to decode layer seven protocols, although this is becoming less useful now that encryption is so widespread.

Our primary use case is troubleshooting. If a user starts to complain about something that's not working, we have a look into it. 

It depends on whether it's already into the mirrors or not, because that's a problem at the moment. We have most of our servers on ESX machines at the moment, and each of these machines have dual 10-giga links. Putting them all together with a mirror onto only one 10-giga link makes it drop packets. I usually don't do that. I usually only put one or two of those servers into the mirror to avoid too much packet loss.

In the future we should go for a more distributed version, where we have Skylight on each ESX server so we don't lose any information at all. But at the moment, we have just one big appliance, called Site Extra Large, and we are running 5.1.3R1.

In terms of network troubleshooting, not so long ago a new company started (IT hosted on our location), and one of them was complaining about QoS trouble with VoIP telephony. They decided to go with a cloud telephony solution where I did have no say in it. I had to tell them, "I have no idea how this thing works. I have no visibility into any of the management of that solution." They were complaining about bad voice quality. Normally, I shouldn't have been able to know anything about it. But I thought I would just put Skylight on it for a moment to have a look at what was happening. I changed the mirror for a while so that Skylight could see the traffic, and it didn't take long to figure out that there were a few agencies that the company was using that had, at certain moments in time, some notable packet loss and some bad MOS scores on the VoIP part.

With that information, I was able to go to the company that gave us the SIP trunk and tell them about it. They changed some parameters, because there was something wrong on their side, and it was fixed. 

Without that hard information it would have been pretty difficult to find things like that. Skylight is not used on a daily basis, but when it's used it usually helps to fix a problem pretty fast. I don't really have to look into multiple devices because it captures most of them. With that information I can usually say, "Okay, why is your server slow? Maybe you should not have all the thin clients asking every ten seconds about some server." If you have 1,000 thin clients, suddenly you have 100 requests every second.

That's what you can easily see with Skylight. 

I have no idea how much time it has saved when it comes to response time, but I am comfortable saying "a lot."

In terms of helping improve interaction between our network, server, desktop and database teams, we're all in the same boat. It has helped me a lot when they have had serious issues. It helps me to say, "You came to me with an issue, I found this information. Which points to a specific problem at some team".  It helps them, of course, in figuring out why something is slow/not working... It helps us saving time. We can look for a solution, not the problem.

In the past, we had some serious loads on a file cluster, which was mainly due to a few procedures doing some bad things, like passing through an entire directory and asking for every file in the directory tree one by one. While doing that, it continually opened and closed all the total transactions. So it was incredibly slow and incredibly heavy, because of one bad procedure. Skylight helped us a lot at that point in time, especially the server team, to figure out why the file cluster was slow. In the end it was a simple procedure creating havoc on the FILE Cluster. Hard to find when you can't see CIFS traffic.

I had prepared for COC and the client. I work as a vendor for a client using Flow Mount for network performance monitoring. I focus on resolving client-side issues related to Packy Performance and quality use. This involves addressing network device issues, specifically Cisco network devices.

One valuable feature we have is real-time monitoring for connection issues. I can find and file any connection issues, providing server connection assurance and real-time monitoring. AI solutions are also good for networking management and family management. There is also local support for our vendor or seller.

We use Skylight to monitor the performance of our network on different sites. We also use it to troubleshoot if there is an issue on the network/standard and the end user experience is good.

Recently, we used Skylight to monitor the amount of external user VPNs. With COVID-19, we have many people who work at home. We used Skylight to count the number of users and monitor the Internet line to see if there was a point of saturation.

It is mostly deployed on-premise with some services in the cloud.

It has sort of improved capacity planning by helping to size the network bandwidth or performance for the infrastructure.

Skylight is on our data center. Only our network teams have access to it. However, the network team does speak about the diagram of the dashboard to the application team, who doesn't have access to Skylight. Thus, the solution has helped to improve the interaction between our network and application teams as we work together.

We use it to troubleshoot application issues. For instance, when an application is slow we can identify where the bottleneck is in the chain.

In terms of network troubleshooting, we had an issue with an application, that it was slow for certain users. We didn't understand why. We started to investigate and, thanks to Skylight, we were able to see that the application was very chatty. It was fetching lines one-by-one. That meant it had to do many transactions to get the full bundle from the host. Skylight helped a lot in this case because we could see that there were 1,000,000 small transactions instead of just a few.

We didn't have a tool before to investigate end-to-end performance issues. Thanks to Skylight we have a tool for this kind of investigation. When business users are complaining about application problems, we are able at least to identify where the bottleneck is. Of course, if the bottleneck is on the server itself, we will use tools other than Skylight because it is more for network investigation than on the server side. With Skylight we can identify that most of the time is spent on the database or on the proxy or on the application server. It has sped up investigations a lot. Before, we had to spend more time to identify in which block the bottleneck was found. Now it's much faster. It has reduced our mean time to respond by at least 50 percent.

In addition, sometimes users complain that they are not getting the data they want. Thanks to Skylight and the PCAP functionality, we are able to capture the traffic and to confirm whether the data was transmitted or not. It's quite important in terms of proof.

Skylight has helped to improve the interaction between our network data center and application teams. The server people are now aware of the possibility of investigating the traffic. Thus, when a technical team is facing an issue, they always think, "Okay, maybe I can get more information thanks to Skylight."