PubSub+ Platform Reviews

We do a lot of pricing data through here, market data from the street that we feed onto the event bus and distribute out using permissioning and controls. Some of that external data has to have controls on top of it so we can give access to it. We also have internal pricing information that we generate ourselves and distribute out. So we have both server-based clients connecting and end-user clients from PCs. We have about 25,000 to 30,000 connections to the different appliances globally, from either servers or end-users, including desktop applications or a back-end trading service. These two use cases are direct messaging; fire-and-forget types of scenarios.

We also have what we call post-trade information, which is the guaranteed messaging piece for us. Once we book a trade, for example, that data, obviously, cannot be lost. It's a regulatory obligation to record that information, send it back out to the street, report it to regulators, etc. Those messages are all guaranteed.

We also have app-to-app messaging where, within an application team, they want to be able to send messages from the application servers, sharing data within their application stack. 

Those are the four big use cases that make up a large majority of the data.

But we have about 400 application teams using it. There are varied use cases and, from an API perspective, we're using Java, .NET, C, and we're using WebSockets and their JavaScript. We have quite a variety of connections to the different appliances, using it for slightly different use cases.

It's all on-prem across physical appliances. We have some that live in our DMZ, so external clients can connect to those. But the majority, 95 percent of the stuff, is on-prem and for internal clients. It's deployed across Sydney, Singapore, Hong Kong, Tokyo, London, New York, and Toronto, all connected together. We are currently working on the cloud setup to connect on-prem with cloud based virtual message routers all connected together.

With the old platforms we were coming from, if we wanted to make changes, some of those changes were intrusive to make. For example, to add a new application into the environment, we would have to make a change that might cause some disruptions to the environment. We only have very limited downtime for our environment on a Saturday after midnight and before midnight again on Sunday. That is our only change-window for the week, if we have to do something intrusive. That limited us to when we could truly make changes. On a lot of other vendors' platforms, to add things, you've got to restart components and cause disruption. 

The benefit of Solace is that we can add an application in the middle of the day, with no disruption to anyone. It's purely based on our access-control list and permissioning. We can add an application in with zero disruption. We can onboard applications during the middle of our business day. It's still under change control, but there's zero impact by doing it. For us, that is super-powerful. Whether we're adding users or adding applications, we can do it, without causing any disruption. For a lot of other products, that's not the case. That's been a huge win for us.

In terms of application design, I've seen applications go live in less than a week, from coding the first line of code to putting something into production. It depends on how complex the application is. We have a central team where we support the wrappers on top of the vendor's API and we have some example code bases where we show a simple application built using our wrapper on top of Solace's API. A developer who joins our company knowing nothing about Solace, can walk through our documentation, have a look at our wrappers, take some of our example code, and get up and running and off to the races pretty quickly. Getting up to speed is definitely not difficult.

We might get a new user in our bank who is familiar with other messaging systems and who has preconceived ideas on how they want to do things. They might ask us, "How do I get access to this messaging system that I used to use with my old organization? That's what I'm familiar with." Sometimes we have to do sessions with those people and say, "Okay, we're familiar with the systems you're talking about. We supported them in the past. Talk us through what your use case is, what it is you are trying to achieve." Once they explain their use cases, we can say, "Okay, great. We actually have this and here's some example code and this how to do it." Within a day, that person has gone from knowing nothing about it to saying, "Okay, you're, absolutely meeting my application needs and now I'm educated on how this works." They're off and running very quickly.

We take all kinds of data onto the environment to share. Because the event bus is the place that every application always needs to start, they're no longer building an application now within the capital markets organization without putting their data onto our bus in some way. It's definitely a way of lowering the barrier to sharing data and getting things up and running quickly. Similarly, they can take data from other teams, once they find out what's available. Someone might say, "I need all the FX prices in the bank. Oh, I can just subscribe them from here. I don't even need to talk to the FX team." Teams can get up and running very quickly without having to spend a lot of time working with other groups to make that happen.

By having all of that data together in one place, Event Broker has definitely reduced the amount of time it takes to get a new application onboarded. We came from a place with six or seven different systems, where we might have bridged some of those together in some way, but it wasn't one common environment. Now, we've got application A that comes online and starts putting that data out for application B to get up to speed and to start looking at that data. That is very quick and easy for us to do. All the messaging that we do is self-describing. They can look at the payload of a message and understand it without even needing to talk to the upstream application. We can have applications starting to look at data where they didn't even have to speak to the upstream application. We've gone from 8 x 1 Gig, 10 years ago, to 8 x 10 Gigs today, and the reason for that is because we keep putting more and more data and applications on here. That continues to grow exponentially. If it wasn't easy to do, the data wouldn't be going up and we wouldn't have all these applications now on here. It's hard for me to say it has definitely increased the productivity, because I don't own the application development piece but, anecdotally, I would say it has.

Another area of benefit is that we're in the process of containerizing all of our applications at the moment, whether they'll be run on-prem or in the public cloud. The underlying piece is that these containers, wherever they run, are going to need to share data between the different applications and then back to the users. The Solace event mesh or event brokers are the underlying lifeblood among all of these containers. They need to have some way of communicating with each other and we see solace as being that connection among all of them. All the different cloud environments have their own messaging and we don't want to build applications that are specific to any one cloud; we want to be cloud-agnostic. To do that, we need to have a messaging system that is equally agnostic. Given that we already have a huge investment on-premise for all of our Solace stuff, we see that the future of containerizing our applications goes hand-in-hand with our messaging strategy around Solace, so we can be totally cloud-agnostic.

Technology, in the last 10 years, has probably become a lot more stable generally, but I can say that with the amount of data we put through these appliances, and route globally every day, if our environment was down capital markets wouldn't be operating for the bank. That's how critical it is. We can't afford to have any issues. At the same time, literally no application can run in our front office without this. If I look back 10 years ago, we might have had six or seven different distributed systems, all with their own problems. Now that we've consolidated all that, there's a huge efficiency by sharing all our data between the different groups. It means we can get up to speed very quickly, but also, what we're enabling from a business perspective, by sharing 120 billion messages a day, is hugely valuable to our front office.

The first use case is technology operations tools. We are a best of breed monitoring shop. We have all kinds of tools that monitor things, like storage, network, servers, applications, and all types of stovepipes that do domain specific monitoring. Each one of those tools was sold to us with what they called a single pane of glass for their stovepipe. However, none of the tools are actually publishing or sharing any of the events that they have detected. So, we have been doing a poor job of correlating events to try and figure out what's going on in our operations. 

Our use case was to leverage that existing investment. For about a year, we have been proving that we can build publishing adapters from these legacy monitoring tools which are each valid in their own right, like storage monitoring tools, network monitoring tools, and application monitoring tools (like Dynatrace), and more modern than other ones. We have been building publishing adapters from those things so we can transport those events to an event aggregation and event correlation service. We're still trying to run through our list of candidates for what our event correlation will be, but the popular players are Splunk, Datadog, and Moogsoft, then ServiceNow has its own event management module. 

From an IT systems management perspective, our use case is to have a common event transport fabric that spans multiclouds and is WAN optimized. What is important for me is topic wildcarding and prioritization/QoS. We want to be able to set some priorities on IT events versus real business events. 

The second use case is more of an application focus. I'm only a contributor on the app side. I'm more of an infrastructure cloud architect and don't really lead any of the application modernization programs, but I'm a participant in almost all of them. E.g., we have application A and application B side by side sitting in our on-prem data center, and they happen to use IBM MQ Hub to share our data as an integration. Application A wants to move to Azure. They are willing to make their investment to modernize the app, not a forklift, but some type of transformation event. Their very first question to us is, "I need to bring IBM MQ with me because I need to talk to app B who has no funding and is not going to do anything." Therefore, our opening position is, "Let's not do that. Let's use cloud-native technology where possible when you're replatforming your application. Use whatever capability you have for asynchronous messaging that Azure offers you. Let's get that message onto the Azure Event Hub. Don't worry about it arriving where it needs to arrive because we'll have Solace do some protocol transformation with HybridEdge, essentially building a bridge between the Azure Event Hub and MQ Hub that we have in our data center."

The idea is to build bridges between our asynchronous messaging hubs, and there's only a small handful of them, where Azure Event Hub is the most modern. We have an MQ Hub that runs on a mainframe and IBM DataPower appliances that serve as our enterprise service bus (ESB). Therefore, if we build bridges between those systems, then our app modernization strategy is facilitated by a seamless migration to Azure.

The most recent version is what we installed about three weeks ago.

The solution is deployed on Azure for now. We will be standing up some nodes in our on-prem data centers during next phase, probably in the next six months. 

The plan is to use event mesh. We're not using it as an event mesh yet, as we are only deployed with Azure. We want to position a Solace event mesh for enterprise, but we're just now stretching into Azure. We're a little slow on the cloud adoption thing. We've got 1200 applications at CIBC with about four of them hosted in clouds: one at AWS and three at Azure. So, we're tiptoeing into Azure right now. We're probably going to focus our energy on moving stuff into Azure. However, for now, because the volume is so low on stuff that's outside of our data center, the concept of a mesh has been socialized. There's not a ton of enthusiasm for it, even though I might be shouting from the rooftops saying, "It's a foundational capability in a multicloud world." It looks like we're putting that funding on the back burner for using it as an event mesh.

This solution has increased our application design productivity compared to other solutions. There is a ton of momentum in our application development space for leveraging Dynatrace with Solace's monitoring tool. We have made the investment in getting Dynatrace to publish events that it detects, mostly application performance related events. The app development teams have taken a liking to implementing the application monitoring tool early in their development cycles, maybe not in development, but in their performance testing cycles. We can practice what a code drop stack shift would look like if they're shifting from stack A to stack B or if they're doing rolling reboots on some of their app servers as they're doing upgrades. We get to exercise that and see what the monitoring patterns look like correlated with servers going up and down along with web services coming up and down. That's been helpful to the development community to see that automation occur in mid-environments.

There have been quite a few incidents where a test infrastructure has become unavailable because of some change going on and the app developers aren't on the nut for fixing the problem in the UAT environment because they know the outage was caused by the fact that they did a code drop 20 minutes earlier, which is a legitimate server outage. We are seeing some benefit, but it's more of an optimization of incident management resources. E.g., we have somewhere between five and 10 Internet-facing applications, and when something goes bump on the firewall that's behind them, we have DMZs (or different zones) where we put our web tier and app tier. Therefore, when something goes bump in our network tier, we got 10 application teams that are all fired up, and say, "What's going on?" Then, they all spin up their own tech bridges. Meanwhile, the firewall guys are working on a problem that we just don't know about. So, we have wasted a lot of time and energy trying to figure out things that aren't our problem. The bad scenario hasn't happened to us in production, but in our test environment, it's happens once where a couple of app dev teams have been able to stand down because we were correlating events correctly.

We struggle with mean time to resolution on things. We do have a lot of change control rigor, but the solution hasn't changed our organization yet. The idea is when we're getting the events for our service provider of operating systems, servers, and storage network correlated intelligently together with our application changes, application performance monitors, and application availability monitoring tools, then we'll make more intelligent decisions about root cause, where problems lie, and be able to react more intelligently. This will reduce mean time to resolution, but we're not there yet. 

The division who has been using Solace for years has a mature costing estimator model for internal projects. That model certainly will be leverageable for the technology operations guys. We haven't crossed that bridge yet because we're still in PoC mode. It's very likely that once we hit prod, we'll have ease of solution design when we have a protocol-agnostic message transport in place, and that our solutions will be easier to craft and give cost estimates.

It is easy for architects and developers to extend their design and development investment to new applications using this solution. In our architecture practices, we are always documenting compositions. We care a lot about the data exchanges between applications or the integrations. We have a lot of contractors and other integrations that we care about. Having transmission facilitators definitely makes the architect's life a lot easier when we just put a message on the queue and it's going to get transported by the facilitators to wherever it needs to go. It is definitely easier when we have Solace and an event mesh up and running. Today, when we have integrations that don't leverage those transmission facilitators, like an MQ Hub or Solace event mesh, those integrations are much harder to get approved because we have to dive into the security, access controls, encryption, and all that other stuff. 

We have a hybrid model because we have a lot of systems on-premise as well as a lot on the cloud. We have one instance of Solace in AWS Europe, and the other one is an on-premise setup in our data center, also in Europe.

Given the levels that we have designed into our topic taxonomy and the hierarchies, Solace gives us decent levels that we can get down to, in terms of granularity. It supports two to three character sets of their entire, end-to-end topic structure, so I can actually get down to level six or seven, or even more than that.

The last couple of releases have brought about event life cycle management. That changes the way a designer or an architect will design a topic and quickly discover what is available, and whether something has to be built out. That's pretty easy. With the life cycle of the event portal and the event cataloging that is available, it makes life easier for them. With all these new features in place it increases our productivity by something like 50 percent. Now, because we have a nice, curated view of the contents of the event in the event portal, it is easy to discover and to publish new topics. What used to take one day can be done in half a day, leveraging all the best-practices and the features that come with this product. Of course, you need to pay more if you use the event portal or catalog, but assuming all those tools are in place, it is beneficial for the productivity side.

There has also been an increase in productivity around solution management because of the ease of the key features that they offer. You don't need to spend time moving around multiple screens to manage something on the monitor, implement fixes, find hotspots, or even to publish something new. Because it is easier to navigate around, following the life cycle of an event, it definitely increases the productivity, whether it is from a solution management point of view or an operations point of view. From whichever angle you look at it, it makes life easier for that particular person.

We are using Event Broker to publish data across the enterprise, then share the transaction data updates in real-time across the enterprise, and also in some cases the telemetry data.

We do use event mesh, but our use is limited. The reason for that is we have our publishers and consumers on-prem while have our applications on AWS, Azure, and SaaS. It's a multicloud hybrid infrastructure, but the majority are still on-prem. We are slowly moving to AWS, Azure, and SaaS. As we expand to AWS and Azure, then event mesh will be a key feature that we would like to leverage.

We are using the latest version.

When publishing a product, it updates across the enterprise. We have 100 to 200 consumers, which are basically the applications interested in product changes in real-time. We could publish these product changes to Solace Event Broker with one update. Then, all 100 to 200 consumers could be listening to this topic or queue. Any time a change happens, it's pushed to this topic. They have access to it and can take whatever actions based on those changes. This all happens in real-time.

We used more point-to-point integration in the past. This solution reduces the latency to access changes in real-time and the effort required to onboard a new subscriber. It also reduces the maintenance of each of those interfaces because now the publisher and subscribers are decoupled. Event Broker handles all the communication and engagement. We can just push one update, then we don't have to know who is consuming it and what's happening to that publication downstream. It's all done by the broker, which is a huge benefit of using Event Broker.

With the event mesh feature dynamic message routing across the enterprise, you could have an event getting published from on-prem and consumers in the AWS public cloud. The publisher doesn't have to know where the consumers are. The publisher will publish it to the event broker, which could be on-prem, and the broker will have the intelligence to route the events to wherever these consumers are, whether it's AWS or a broker. If there's another broker in agile, then it will have the intelligence to route it dynamically so the publisher doesn't need to know where the consumers are. Event mesh's ability to have brokers installed across a diverse multicloud on-prem infrastructure gives us the flexibility to support applications across our enterprise. That has a big advantage.

If you just have one broker trying to do all this routing of events to different subscribers across different infrastructures, it will have a huge impact on performance. With Solace, events are routed based on the load of the broker. It can dynamically adjust the burst capacity and scale based on the events being pushed as well as events that aren't getting consumed. The logic about how to manage the routing and scaling happens dynamically between brokers. 

We're a capital markets organization, so we primarily use it for our trading algos order management, streaming market data, and general application messaging. Those are our key use cases. 

Our other use cases are for more guaranteed messaging-type or things where we absolutely need to have the resiliency of every message for higher performance streaming market data, meaning, millisecond latency-sensitive algorithm operations that are running as well.

We also use it for general messaging and to displace some of our legacy messaging applications such as MQ, EMS, and things of that sort. We are standardized on Solace PubSub+; it's an architectural standard at our company.

Our standpoint is, Solace has been tremendously successful in allowing for businesses and cross-business synchronization to be able to share data across application teams and across business verticals. It's been the success and the uptime stats that Solace has been able to provide that have allowed for more and more users to use the system. 

Going from something where we had outages and capacity issues constantly to a system that was able to scale with the massive market data and messaging spikes that happened during the initial stages of the COVID crisis in March, we were able to scale with 40-plus percent growth in our platform over the course of days. We never had any outages and never had any issues despite those massive spikes. The system was able to cope. There are things that we needed to do and work off to make sure we had headroom and then we were able to scale out, but we can do that seamlessly without our users even knowing.

It has increased application design productivity compared with competitive or open-source alternatives. From my pursuit of the "secret sauce", I found that Solace really is, for us, the service and the partnership that we have with the company. Having the level of expertise and the subject matter knowledge available to us is second to none. It's not just that Solace is entrenched at my company, is the standard, and it would be too hard to roll off. It's more that the level of performance and the level of service we get from Solace precludes that. I'm looking at where other vendors are and I just can't see them getting to the level that Solace is at for us, to really make an impact. Again, there's a small use case here and there where potentially it could be of use, but for the most part, it's not something that's going to get too much traction, given the success of the platform currently.

One of our use cases at our global company went live recently. We have a lot of goods that move via sea routes. While there are other modes of transport, particularly for the sea route, we wanted to track our shipments, their location, and that type of information and generate some reports. Also, there are multiple applications which need this data.

With Solace, we are bringing information in every minute (almost real-time) from our logistic partners and putting it on Solace. Then, from Solace, the applications that want to consume the information can take it. E.g., we are generating some dashboards in Power BI using this information. We are also pushing this information into our data lakes where more reporting plus slicing and dicing is available. In future, if more subscribers want this information, they will also be able to take it. 

We have both our private cloud and a version completely hosted on SaaS by Solace. 

The base use case is that we wanted the shipment tracking information in multiple applications, like Power BI and data lake, for the more reporting, etc. If we would not have got Solace, then we need to extract this information multiple times from source application. Now, we pull this information only once and then put it on Solace. Anybody can take it from Solace because the information is readily available. We are generating the data only once. 

Every organisation has exposure of their data with their devices being interconnected. We don't want to transfer the same data multiple times. Solutions like Solace can help us in publishing data only once, then anybody can pick it from there. This reduces costs of data transfer. It reduces the load on data sources, because we aren't asking them to generate the same data multiple times.

Our company is an SAP centred company. A lot of our key applications are using the SAP product suite. When we talk about transaction data and master data, that is where the real complexity comes into play. There are a couple of use cases that we have discussed with Solace for topic hierarchy. E.g., a product master might be sold by multiple channels, produced in multiple factories, and sold in multiple geographies, so creating a topic hierarchy for these could be challenging. When we started, we discussed this complexity with Solace. They helped us arrive at an initial topic hierarchy based on some similar use cases which have been implemented for other customers, sharing their insights.

Another point is their overall approach to topic building. They have very good documentation. It will be our own internal complexity that will drive the topic hierarchy. We are currently in the early stages, and so far journey has been good. Right now, we are comfortable with information and help we are getting from Solace along with the overall approach recommended for topic building. However, time will tell, especially after we generate very complex cases with Solace, how this topic hierarchy functions.

We use PubSub+ primarily for long processing messages. For instance, we generate customer forms, which take a longer time. We place these requests in queues, process them, and return them to the queues. This ensures that messages are not lost due to the time-consuming nature of such tasks. We also use PubSub+ for audit trails where we intercept requests, apply them, and store them in MongoDB, placing them in the queue before final processing.

PubSub+ has enhanced our asynchronous mode processing, ensuring that messages are not lost and requests are kept until fully processed. This means that messages do not get lost in the network or time out, thus improving reliability and efficiency.

We handle publishing/emitting events to the PubSub+ Broker via one protocol, and it seamlessly makes the event available for multiple consumers with different protocols without writing code is one of the primary use cases for us. 

As an architect, I wanted to implement an event-driven architecture, majorly focusing on a cloud solution with a PubSub pattern. 

Out of curiosity, our CoE team explored Solace PubSub+, and after several months of testing with our requirements, this was the kind of solution we needed.

Previously, we used on-prem Kafka for event distribution. The Kafka cluster was self-managed and hence required a dedicated team and infrastructure to manage. We wanted a cloud solution, and Kafka was already available as a SaaS offering, yet we wanted a better and more compatible solution with respect to protocols and security. Hence, we opted for PubSub+ with the support of AMQP, JMS, MQTT, and HTTPS. Without writing code, we were successful in creating event-driven microservices; parallelly, administering the cluster was not our responsibility anymore.