A powerful ESB is a perfect companion to Big Data systems. But the relation between ESB and the Big Data system is a one-way relation. It is illusory to think that an ESB is accurate to process information from Big Data Systems. By design, an ESB focuses on integration but not on processing a chunk of data. If you require to generate analytics from Big Data analytics, your first choice should be tools such as Apache Storm, Elasticsearch or Apache Geode that could process millions of messages in a short time.
Use ESB to feed your Big Data systems.
On the other hand, if you are required to populate a Big Data system, generate technical events and business messages from your business processes and use them to feed your analytics system, the ESB is perfect for this job.
Advanced ESBs such as OpenESB, Tibco or Oracle offer an orchestration (BPEL, BPMN) where your processes run. Your business process context (what happens during your processes) is located at the Orchestrator level and not spread over several microservices. So, you can easily extract messages indicating how the processes work (technical events).
Also, you can extract information from the business processes themselves and take advantage of the ESB integration work and feed your Big Data System with information coming from different sources or partners involved in your processes.
The business message generation must not affect the process design and development itself. The best ESBs offer a strong separation between business process definition and event and message generation. Focus your choice on that type of tool.
Architecture lambda.
A complete Big Data system contains many layers (ex: batch, speed, and serving layers). Each layer could have its own message’s schemas, and the ESB must generate accurate technical events or business messages to a layer without polluting the other layers. OpenESB, for example, can address up to 7 layers for the same event in your business process.
Conclusion.
As explained above, ESB tools were not designed to process Big Data but to integrate services and partners. Don’t rely on ESB to create analytics reports or any similar works.
On the other hand, ESB and the orchestration pattern combination are the perfect tools to feed Big Data systems and address lambda architecture requirements.
Now let’s reply to the question: Which areas need to be focused on while evaluating ESB with Big Data systems.
Don’t use ESB to process information from Big Data systems.
Use ESB to feed Big Data Systems or lack of data.
Prefer ESBs with orchestration (OpenESB, Oracle…) to ESBs with workflow (WSO2, ESB MULE), so your processes contexts will not be spread between services. You could access quickly all the elements making up your process when generating business messages for your analytics.
Check if the orchestrator allows you to generate business messages without impacting the business process.
Following these rules, Pymma developed a powerful feeding system for a governmental organisation in Europa for Big Data. More than 250 services’ behaviour and business messages are stored in Big Data systems for analytics. Feel free to contact me for more details.
Enterprise Service Bus (ESB) facilitates communication between various applications in an organization, enhancing operational efficiency by providing a centralized platform for integration.
ESB is a middleware solution that integrates applications and services. It enables seamless exchange of data and services, ensuring smooth workflows across the IT landscape. ESB also simplifies complex integrations by reusing services, leading to more agile and adaptable IT environments.
...
Hello all,
Don't use ESB to generate analytics.
A powerful ESB is a perfect companion to Big Data systems. But the relation between ESB and the Big Data system is a one-way relation. It is illusory to think that an ESB is accurate to process information from Big Data Systems. By design, an ESB focuses on integration but not on processing a chunk of data. If you require to generate analytics from Big Data analytics, your first choice should be tools such as Apache Storm, Elasticsearch or Apache Geode that could process millions of messages in a short time.
Use ESB to feed your Big Data systems.
On the other hand, if you are required to populate a Big Data system, generate technical events and business messages from your business processes and use them to feed your analytics system, the ESB is perfect for this job.
Advanced ESBs such as OpenESB, Tibco or Oracle offer an orchestration (BPEL, BPMN) where your processes run. Your business process context (what happens during your processes) is located at the Orchestrator level and not spread over several microservices. So, you can easily extract messages indicating how the processes work (technical events).
Also, you can extract information from the business processes themselves and take advantage of the ESB integration work and feed your Big Data System with information coming from different sources or partners involved in your processes.
The business message generation must not affect the process design and development itself. The best ESBs offer a strong separation between business process definition and event and message generation. Focus your choice on that type of tool.
Architecture lambda.
A complete Big Data system contains many layers (ex: batch, speed, and serving layers). Each layer could have its own message’s schemas, and the ESB must generate accurate technical events or business messages to a layer without polluting the other layers. OpenESB, for example, can address up to 7 layers for the same event in your business process.
Conclusion.
As explained above, ESB tools were not designed to process Big Data but to integrate services and partners. Don’t rely on ESB to create analytics reports or any similar works.
On the other hand, ESB and the orchestration pattern combination are the perfect tools to feed Big Data systems and address lambda architecture requirements.
Now let’s reply to the question: Which areas need to be focused on while evaluating ESB with Big Data systems.
Following these rules, Pymma developed a powerful feeding system for a governmental organisation in Europa for Big Data. More than 250 services’ behaviour and business messages are stored in Big Data systems for analytics. Feel free to contact me for more details.
I hope this is useful.