I am a director at a business enhancement solutions company that provides mechanical systems and distributes high-quality bearings and related products.
I am planning for b2b e-commerce. I have operations in Singapore, Malaysia, Indonesia, China, and Mongolia. Each country uses a different ERP. Singapore HQ uses JDE (one world), that was last implemented 20 years ago.
I intend to consolidate the data to one 'location' for the integration of data into e-commerce or e-marketplace. Is a data warehouse the best way to go?
Thanks! I appreciate your help.
Hi there! Consolidating data from multiple ERP systems into one location for seamless integration into e-commerce or e-marketplace is a common challenge faced by businesses operating in multiple countries. While a data warehouse can be a suitable solution for this purpose, it's important to consider factors like data volume, complexity, and real-time requirements. To gain a deeper understanding of the benefits and considerations of implementing an Enterprise Data Warehouse (EDW) in your scenario, I recommend checking out this informative article (https://www.cleveroad.com/blog/enterprise-data-warehouse/). It provides valuable insights into EDW architecture, data consolidation, and scalability, which can help you make an informed decision.
For most clients, we suggest a data pipeline that includes an inexpensive storage step along the way such as AWS S3 or AZURE Block Store.
Once the data is landed at rest it can then be loaded into any EDW or CDW (Snowflake) as needed based on your consumer persona needs.
In addition, depending on timeliness and other requirements, you can retrieve the data directly from S3 storage for example using Athena without having to move it to any other platform.
Of course, like with anything else the devil is in the details. Each platform/approach should consider how you intend to costume and utilize the underlying data so that performance and costs are projected into your decision process.
The answer is yes.
If this platform is supporting e-commerce, distribution or manufacturing then a critical component of the data warehouse may be the ability to support tactical or operational queries on data that is updated throughout the day. This considerably narrows the choices.
I teach a course on how to select a cloud data warehouse platform for modern requirements. You can watch last December's version online at https://tdwi.org/Events/Confer... also, I am speaking on the subject at a TDWI event next week https://tdwi.org/events/virtua... the course (updated) will be offered live at a TDWI conference in November (virtual + live in Orlando).
Or book me for a private conversation at https://tinyurl.com/bookrw30
The idea of an EDW or enterprise data warehouse is the right path. That said, it's all about choosing the appropriate set of technology platforms and tools. Bringing together data from disparate platforms (you mentioned multiple ERP implementations) is not a trivial task and the choices you make for your EDW platform and the related data management tool will greatly affect your timeline, costs, and maintenance. Best of luck let me know if there is anything else we can do. Dave
A definition of a data warehouse was been for years anchored in retaining data that is no longer changing. A B2B e-commerce environment includes many dynamics of data while being volatile, with factors that influence the frequency of queries and updates. eCommerce is anchored in dynamic operational data for applications that support both internal and external visitors.
The ultimate purpose of a traditional enterprise data warehouse (guided by a "top-down" methodology) is a that and EDW imports operational records after achieving a state of being complete.
I would call your target environment a data repository instead. Even so, there may be many elements from your original source databases that are not needed and are no longer relevant to being integrated together into a single location.
I would proceed only after answering questions about:
1. What outcomes you are seeking and who will be the users who believe that all data needs to be integrated into a single location? and
2. Consequences on software that is now processing source systems databases that have different purposes and maybe to persist.
@Jay Allen
Jay, Look again at sources for your own understanding. My understanding of top down data warehouse for an enterprise has an origin that has been trusted for decades. "Subject-oriented, non-volatile, integrated..." (crediting Bill Inmon). As a practitioner, I do admit that in my years a data administrator, I have encouraged a bottom up approach which imports recently historical data from operational sources into data mart(s) designed to be fact based with descriptive dimensions ... (credit to Ralph Kimball).
With regard to B2B, partners in data exchange can link to either to operational database sources which are dynamic, or to data stores into which no-longer-volatile source data have been imported. -- Barbara
Your question is one about architecture, but I see some answers about the technology.
Choosing technologies is just as critical, but to your question, @David Rossi has it right. It sounds like a straightforward DW requirement. With the multiple source systems and potential data integrity issues, it also seems like a top-down DW requirement. Your operations seem like they may be complex. So you may be facing requirements that are a bigger question than the one DW answers.
Also, take a look at Corporate Information Factory (CIF) Architecture and its evolution to see if this more specific architecture offshoot of DW addresses your particular complexities.
https://www.sciencedirect.com/...
IMHB ... yes and no, i.e. you should really clarify first which use case you have in mind in sharing the data - especially if it comes to analytics, versus sharing data for integrated processes towards customers (i.e. yous service desk vs your eCommerce vs...)
The "yes" piece is that an analytical centralized DWH is for sure a must for the architecture to have the ability of advanced reporting, cross-analysis, etc. In doing so, I'm biased from my past, but as many mentioned, a good analytical modern data warehouse (better if columnar) scalable with the ability to connect to lower-cost storage (S3, Azure blob...) options, leaving the data there, would be terrific. Loading there all, with historical data is for sure good. Someone mentioned Snowflake, I was a Vertica champion... your call.
The "no" part is you might still need a more transactional integration layer to connect your different applications from legacy CRM, service desk, e-commerce, marketing. Depending on what you have here you have few different options which would then synch and load data in the analytical data warehouse.
The answer is a YES.
We have almost the same scenario with various domains of business using different systems from instore to eCommerce. And finally, JDE is used for accounting.
We designed a central data warehouse with a strong transformation engine for the past 10 years and everything works like a magic. Any new integrations, systems, countries and even new business lines consolidate in the central data warehouse, making it a cake walk for reporting and analytics. Also, the data warehouse has become a single version of truth providing tremendous power to business to use the data for financial and business integrations and monetizations.
Just a piece of advice, when you plan for the business data warehouse, be it on cloud or premise, go for a less technical complex architecture one. This will allow you to focus on business and speedy integrations. More tech teams and more tech products for integration may look interesting and futuristic, but take care and do a proper analysis before you take the next step. All the best.
It depends on whom do you ask the question. I personally say, Yes, Absolutely required. If you wanted to combine the data from multiple systems. For sure it is definitely required.
Regards
Ram
Nope, the Data warehouse approach is analyze and decision platform for business.
You need a central data mart platform to provide data to 3rd party connectors. Sometimes you need near-realtime data. Data Warehouse platforms commonly present T-1 data. Also, in the future, lots of 3rd party companies want to access this data and you don't want to provide access to your analytics platform.