Project Manager at a tech company with 51-200 employees
Vendor
Jun 23, 2014
Mostly, it depends.
I agree with Martin Fowler's position for the line of this question. I can't provide the link to his words exactly but the main idea was to hold some 'base-line' of your data as simple as possible. E.g., the set of CVS files, or some other plain representation as a universal base-line of the structure. Then it's possible to process this raw data in any way you need. Load to some system for further indexing, aggregation, analysis, etc. Also, it would be ok to switch from one tool to another. Keep in mind, even if today they only asked for simple aggregation of several counters, that doesn't mean they won't ask tomorrow for much more sophisticated calculations + integration with another sources + anything else it needs for the business. We can't just say "that's impossible because we modeled some specific structure by yesterday specifications or sticking to some specific system". We should be prepared for such cases in our agile/crazy world. Data of the business lives the same term. as business. Nowadays, the Data means both the actual data and the methods/ways we can process/use it.
There is no definitive answer to when a business should invest in building a data warehouse, as it depends on various factors such as the size, complexity, and goals of the business. However, some common signs that indicate a need for a data warehouse are:
The business has multiple sources of data that are not integrated or consistent
The business needs to perform complex or ad-hoc queries and analyses on large volumes of historical data
The business wants to improve its decision-making and performance by using business intelligence (BI) and data analytics tools
The business wants to reduce the load and latency on its operational systems by separating the analytical and transactional workloads
One article that I found helpful and relevant to your question is this one: https://www.cleveroad.com/blog.... It gives a comprehensive overview of what is a business intelligence data warehouse, how it works, what are the benefits and challenges, and what are the best practices for building one. I think it can help you understand the concept and value of a data warehouse better and decide whether it’s suitable for your business. I hope this helps!
Consultant at a tech services company with 51-200 employees
Consultant
Jun 25, 2014
Hi.
DWH systems usually are designed in order to support decisions and they are
driven from business needs.
When an organization solves the operational needs by setting up core
systems like ERPs then they ask information to support better decisions
through MIS reports, dashboards and analytics. This is the time than a DWH
system rises as a must.
A DWH system is designed in three high level layers : data source layer,
data integration layer and presentation layer.
In order to propose a DWH solution, we need to know the requirements for
each layer:
1. Data Source Layer: Number of source system and vendors
2. Data Integration Layer: Data Integration Complexity - transformations,
aggregations,historical data, and data volumes
3. Presentation Layer: Reports, Dashboards, Mobility, Exploration,
Analytics
There are a lot of vendors offering cutting edge technology for Data
Integration and Presentation Layers supporting both classic BI needs and
big data and business analytics. When designing the solution is nice to
keep in mind the client budget and of course the future needs in order to
be able to propose a solution that will not be obsolete in the next few
years.
When? The moment you acknowledge you cannot run blind. The moment you
acknowledge the competitive advantage of making data driven decisions.
What? Depends on many factors including size of your enterprise and
available budget and available skills and propensity to build skills in
Analytics to develop, manage and use the analytic solutions. Highly
recommend you have a qualified consultant perform a readiness assessment
and give you a plan that meets your short term and long term business
objectives.
Asst. Senior IT Manager at a retailer with 5,001-10,000 employees
Real User
Jun 24, 2014
A business will typically feel the need to invest in a business warehouse solution when there are many standalone systems for various modules viz. Financial accounting, Purchase Ordering, Materials Management, Purchase, Sales, HR, etc. and there is a need to generate information using more than one of the above modules in an integrated way. Secondly. there is also a need to derive trends and insights from data across functions which when collated provides additional insight into the functioning of the business. Thirdly, collation of information from a variety of data sources or social media should be possible and it should be possible to process and analyse large volumes of present and historical data quickly so as to provide competitive advantage.
I would say the answer depends on several things as there is no “typical” point where a business decides to invest in data warehousing, analytics, etc.
There are also several items to consider in addition to the “platform” (and all of its components, such as software, scalability, backup, cloud, access, data management, data governance, in addition to the culture of the organization...as in do they understand and believe in the power of business data integration and how data can be used to gain better business insight...and oh don’t forget the in-house or 3rd party skill sets and what will they need to learn to correctly leverage data for analytics...). A company also has to believe that their data is a valuable/proprietary corporate asset and is willing to put the same rigor and disciplines in place to manage it as they would capital, people, and facilities.
Perhaps a better way to look at this is ‘what business potential do I think I have in my data and what will it take to leverage it’? ‘What is my current pain with data’? ‘How do I use all of these new technologies to my business’ advantage’?
As for ‘solutions’, there is no one right answer. I would recommend starting slowly. Take a subject area, like product or customer, and do some investigating up front. Profile the data from the various operational systems (sources to the eventual data warehouse or analytic environment) to gain a better understanding of what you currently have. From there, I would consult with an organization that knows how to use data strategically, so a strategic analytic roadmap can be built. This roadmap maps to current business initiatives and goals and shows how the data and applications support these initiatives. The systems piece should be the last, not the first, consideration. And I would recommend looking at a solution that can scale as your needs grow. If you start with an appliance, make sure it has a clear path to a full scale EDW-type platform to support growth without having to throw away and re-do anything.
I would definitely go with a vendor who understands data, analytics and knows how all of your data can and should be integrated (big, structured, unstructured, and multi-structured).
Use an iterative approach as you develop...a data warehouse/analytic environment is not a one shot and done...it is a continual asset as new data types come into the mix. Using an iterative approach allows you to get some wins under your belt and use those successes a momentum for the next subject area, project, data source, etc.
I am happy to discuss more, but each business situation is unique and needs to be evaluated. The analytic approach/framework that it used is the same, however each company’s situation will dictate the right approach.
One final thought...I believe all companies, regardless of size, can use an analytic approach, including a ‘Discovery’ function... As long as data is part of what they do, there is value in collecting, analyzing and acting on insights derived from this data.
Solution Architect at a tech company with 51-200 employees
Real User
Jun 23, 2014
As the Organization matures there is a need to make decisions to the direction of the growth of the organization and a need to have the right data at right time. But for this to happen the transaction data from the various functions needs to be structured and in proper format, The Datawarehouse needs data from a transactional ERP tools.
The need for a Datawarehouse is felt as the organization matures, it need not be based on number of employees, but the business needs drive the need for BI and Analytics.
Business Intelligence Specialist at a media company with 1,001-5,000 employees
Real User
Jun 23, 2014
At the point where it's reporting/analytical requirements are not fulfilled i.e.
- reporting on the transactional systems must be carefully done so the business cannot get its reports as soon/fast as it demands.
REQUIREMENT FOR A SEPARATE DWH PLATFORM
- combined reports from various source dbs. Doing it on the reporting layer may be a quick and dirty solution but the real cure comes with dwh.
REQUIREMENT FOR A DWH MODEL (done on the DWH)
- the history hold on the transactional DB by default is not enough. AGAIN REQ FOR A DWH (MODEL)
Oracle Web Administrator at a tech services company with 1,001-5,000 employees
Consultant
Top 20
Jun 23, 2014
Data warehouses are widely used within the largest and most complex businesses in the world.
Use within moderately large organizations, even those with more than 1,000 employees remains surprisingly low at the moment.
We are confident that use of this technology will grow dramatically in the next few years.
In challenging times good decision making becomes critical.
The best decisions are made
when all the relevant data available is taken into consideration.
The best possible source for that data
is a well designed data warehouse.
The concept of data
warehousing is deceptively simple.
Data is extracted periodically from the
applications that support business processes and copied onto special dedicated computers. There it
can be validated, reformatted, reorganized, summarized, restructured, and supplemented with data from other sources. The resulting data warehouse
becomes the main source of information
for report generation, analysis, and presentation through ad hoc reports,
portals, and dashboards.
Data Warehouse is an attempt to correct the poor database designs in which the transactional data resides. So a DW is an "after the horse has escaped the barn" solution attempt. The more horses that have escaped, and it may not even be certain how many horses, the more expensive/painful would be the solution. Also if the approach to solving the problem is similar to the one that initially led to the problem, several Data Marts for example, then eventually there will be yet another problem waiting to happen. So investment in DW means that the business has already gone down the wrong path in deploying IT and wrong decisions have already been made. When that realization occurs is when investment in DW begins; some soltions are self correcting while others just a perpetualtion of the same old, same old in new clothes or rather no one is willing to say that the kings has no clothes.
The most important aspect, in order to think in investments about Business DW, is to have a mature organization in planning, business processes modeling and information technology. This means that the organization must have, multidisciplinary and competent personal, in each critic area, so these people impulse new processes, technologies and changes that the organization requires.
Data Warehouses efficiently store and manage large volumes of data, allowing organizations to facilitate reporting and data analysis. They support decision-making processes by integrating data from various sources into a centralized repository.A Data Warehouse is designed to enable business intelligence by providing a consolidated view of data from disparate systems. It supports complex queries and quick retrieval of information, thus serving as a backbone for analytics. As data quantities...
Mostly, it depends.
I agree with Martin Fowler's position for the line of this question. I can't provide the link to his words exactly but the main idea was to hold some 'base-line' of your data as simple as possible. E.g., the set of CVS files, or some other plain representation as a universal base-line of the structure. Then it's possible to process this raw data in any way you need. Load to some system for further indexing, aggregation, analysis, etc. Also, it would be ok to switch from one tool to another. Keep in mind, even if today they only asked for simple aggregation of several counters, that doesn't mean they won't ask tomorrow for much more sophisticated calculations + integration with another sources + anything else it needs for the business. We can't just say "that's impossible because we modeled some specific structure by yesterday specifications or sticking to some specific system". We should be prepared for such cases in our agile/crazy world. Data of the business lives the same term. as business. Nowadays, the Data means both the actual data and the methods/ways we can process/use it.
There is no definitive answer to when a business should invest in building a data warehouse, as it depends on various factors such as the size, complexity, and goals of the business. However, some common signs that indicate a need for a data warehouse are:
One article that I found helpful and relevant to your question is this one: https://www.cleveroad.com/blog.... It gives a comprehensive overview of what is a business intelligence data warehouse, how it works, what are the benefits and challenges, and what are the best practices for building one. I think it can help you understand the concept and value of a data warehouse better and decide whether it’s suitable for your business. I hope this helps!
Hi.
DWH systems usually are designed in order to support decisions and they are
driven from business needs.
When an organization solves the operational needs by setting up core
systems like ERPs then they ask information to support better decisions
through MIS reports, dashboards and analytics. This is the time than a DWH
system rises as a must.
A DWH system is designed in three high level layers : data source layer,
data integration layer and presentation layer.
In order to propose a DWH solution, we need to know the requirements for
each layer:
1. Data Source Layer: Number of source system and vendors
2. Data Integration Layer: Data Integration Complexity - transformations,
aggregations,historical data, and data volumes
3. Presentation Layer: Reports, Dashboards, Mobility, Exploration,
Analytics
There are a lot of vendors offering cutting edge technology for Data
Integration and Presentation Layers supporting both classic BI needs and
big data and business analytics. When designing the solution is nice to
keep in mind the client budget and of course the future needs in order to
be able to propose a solution that will not be obsolete in the next few
years.
When? The moment you acknowledge you cannot run blind. The moment you
acknowledge the competitive advantage of making data driven decisions.
What? Depends on many factors including size of your enterprise and
available budget and available skills and propensity to build skills in
Analytics to develop, manage and use the analytic solutions. Highly
recommend you have a qualified consultant perform a readiness assessment
and give you a plan that meets your short term and long term business
objectives.
A business will typically feel the need to invest in a business warehouse solution when there are many standalone systems for various modules viz. Financial accounting, Purchase Ordering, Materials Management, Purchase, Sales, HR, etc. and there is a need to generate information using more than one of the above modules in an integrated way. Secondly. there is also a need to derive trends and insights from data across functions which when collated provides additional insight into the functioning of the business. Thirdly, collation of information from a variety of data sources or social media should be possible and it should be possible to process and analyse large volumes of present and historical data quickly so as to provide competitive advantage.
I would say the answer depends on several things as there is no “typical” point where a business decides to invest in data warehousing, analytics, etc.
There are also several items to consider in addition to the “platform” (and all of its components, such as software, scalability, backup, cloud, access, data management, data governance, in addition to the culture of the organization...as in do they understand and believe in the power of business data integration and how data can be used to gain better business insight...and oh don’t forget the in-house or 3rd party skill sets and what will they need to learn to correctly leverage data for analytics...). A company also has to believe that their data is a valuable/proprietary corporate asset and is willing to put the same rigor and disciplines in place to manage it as they would capital, people, and facilities.
Perhaps a better way to look at this is ‘what business potential do I think I have in my data and what will it take to leverage it’? ‘What is my current pain with data’? ‘How do I use all of these new technologies to my business’ advantage’?
As for ‘solutions’, there is no one right answer. I would recommend starting slowly. Take a subject area, like product or customer, and do some investigating up front. Profile the data from the various operational systems (sources to the eventual data warehouse or analytic environment) to gain a better understanding of what you currently have. From there, I would consult with an organization that knows how to use data strategically, so a strategic analytic roadmap can be built. This roadmap maps to current business initiatives and goals and shows how the data and applications support these initiatives. The systems piece should be the last, not the first, consideration. And I would recommend looking at a solution that can scale as your needs grow. If you start with an appliance, make sure it has a clear path to a full scale EDW-type platform to support growth without having to throw away and re-do anything.
I would definitely go with a vendor who understands data, analytics and knows how all of your data can and should be integrated (big, structured, unstructured, and multi-structured).
Use an iterative approach as you develop...a data warehouse/analytic environment is not a one shot and done...it is a continual asset as new data types come into the mix. Using an iterative approach allows you to get some wins under your belt and use those successes a momentum for the next subject area, project, data source, etc.
I am happy to discuss more, but each business situation is unique and needs to be evaluated. The analytic approach/framework that it used is the same, however each company’s situation will dictate the right approach.
One final thought...I believe all companies, regardless of size, can use an analytic approach, including a ‘Discovery’ function... As long as data is part of what they do, there is value in collecting, analyzing and acting on insights derived from this data.
Hope this helps.
Dave
As the Organization matures there is a need to make decisions to the direction of the growth of the organization and a need to have the right data at right time. But for this to happen the transaction data from the various functions needs to be structured and in proper format, The Datawarehouse needs data from a transactional ERP tools.
The need for a Datawarehouse is felt as the organization matures, it need not be based on number of employees, but the business needs drive the need for BI and Analytics.
At the point where it's reporting/analytical requirements are not fulfilled i.e.
- reporting on the transactional systems must be carefully done so the business cannot get its reports as soon/fast as it demands.
REQUIREMENT FOR A SEPARATE DWH PLATFORM
- combined reports from various source dbs. Doing it on the reporting layer may be a quick and dirty solution but the real cure comes with dwh.
REQUIREMENT FOR A DWH MODEL (done on the DWH)
- the history hold on the transactional DB by default is not enough. AGAIN REQ FOR A DWH (MODEL)
Data warehouses are widely used within the largest and most complex businesses in the world.
Use within moderately large organizations, even those with more than 1,000 employees remains surprisingly low at the moment.
We are confident that use of this technology will grow dramatically in the next few years.
In challenging times good decision making becomes critical.
The best decisions are made
when all the relevant data available is taken into consideration.
The best possible source for that data
is a well designed data warehouse.
The concept of data
warehousing is deceptively simple.
Data is extracted periodically from the
applications that support business processes and copied onto special dedicated computers. There it
can be validated, reformatted, reorganized, summarized, restructured, and supplemented with data from other sources. The resulting data warehouse
becomes the main source of information
for report generation, analysis, and presentation through ad hoc reports,
portals, and dashboards.
Data Warehouse is an attempt to correct the poor database designs in which the transactional data resides. So a DW is an "after the horse has escaped the barn" solution attempt. The more horses that have escaped, and it may not even be certain how many horses, the more expensive/painful would be the solution. Also if the approach to solving the problem is similar to the one that initially led to the problem, several Data Marts for example, then eventually there will be yet another problem waiting to happen. So investment in DW means that the business has already gone down the wrong path in deploying IT and wrong decisions have already been made. When that realization occurs is when investment in DW begins; some soltions are self correcting while others just a perpetualtion of the same old, same old in new clothes or rather no one is willing to say that the kings has no clothes.
The most important aspect, in order to think in investments about Business DW, is to have a mature organization in planning, business processes modeling and information technology. This means that the organization must have, multidisciplinary and competent personal, in each critic area, so these people impulse new processes, technologies and changes that the organization requires.