Present day error processing:
All ETL jobs are scheduled to execute though scheduling tool, during the execution When there is an user defined error, we store them to error table and proceed with the cycle or stop the cycle depending on the error severity. At this stage, assume if error is very severe then operations team will reach out to the technical team to analyze, technical team will analyze and share records with customer/data stewards , after customer review, they correct the files and technical team starts the cycle again.
IDQ human task:
This process is developed keeping in mind to ease/improve the present day error processing. This process allows data steward/customer to halt the cycle and review the data (also provides the ability to correct data if required) when data issues occur and provides the cycle control to them, they can release cycle after they are done with the work.
This minimizes the technical team dependency (operation, analyst/support developers) and moving/sharing data with business, also provides the ability to correct data if required. In the present day processing this whole cycle is very time consuming, requires lot of communication.
Only time data stewards/customer need to contact operations team is when the whole file cannot be corrected or cannot be processed, needs to be stopped.
IDQ human task can do:
1. During the job execution though scheduling tool, if there are errors, we can capture data and give the cycle control to customer.
2. Customer can review error data in the data analyst tool, they can correct/select the data and release the cycle.
