Attend FREE Webinar on Digital Marketing for Career & Business Growth Register Now

Data Analytics Blog

Data Analytics Case Studies, WhyTos, HowTos, Interviews, News, Events, Jobs and more...

AIRflow at Scale: Webinar Recording

    -  

5 (100%) 1 vote

This webinar on ‘AIRflow at Scale’ covered the walkthrough of how Qubole extended Apache AIRflow to manage the operational inefficiencies that arise when you manage data pipelines in a multi-tenant environment. And also the process of making the data pipelines robust by adding data quality checks using CheckOperators.

Key Takeaways:

Listed below are the other major key takeaways: 

  • Overview of major type of data pipelines
  • How Qubole manages deployments and upgrades of data pipelines in a multi-tenant environment
  • The Data quality issues which Qubole faces in process of data ingestion or transformation.
  • The approach that Qubole has adopted using Apache Airflow Check operators.
  • Enhancements we had to make to Check operators.
  • Integration of Apache Airflow Check operators with our ETLs.
  • Challenges Qubole faced in developing the alerting framework.
  • The best practices in using Apache Airflow for data quality checks.

Did you find the webinar interesting? Register here for more.

  • Data-Analytics

  • Your Comment

    Your email address will not be published.