The origins, purpose, and practice of data observability

Data Observability (DO) is an emerging category that proposes to help organizations identify, resolve, and prevent data quality issues by continuously monitoring the state of their data over time. This talk is a deep dive into DO, starting from its origins (why it matters), defining the scope and components of DO (what it is), and finally closing with actionable advice for putting observability into practice (how to do it).

We’ll rigorously define data observability to understand why it is different from software observability and existing data quality monitoring. We will derive the four pillars of DO (metrics, metadata, lineage, and logs) then describe how these pillars can be tied to common use cases encountered by teams using popular data architectures, especially on cloud data stacks.

Finally, we’ll close with pointers for how to put observability into practice, drawing from our experience helping teams across sizes, from fast-growing startups to large enterprises, successfully implement DO. Successfully implementing observability throughout an organization involves not only using the right technology, whether that be a commercial solution, an in-house initiative, or an open source project, but implementing the correct processes with the right people responsible for specific jobs. Talk participants can expect to leave with new concepts to understand how DO can help their organizations and ideas for how to implement DO.

Kevin-Hu-Metaplane.jpg

Kevin Hu

Co-founder

metaplane-logo-transparent.svg

What is the cost to attend and watch the virtual sessions?

DataOps Unleashed is always free and open for all to attend.

Unleashed 2022 was held live, virtually on February 2nd, 2022.

What is DataOps Unleashed?

DataOps Unleashed is the official DataOps peer-to-peer community.

It's a time for everyone, from DataOps, CloudOps, AIOps, MLOps, to other technology professionals, to gather virtually to share the latest trends and best practices for running, managing, and monitoring data pipelines and data-intensive analytics workloads.

Sessions include talks by DataOps professionals at leading organizations, detailing how they’re establishing data predictability, increasing reliability, and reducing costs.

New to DataOps?

DataOps is a holistic approach to the creation, deployment, monitoring, management, and optimization of data-driven applications. It describes the culture and rules of engagement that allow data teams to deliver and maintain high-quality, on-time data products, often powered by AI and machine learning, in an agile and cost-effective way.

DataOps defines how data teams work and also affects data consumers and those whose work causes new data to be created and used within the organization. Their work enables the entire organization to access data efficiently for data-driven decision-making and for the creation and delivery of data-driven applications.

Organizations with well-developed DataOps strategies, governance, and processes can expedite the delivery of data-driven workflows and results faster and better than others.

Who comes to DataOps Unleashed?

DataOps professionals and experts including data administrators, data architects, data engineers, data analysts, AI/ML professionals, and data technology leadership.

Join us for sessions on:

  • Data pipelines
  • Data orchestration
  • Data team composition
  • Data architecture
  • Data quality
  • Data governance
  • Data observability
  • Data operations
  • Data optimization
  • Data cost governance
  • Data migrations
  • Data modernization
  • MLOps/AIOps

Want to speak at the next session?

Send us a note to astronaut@solutionmonday.com or submit a talk proposal here: dataopsunleashed.com/cfp

Didn't make it to DataOps Unleashed 2022? Enter your email address below for free access to the next DataOps Unleashed!

Interested in speaking at the next DataOps Unleashed or participating as a community sponsor?

Please contact astronaut@solutionmonday.com.