An On-Call Scenario

Data engineering and GTM (Go-To-Market) operations teams, much like their engineering-team counterparts, maintain their own infrastructure. Though data pipelines are at the heart of this infrastructure, the maintenance dynamics are the same as that of any engineering team: uptime and correctness are paramount

Because data pipelines involve multiple systems (very few of which the data engineer controls), the possible points of failure greatly multiply. The data engineer is responsible for the reliability of a process whose actually reliability involves multiple parties.

AI agents are becoming central to data engineers’ debugging workflows. When given access to Polytomic, the agent obtains full visibility into all systems and their data flows.

Let's take a basic example pipeline:

One day, the person looking at the spreadsheet complains about a column value that is empty when it shouldn't be. At the same time one of the SQL transformations starts failing.

Are these two incidents related, or are they separate? Why did they occur? Who and what is impacted? How badly? Is the null-value incident even an incident, or is this data expected?

Answering these questions often involves:

• Logging into different systems and clicking around.
• Talking to the admins of those systems.
• Running shell scripts or SQL queries to verify assumptions.

Once the full picture of dependencies and events is assembled, a bit more brainpower is dedicated to reasoning through everything before finally arriving at an explanation.

Any engineer, data or not, will recognise this as an example of being on-call. It's all investigations and conclusions, sometimes followed by actions.

It's Not Just On-Call

Clicks, conversations, and scripts are actions that take place outside of on-call scenarios too. Thing of a sales analyst who is SQL fluent but is not on the data team. This person may want to analyse data from disparate systems that he is deeply familiar with, but does not want to have to talk to people, click around, and run scripts just to assemble it before his analysis.

The AI DataOps Agent and Polytomic

AI models have improved to the point that having them do all the above is becoming realistic. Having an agent that is automatically both watching and reacting to issues is a powerful concept, a bit like having an on-call engineer on hand who instantly discovers issues, investigates them, and provides a resolution.

Similarly so for an agent that takes an analysis request, instantly runs the necessary data pipelines to assemble the data, then runs the analysis in one fell swoop.

Polytomic is a central character in the AI DataOps Agent story. That is why, over the next few weeks, we will be publishing stories of how our most sophisticated customers are beginning to put this vision to practice, saving hours that would otherwise have gone to clicks, conversations, scripts, and investigations.