Presenting Yuki Kakegawa

Yuki is a Staff Data Engineer, author of the Polars Cookbook, and the founder of Orem Data. He writes about data tools and independent consulting on Substack and LinkedIn.

I asked , Staff Data Engineer and author of the Polars Cookbook, what the biggest adjustment was when he became a team of one.

He skipped tooling. He skipped workload. He said this:

It’s a shift in mindset. You’re not a data engineer anymore. You’re a data person helping the business with data and analytics

That sentence landed harder than I expected, because most people assume the challenge is the technical breadth: wearing all the hats, being the analyst, the engineer, and the data scientist rolled into one.

That part is visible. What nobody talks about is the identity part.

You Were Hired as a Data Engineer. That Is the Problem.

At a large company, your identity is your scope. You do pipeline work, pass it to the analyst, the analyst makes the report, the report goes to the business. You are one node in a chain, and that chain insulates you from the messiness of what the business needs.

When you’re a team of one, that chain is you.

Yuki described his current day-to-day:

• talking to the business to understand what they care about

• translating that into projects

• building the pipelines

• building the models,

• building the reporting layer

• delivering insights directly to stakeholders

He skips the handoff entirely.

That scope describes a different job, built around a different identity.

Subscribe now

The Selfishness That Kills You

He used a word that surprised me: selfish.

When I was first getting started, I wanted to work on pipelines and I didn’t want to work on reporting. That selfishness kills you in this role. You have to be flexible and you have to care about the business first.

Most data engineers have strong opinions about which work is worth doing. Pipelines are interesting. Reporting is boring. Data modeling is craft. Dashboards are noise.

Those opinions are how you survive in a large org where you can negotiate your scope.

A team of one has no scope to negotiate. The business has no interest in which part of the stack you find meaningful.

The work that needs doing is the work you do, and if you have not made peace with that before taking the role, the first six months will feel like a slow grind against a situation you agreed to.

What Is Broken at Every Startup You Walk Into

I asked Yuki what he finds broken almost every time he walks into a young company.

Tooling is rarely the issue. It’s the processes around the processes that produce the data used for reporting and analytics downstream. And the alignment on what’s important, the definition of metrics, what we want to build.

Two things. The upstream processes that generate data, and whether the business has agreed on what the numbers mean. The stack is almost never the problem.

The Metric Definition Problem

This is the one that is hardest to fix and most commonly ignored. Yuki described the ideal state: every metric the business cares about is written down, defined, and agreed on before the data team touches a model.

When that is true, the data team’s job is implementation. The hard part is already done for you.

The real state at most companies is three departments with three definitions of the same number.

If you use one team’s definition, you put another team in a bad spot. If you try to reconcile them, you are suddenly in a political conversation nobody hired you to have.

I’ve been in that position. That’s the part I hated the most.

Defining core metrics early, before the company grows, before each department builds its own reporting layer, before everyone has opinions baked into their own numbers, is more valuable than any pipeline you will build.

If you’re walking into a company where this work is undone, do it first.

How to Handle the Stakeholder Who Thinks Your Work Takes Two Weeks

Yuki described a pattern that every team-of-one data person will recognise. A stakeholder requests a report, thinks it can be done in two weeks, and the data behind it is messy enough that you know it will take four.

I can try, but I can’t promise, because I think these things will be bottlenecks.

Two things are happening in that exchange. Setting expectations is the obvious one. Making the complexity visible is the one that changes the relationship long-term.

When you’re the only data person, the rest of the company defaults to assuming data work is fast. Pull the numbers, build the report, done.

They genuinely have no frame of reference for what it takes to model messy source data into something accurate enough to make decisions with. Your job is to narrate the work before you do it, and skip the explanation after you miss the deadline.

The Transparency Stack

Yuki added something that makes the whole prioritisation problem easier: making your priority stack visible to the entire business.

Being transparent about what you’re working on and what the priorities are is important. Because if one stakeholder thinks his project is the top priority, but I’m working on a priority item requested by the CEO, then that stakeholder will understand that his project is not at the top.

This works because it shifts who does the priority negotiation. When everyone can see the queue, the conversation moves from “why isn’t my thing done“ to “is my thing in the right place in the queue“. The second conversation is faster and involves fewer emotions.

Accuracy vs. Speed Is a Trade-Off to Name, Not a Choice to Make Alone

I pushed Yuki on a question that sounds simple: accuracy or speed?

Definitely accuracy. But that’s where your skill comes in. How do you deliver projects fast while ensuring accuracy?

The real answer lives in the communication around the trade-off, not the trade-off itself. If you can deliver in one week at 80% data quality, or in two weeks at 100%, the stakeholder should make that call. They can only make it if you put the options in front of them explicitly.

If you wait two weeks, I can ensure 100% accuracy. If you want it in one week, there might be things that still need to be pinned down.

Yuki’s underlying point is worth sitting with: making decisions on low-quality data defeats the entire point of data-driven decision making. Speed that produces bad numbers is a liability that takes three times longer to fix than the shortcut saved.

On Using AI When You Have Nobody to Think With

One thing Yuki said is worth pulling out directly, because it is the most honest framing of AI use I have heard from anyone working in the data space:

I use AI especially when I want to bounce ideas around. I use it because I don’t have anybody else to talk to, because I’m the only data person.

AI as a rubber duck rather than a replacement for judgment. At a startup where you’re the only data person, you have no peer to sanity check your modeling approach, your architecture choices, or whether your prioritisation call makes sense. AI fills a small part of that gap.

But Yuki also admitted he is not sure AI is making him faster overall. The time he used to spend building the solution, he now spends validating what the AI built. The total may net to zero, with the shape of the work changed and the volume unchanged.

That matches what I have seen. AI handles narrowly scoped, well-defined tasks well: understanding what a table column means by parsing the codebase, generating boilerplate inside a framework you have already defined, drafting initial SQL that you then refactor.

For architectural decisions and anything that requires sustained judgment, the model is an input, and the judgment stays with you.

Connect With Yuki

Yuki is very active online. Here are some of the best ways to connect with Yuki:

• Substack profile

• The Data Toolbox

• The Independent Insight

• LinkedIn profile

• The Polars Cookbook

• Consultancy website

—

Yordan

I was coaching Hamza last year.

He was one of the best engineers I had worked with outside my team. Sharp, reliable, the kind of person who fixes the production incident at 2am and never makes it anyone else’s problem. He had delivered more that year than most people deliver in two.

He got a 3% raise in January.

When we talked about it, the pattern was immediate:

• He had done everything right technically and nothing right strategically.

• He never flagged his expectations.

• He had no record of what he had delivered.

• He walked into the review meeting with memories and feeling instead of a number.

His manager liked him, but had no ammunition.

Getting a raise has almost nothing to do with whether you deserve one. It has everything to do with whether your manager can build a case for you in a discussion you are not in. The people who get raises are the people whose managers have the data to justify it, the evidence to present it, and the confidence the conversation will stay professional.

The research backs this up. According to Pew, 38% of workers who did not negotiate said they felt uncomfortable asking. The same study found that 66% of workers who did ask got what they asked for. Two thirds. The ask works.

This article is the system Hamza did not have before we worked together.

Download the complete manual at the end of the article.

Why Good Engineers Stay Underpaid

Most of the data engineers I know who are underpaid are genuinely strong. They ship, fix things before anyone notices they are broken, and mentor the junior engineers without being asked. They are the people their companies cannot afford to lose.

And yet they lose money every single year.

The problem is structural. Salary decisions do not get made by the person who sees your work every day. They get made in budget meetings, by people who know you mostly by reputation, based on recommendations from managers who have five other people to advocate for at the same time.

The engineer who gets the raise is the one whose manager walks into that room with the strongest case.

Everything else is secondary.

The math your company will never show you

Here is something your HR department knows and hopes you never think about:

According to SHRM, replacing an employee costs between 50% and 200% of their annual salary. For senior roles, the Center for American Progress puts that number at up to 213%. Recruiting fees, onboarding time, lost productivity, the six months before a new hire operates at full speed.

Giving you a 10% raise costs them 10% of your salary.

The math is on your side. The problem is you are not using it.

Subscribe now

Why the conversation never happens

The Pew Research Center found that 38% of workers who did not negotiate said they simply felt uncomfortable asking. Not a bad manager. Not bad timing. Discomfort is the real enemy.

So engineers stay quiet. They assume good work speaks for itself. They wait for their manager to bring it up. And their manager, who is busy and conflict-averse and assumes you would say something if you were unhappy, never does.

Nobody brings it up, and another year passes.

The 90-Day System

A raise is a system you run across the whole year.

The engineers who get what they ask for flag their expectations early, build a running record of their impact, research their market value before they need it, and invest in the right relationships long before budget season.

By the time the review meeting happens, the outcome is mostly already decided.

The rest of this article walks through each part of that system in full:

• How to flag your expectations early enough to actually influence the outcome

• How to build a brag list that gives your manager something to fight with

• How to find your market number before the conversation

• How to win the sponsors who have real budget influence

• How to run the conversation itself, including what to do when they say no

This is the system I walked Hamza through. He got a 13k salary increase. Keep reading and download the ebook at the end.

The content below is for paid subscribers. If you are not subscribed yet, this is the part worth paying for.

Read more

Presenting Prajakta Yerpude

Prajakta spent a decade climbing every rung of the data engineering ladder, intern, engineer, senior, lead, manager, and somewhere along the way realized her biggest impact was no longer in the code she wrote but in the people she unblocked.

Connect with Prajakta

Getting promoted into management felt like winning. I had the title, the 1:1s, the org chart with my name at the top. What nobody told me was that everything I had spent years getting good at had just become irrelevant.

The skills that got me the promotion were the wrong ones for the job. Speed, ownership, technical control, those are IC superpowers. In management, they become liabilities.

figured this out the hard way too, after a decade climbing every rung of the data engineering ladder. What she learned on the other side is clearer than anything I’ve read on the topic, and it maps almost exactly to where most senior data engineers quietly stall

The data teams that are ahead right now aren’t just experimenting with AI. They’re shipping agentic analytics systems that reason, recommend, and explain their work. And they’re learning things the rest of the industry hasn’t caught up to yet.

Cube’s Agentic Analytics Summit on April 29 is where some of those teams share what they’ve found. Dan Meshkov from Brex will talk about what a data foundation looks like for a mission-driven company. Gabe Romero from Jobber will cover how financial reporting changes when you move beyond charts. Joe Reis will dig into data engineering in the agentic era.

If you lead a data team or build data infrastructure, the insights here could save you months of trial and error.

Grab your free seat here

You Are Already a Manager Before the Title

Most people wait for the promotion to start operating differently. That is the wrong order.

The data professionals who get promoted into leadership are not the ones who wanted the title. They are the ones who were already doing the work before anyone gave it to them. Clarifying requirements with stakeholders when nobody asked. Helping teammates get unblocked when they had their own deliverables to ship. Asking the why behind the problem instead of just solving the what.

The Week Your Code Sits Untouched

There is a specific moment that signals the shift. You get to the end of a week and realize you made almost no progress on your own work — but the team moved forward because of the conversations you had.

That discomfort is the job changing underneath you.

If you wait until the title to start operating at the next level, you are asking your manager to take a bet on future behavior. That is a harder sell than showing them behavior that is already there.

Speaking Up Is a Skill

Early in any leadership journey, the most uncomfortable moment is the same for almost everyone: being in a room of more experienced people, knowing something is wrong, and deciding whether to say it.

The imposter syndrome is real. The difference is learning not to let it make the decision for you.

Facts Remove the Politics

In data, you have a specific advantage: you can ground a challenge in numbers and evidence. When you do that, you shift the conversation from opinion to information. The most experienced person in the room does not win by default anymore.

Leadership is about stepping up when something needs to be said. That is available to you at any level. An intern can flag a bad assumption. A senior IC can challenge a decision made three levels above them.

The cost of staying quiet is harder to measure than the cost of speaking up. But it compounds over time.

Empathy Is Leverage.

As engineers, we are trained to solve technical problems. The higher you go, the more you realize the hardest problems are people problems.

Misalignment. Stress. Lack of clarity. Someone having an off day. These do not have a clean solution. Some of them do not need a solution at all. Instead, they need listening.

The Problem-Solver Trap

The instinct to fix things immediately is exactly what gets senior ICs into leadership. It is also the thing that makes early management hard.

Not every situation calls for a solution. Sometimes the person in front of you just needs to feel heard. When you learn to read that distinction — when you can tell the difference between a problem that needs solving and a person who needs to know you are paying attention — the team responds differently.

People who feel understood and supported perform better without you pushing harder. You do not manufacture that with a process or a framework. It is built through consistent, genuine attention to what is actually going on beneath the surface.

Technical skills get you to the role. Empathy and emotional intelligence determine how far you go inside it.

Subscribe now

Your Impact Becomes Invisible. That Is the Whole Point.

As an IC, your output is visible. Code you wrote. Pipelines you fixed. A cost number you moved. You can point to it directly.

As a manager, your impact shows up through other people. Better decisions. Stronger execution. A team delivering consistently and growing in the process. None of that has your name on it.

Unlearning the Need to Own the Outcome

The hardest thing to unlearn is tying your personal value to what you directly produce.

Andy Grove put it clearly in High Output Management: your output as a manager is your team’s output. That is the measure. You do not have a separate output anymore. The team speaks on your behalf.

That means impact can look like:

• Creating the right direction when things are ambiguous

• Removing friction before it slows the team down

• Helping someone succeed in a way they would not have without your support

• Building the environment in which good work happens consistently

You stop building systems. You start building the conditions in which good systems get built. That is more scalable than anything you could ship yourself.

The Myth That Needs to Die

The most persistent management myth: a good manager always has to be in control of everything.

New managers fall into it. Experienced managers fall into it. They believe that effectiveness means knowing every detail, monitoring closely, and constantly stepping in. What that actually creates is dependency. It slows people down. It makes you a single point of failure.

The Real Measure

The actual measure of good leadership is when your team can collaborate, problem-solve, and resolve issues without you needing to be in the room. When something gets fixed before you even know it happened. That is the job done well.

High standards work better when trust already exists. When people know you genuinely support them, they are more open to feedback, more willing to stretch, more willing to take on work that makes them uncomfortable.

You can be kind and still be clear. You can be supportive and still hold people accountable. The balance is a more effective version of both.

Final Thoughts

The move from IC to manager is a step sideways into a completely different job that happens to sit inside the same organization.

The data professionals who stall at senior are almost always technically strong. They can build. What they have not yet built is the capacity to make others better at building.

If you are waiting until you feel ready to step into that, here is the honest version: you will never feel fully ready.

Growth happens because you stepped forward before you did.

—

Yordan

More on the Topic

These are some of the articles Prajakta mentioned in our chat:

• The Operating System Every Data Engineering Leader Needs

• Business value mapping playbook: How to translate data work into executive language

• How To Stop Waiting For Permission To Lead Data Engineering

• Why Senior Data Engineers Lose Their Velocity (Even When Their Skills Improve)

Reading time: 17 minutes

Nobody teaches you how to manage a data engineering team. You get promoted because you were good at the technical work, handed a calendar full of meetings, and expected to figure the rest out. Most people never do.

Less than half of data and analytics teams effectively deliver business value to their organizations, according to Gartner.

Not because data engineers lack skill. Because the leaders above them have no system for turning that skill into consistent delivery.

Why Managing Data Engineering Teams Is Hard

The skills that got you promoted are the wrong skills for the job you now have

You spent years mastering pipelines, architecture, and debugging. Those skills made you visible. They also made you exactly the wrong candidate for what leadership actually requires.

Technical skill is individual, but leadership is organizational. The moment you become responsible for a team’s output rather than your own, you need a completely different set of tools.

Nobody in data engineering ever gets taught what those tools are.

The instinct most new leaders follow is to stay close to the technical work. It feels productive, because you can add visible value there. What you are actually doing is avoiding the harder, less legible work of building a functioning team.

Most data engineering teams run on luck

85% of big data projects fail, and the primary causes are not technical. They are misalignment, poor communication, and leadership that cannot bridge the gap between what engineers build and what the business needs.

Your team probably looks fine from the outside. People are busy, work is moving. Nobody is visibly on fire. What you cannot see without a system:

• Who is blocked but not saying it

• Which project is slipping its deadline

• Whose work is duplicating someone else’s

• What the team thinks is a priority versus what actually is

By the time those things become visible, you are already behind.

The consequence of no system is invisible, until it isn’t

Failure in data engineering leadership accumulates.

A blocker sits for four days because nobody asked the right question. A project drifts because the goal was never written down. Someone burns out because you had no signal they were underwater. A stakeholder loses confidence because the same incident happened twice without any visible response.

Only 48% of digital initiatives meet or exceed their outcome targets globally. The organizations that hit 71% success rates share one thing: co-owned delivery with clear accountability structures between technical and business leadership. Not better engineers. Better operating infrastructure.

You need a system. Here it is (download the templates at end of the article).

The Operating System

The OS has two layers:

• Rhythm is the recurring rituals that create a shared cadence, so everyone knows what is happening, when, and what is expected.

• Memory is the processes that capture information so the team is not dependent on any individual’s recall.

Most leaders have a broken version of the rhythm layer. Almost nobody has the memory layer. That gap is where teams fail.

The four rituals that run a high-performing data engineering team

Each ritual has a specific job, and when you know what that job is, you stop letting them bleed into each other.

• Standups stop turning into planning sessions.

• 1:1s stop being status updates with one person.

• Check-ins stop being the meeting where people vent and nothing changes.

• Retros stop being a fake appraisal meetings.

The Daily Standup

Most standups are not a ritual

Your standup probably goes like this: everyone says what they did yesterday, what they are doing today, whether they have blockers. People say no blockers. The meeting runs 45 minutes. Nobody learned anything useful.

You keep running it this way because it feels like structure. It seems to work.

It does not work. It is a reporting session disguised as a coordination ritual. And the reason blockers never surface is that you are asking a binary question about blockers. But things slow you down long before they stop you completely.

Run the standup in three phases

Phase 1: Catch-up (first 10 minutes). No agenda. People talk, share what is on their mind, ask each other things. Research on high-performing data teams consistently identifies strong cross-functional relationships as a differentiating factor. Those relationships do not build inside structured agendas.

Phase 2: The three questions (remaining time). Every person answers:

• What are you currently working on? Not yesterday or tomorrow. Right now.

• Is anything slowing you down? Not “do you have blockers“. Slowing you down catches problems before they become stops.

• Do you need anything from anyone in this room? This is the question that makes the standup a coordination tool instead of a report.

Phase 3: Announcements (if time remains). Anything that does not fit elsewhere. If there is no time, skip it and end.

What to watch for

The standup tells you more through silence than through words. Watch for:

• The same task appearing three standups in a row without progress

• The person who never has anything slowing them down

• Everyone’s answers clustering around the same project for days

Each of those is a signal that a conversation needs to happen outside the standup. Never handle it inside it. The standup is not a debugging session for individual blockers. It is a radar sweep. You note what looks wrong and follow up separately.

The standup is not your briefing

If you are using the standup to find out what your team is working on, your system has a gap somewhere else. By the time your standup starts, you should already know.

The standup confirms what you know and surfaces what needs attention, it does not reveal your team’s work to you for the first time.

The 1:1

Most 1:1s are status updates with one person

• You meet

• You ask what they are working on

• They tell you

• You talk about a project

• You schedule the next one

That is a meeting that could have been a Slack message and would have been better as three bullet points.

You keep running it this way because it feels efficient. What is actually happening is that you are using the highest-leverage ritual in your calendar to do the lowest-leverage thing possible.

70% of the variance in employee engagement is attributable to the quality of management. The 1:1 is the primary channel through which that quality gets expressed, or wasted.

Structure the 1:1 so the engineer sets the agenda, not the manager

The 1:1 runs in three parts:

• Their agenda (first 10-15 min): Ask “what do you want to cover today?“ then stop talking. If they have nothing, that is the most important signal in the room.

• Your agenda (next 20-30 min): Feedback collected since the last 1:1. Career conversations. Tensions you observed. Things that have no home in any other ritual.

• The close (last 5-10 min): Two or three things most important for them to focus on before you meet again. Anything you committed to follow up on.

Silence in a 1:1 is the most important data point

When someone has nothing for their agenda, there are two explanations:

• They are heads-down and genuinely fine

• They have learned that bringing things up here does not lead anywhere

Your job is to figure out which one it is. Ask directly: “Is there anything you have been sitting on that you have not brought up?“ Most people will tell you if you actually ask.

Forgetting what you committed to is a management failure

Every commitment you make in a 1:1 and forget by the next one erodes trust. The person noticed. They always notice.

Keep notes after every 1:1. Just two or three sentences: what was discussed, what you committed to, what to follow up on. This is the difference between a leader who has a relationship with each person on their team and one who is starting from scratch every two weeks.

By 2028, one in four regretted staff attritions will be attributable to managers’ lack of data literacy. Employees leaving not because they lack skills, but because their managers cannot develop or support them. The 1:1 is where that problem either gets caught early or quietly compounds into a resignation.

The Sprint Check-in

Skipping the sprint check-in is why your delivery keeps drifting

Most data engineering leaders skip this ritual. They have standups and 1:1s. But because you have no prioritization process and put fires all the time, you never really need a planning meeting.

The standup is too short for sprint-level visibility. The 1:1 covers one person at a time. The sprint check-in is the only ritual where the whole team looks at the same picture together and asks whether it is accurate.

Without it, delivery drift is invisible until it becomes a missed deadline.

Review delivery honestly

The sprint check-in runs in three parts:

Delivery review.

• What was the sprint goal?

• What got done?

• What did not?

For everything that did not land, the question is not who is responsible. The question is “what did you not understand when you planned this?”. Estimation fails because of missing information. Naming what was missing is the only way to plan better next sprint.

Blocker review. Not individual blockers that got resolved, but patterns. If access issues slowed three engineers this sprint, that is a systemic problem someone needs to own.

Research on data team performance specifically identifies shared accountability for outcomes over reactive firefighting as a behavior that distinguishes high-performing teams. The blocker review is where that distinction gets made real.

Next sprint preview. Ten minutes of context before planning starts.

• What matters most in the next two weeks?

• What is the business reason?

• What does the team need to know that will change how they work?

This is where you close the gap between engineers who are executing tasks and engineers who understand why those tasks exist.

Patterns you refuse to name in the check-in become permanent

The temptation in every check-in is to stay positive. Things mostly went well. The miss was a one-off. You will do better next sprint.

That refusal to name patterns is how recurring problems become permanent friction. The same root cause surfaces every two weeks. Everyone knows it. Nobody says it out loud.

Nothing changes.

Say it out loud. Assign it. Fix it or decide you are not going to fix it. Either is better than pretending it is not there.

The Team Retro

Most retros fail because they produce catharsis, but not change

People raise the same things retro after retro. Requirements come in incomplete. Estimation is always off. Two people are still not communicating clearly. And nothing is different.

Eventually, people stop raising real issues. You have trained them that feedback is a ritual, not a mechanism.

You keep running retros this way because producing a long list of observations feels productive.

It is not. A list with no owner, no timeline, and no definition of done is documentation of complaints.

One change per retro, not a wishlist

Run the retro in three columns:

• What went well (5 min). Round-robin, read aloud, group themes. Skip this column and the team starts feeling like everything is broken.

• What could be better (15 min). Push for specificity. “Communication is bad“ is not a problem. “I found out the pipeline broke the downstream model when three stakeholders were already angry“ is a problem.

• What we are doing about it (remaining time). One thing per owner. One definition of done.

That constraint is the entire point. One change per retro means it actually happens. Three changes means two of them disappear by Tuesday.

Protect the third column

The reason most retros fail is that the first two columns consume almost all the time. The third column gets five minutes of tired people picking something vague before everyone has to leave for the next meeting.

Timebox the first two. Hard stop. The third column is the only reason you are in the meeting.

The Memory Layer

Your team runs on short-term memory and it will eventually cost you

Most data engineering teams capture almost nothing. Work moves through people’s heads, conversations happen in Slack and evaporate, decisions get made and their reasoning is never recorded.

This looks fine until someone is out sick for a week, or a senior engineer leaves, or a stakeholder asks why a pipeline works the way it does and the only person who knows has not worked there for two years.

Silos of knowledge, where critical information exists only in individual heads, are one of the four core failure modes of data science teams, alongside deployment friction, tool mismatches, and unmonitored models. The fix is building the habit of capturing information as a team practice.

Know your people beyond what they are working on

Before you can lead someone well, you need to know them. Not their job title or years of experience. Who they actually are as an engineer and where they are trying to go.

For each person on your team, you need a current, accurate picture of:

• What they are actively developing this quarter

• What their current capacity actually is, not what it looks like in the standup

• What kind of work drains them versus what makes them do their best work

• When you last had a real career conversation with them

Research on team development identifies consistent failure patterns in poor management: waiting too long to address issues, and treating development as one-size-fits-all. Both failures come from the same root cause:

The manager does not have an accurate, current picture of each person.

Update what you know about each person after every 1:1 cycle. The leaders who skip this are running their team on impressions. They are surprised by resignations that should not have been surprising.

Track active work so the standup confirms what you already know

Every piece of active work needs four things:

• One owner. Not a team, not a pair, one person accountable for it moving

• A current status - Not Started, In Progress, Blocked, In Review, Done

• The project it belongs to

• A flag if it is blocked or at risk

That is the whole system. Five status states. One owner. Two flags.

The hardest part is getting the team to maintain it. If you are updating the task list yourself, you have taken ownership of something that belongs to your engineers. Your job is to check it, ask about what looks wrong, and make updates non-negotiable.

Watch the Blocked status most closely. Anything sitting there for more than two days without movement is a conversation you need to have before the next standup.

Document projects so decisions survive the people who made them

Every significant project needs a goal, a set of stakeholders, a timeline, and a definition of done, written down before the work starts.

When a project does not have a documented goal, the first thing that happens when it hits a snag is that everyone has a different idea of what success was supposed to look like. Scope drifts. Decisions get made by whoever speaks loudest. The timeline slips in ways nobody can explain because nobody wrote down what was originally intended.

Only 48% of digital initiatives meet their outcome targets globally. The organizations that hit 71% share one characteristic: success was defined and co-owned before delivery started, not reconstructed after it missed.

Review active projects weekly. Look for what is at risk and what has not moved. Both are signals that need attention before they become announcements in a stakeholder meeting.

Treat meetings as data

Every meeting your team has should produce three things:

• what was discussed

• what was decided

• what happens next

Write three or four sentences somewhere the whole team can find them.

The reason is pattern recognition.

When the same problem surfaces in sprint three that surfaced in sprint one, you want to know whether you discussed it before. When a retro action disappears without anyone noticing, the record shows it was assigned. When a stakeholder claims a decision was never made, you have the date it was.

High-performing data teams consistently maintain visible decision logs and documented assumptions. Most teams treat meetings as ephemeral. The teams that learn fastest treat them as the raw material for every difficult conversation they will eventually need to have.

Run postmortems to learn from failures, not to survive them

Things will break. Pipelines will fail. Data will be wrong. Incidents will happen.

The question is not whether. It is whether anything changes because of it.

A postmortem is a structured conversation after a failure:

• What happened

• Why it happened at the system level

• What the fix was

• What changes prevent the same failure recurring

Blameless postmortems are not a kindness to your engineers. They are the only way to get accurate information. When people fear blame, they tell you a version of events that protects them. You treat a symptom while the root cause stays intact and the same incident happens again six weeks later.

Unmonitored systems and processes without structured review are a core failure mode of data science teams, and the damage compounds. Each unresolved failure erodes stakeholder confidence and makes the next investment request harder to justify.

Over time, the postmortem record becomes a knowledge base:

• why things work the way they do

• where the structural fragility is

• what has been tried before

Most data engineering teams treat incidents as things to get through. The teams that improve treat them as information.

Your First Week With the OS

You do not need to build the whole system at once. You need to start it.

Day 1: Know your people. Write down everything you know about each person: current focus, development goals, capacity. The gaps you find are your agenda for the first 1:1 cycle.

Day 2: Set up the standup. Do not just block the calendar. Send a message explaining the format, the three questions, the catch-up phase. Set expectations before the first one runs.

Day 3: Pull active work into one place. Whatever lives in Jira, Notion, Slack, or someone’s head, get it into a single list with one owner, one status, one flag for blocked or at risk.

Day 4: Schedule 1:1s. Five or fewer reports: weekly. More than that: bi-weekly. Send the invite with a short note about what you want to cover in the first one, and that the first ten minutes are always theirs.

Day 5: Put the sprint check-in and retro on the calendar. They do not have to happen this week. They have to exist in the calendar so they do not become things you fight for space for later.

The goal is that next Monday looks different from last Monday.

What Changes When You Have an OS

You stop being surprised by things that were always visible

The most significant change is timing.

When work is tracked, when standups surface blockers, when projects have documented goals and weekly reviews, you see problems coming.

Not always early enough to prevent them, but hopefully, early enough to respond before they escalate.

The shift from reactive to proactive is the one that changes what leading a team actually feels like.

Your engineers stop carrying problems alone

One of the most expensive things that happens in data engineering teams is invisible load: Someone is stuck. They do not want to admit it. They sit on it for three days. By the time it surfaces, a week is gone and the sprint is already at risk.

Rituals designed to surface that load. People stop carrying things alone because the system makes it normal to say when something is wrong.

Your most difficult conversations get easier

Performance reviews stop being stressful when you have 1:1 notes instead of six months of impressions. Reliability conversations with stakeholders have postmortems behind them. Headcount requests have project history and capacity data to support them.

48% of HR leaders agree the demand for new skills is evolving faster than existing talent structures can support, which means every conversation about team development and investment is becoming more important, not less.

The leaders who can back those conversations with evidence have more credibility and get more resources. The ones running on instinct get told to come back with data.

You are a data person. Come with data.

Get the Full System

Everything in this article is built into a Notion template you can copy and use today. All five components:

• team tracking

• active work

• project management

• meeting records

• postmortems

Everything is pre-configured with the structures, properties, and templates that match what you read above.

Paid subscribers get the Operating System for free here

Read more

I had a chat with Mihail, a software engineering lead who also lectures at Plovdiv University. He teaches data engineering as an elective because the core CS curriculum has zero room for it.

That tells you everything about the state of data education.

Universities produce graduates who know SQL syntax, basic normalization, and enough theory to pass an exam. Then those graduates walk into jobs where the theory is the easy part. The pipelines, the warehouses, the cloud platforms, the stakeholder conversations, the ambiguity of real problems. None of that shows up in a lecture hall.

I got my databases grade in university and it was mediocre. I didn’t care because the course felt abstract and disconnected from anything real. Most students feel the same way. And the system does nothing to change that.

The gap between what university teaches and what the job demands keeps growing. But the fix is simpler than most people think, and it starts long before graduation day.

The Curriculum Stops at SQL

University data education peaks at a single databases course in your second year.

You learn what SQL is, what a database does, and some theory about normalization and data storage. The course is abstract. Students treat databases like glorified Excel files because nobody shows them why the syntactical overhead matters.

Until you work on large-scale projects, the difference between a spreadsheet and a relational database feels like a bureaucratic formality.

And that databases course is the baseline for everything data-related in the entire degree.

Everything Beyond SQL Is “Too Specialized”

From that point, data science branches into math and statistics. Data engineering branches into pipelines, warehouses, cloud platforms, ETL, governance. The university follows neither branch with any depth.

The reason is structural. Introducing a new subject into a CS curriculum takes years of bureaucratic and administrative effort.

Universities teach foundations because foundations are stable. Specific technologies move too fast for a system designed to change slowly.

• SQL is foundational enough to make the cut

• Snowflake, Databricks, dbt, Airflow are all “too specialized”

• Anything resembling a real data engineering stack gets classified as professional training, outside the university’s scope

The 10-Year Lag Is Built Into the System

Mihail put it bluntly. Universities are roughly a decade behind the industry. By the time a discipline earns a permanent spot in the curriculum, the profession has already moved on to the next generation of tooling.

The workaround at Plovdiv University is elective courses. Mihail and his colleagues created an entire data engineering track outside the required curriculum.

Students who are dedicated sign up semester after semester and piece together the big picture on their own initiative. A full data engineering discipline exists in the master’s program, but only because individual lecturers pushed for it.

The core curriculum still treats data engineering as a niche profession. The students who figure out it matters do so despite the system, not because of it.

Hard Skills Used to Be Enough. They Aren’t Anymore.

A few years ago, knowing one programming language and one framework got you hired.

The COVID-era hiring boom rewarded narrow skill sets. Companies needed seats filled. They needed people who could write code in a specific technology and ship tickets.

Thousands of junior developers entered the industry every month on the back of a single concentrated skill. No curiosity required and no product understanding expected. Know React, get a job.

That worked because the economics supported it.

The Market Shifted and the Craftsmen Got Stuck

The economics changed. Companies stopped hiring for volume. AI tools absorbed the foundational knowledge work.

People who built their entire career around one framework or one language found themselves competing with a chatbot that explains object-oriented programming better than most university courses.

Information is everywhere now. What separates a valuable engineer from a replaceable one is everything around the information.

• Understanding the product and the customer

• Adapting when the tooling changes underneath you

• Operating with judgment in ambiguous situations

• Communicating with stakeholders who speak a different language than you do

Universities Still Train for the Old Game

The university teaches you how to use tools. The job requires you to understand why you’re using them and what problem they solve for the business.

Mihail made a sharp distinction. Universities produce IT people, not developers. The degree gives you logic, math, and a surface-level tour of programming concepts.

But the industry needs people who solve problems related to communication, technology, and economics. Those are three different skill sets, and a CS curriculum addresses one of them on a good day.

The people who stall are the ones who treat graduation as the finish line. The ones who keep going treat it as a starting point for a much longer education that happens inside the business, inside the community, and inside the messy reality of real projects.

Your Network Is Your Actual Career Engine

The cliche says your network is your net worth. It holds true even when you look for a job.

Blasting CVs across LinkedIn is the path of least resistance. It feels productive because you’re doing something. But it’s a low-leverage move that puts you in a pile with hundreds of other applicants who look identical on paper.

Mihail’s company has never posted a single job offer on a popular platform. They hand-pick students directly from the university. Every hire comes through relationships built during lectures, projects, and community events.

That pattern repeats across the industry more often than people realize.

Show Up Where the Decisions Happen

Conferences and user groups are where project leads, hiring managers, and senior engineers gather to talk about the work they care about. These people are approachable in that setting. They want to talk shop.

The move is to ask about their problems, not their openings.

• What kind of scaling challenges are you dealing with?

• Where does delivery break down on your projects?

• What does your data stack look like and where does it hurt?

These questions signal curiosity and understanding. “Do you have any open positions?“ signals desperation.

One leads to a conversation. The other leads to a polite nod and a business card that goes nowhere.

Subscribe now

Make Yourself Visible Before You Need a Job

If you’re early in your career and don’t have deep experience to share, you still have options. A GitHub project, a blog post about a technology you’re learning, a short tutorial, a LinkedIn thread documenting what you figured out this week. None of these need to be brilliant. They need to exist.

Visibility compounds. The people who run conferences and user groups see the same faces year after year. They remember your questions, and when a role opens up on their team, they think of the person they’ve been talking to for the last six months before they think of the 200 CVs sitting in their inbox.

I found my current company through conferences. Several people I’ve worked with came from user groups and community events. This is how the industry works when you look past the job boards.

Find Misho

Here’s how you can find more about Mihail and his initiatives for data newcomers.

Everything he does at this point is in Bulgarian, but Mihail promised to start building educational content in English, too.

• LinkedIn profile

• Snowflake user group

• Free Snowflake course

Closing Words

The university gives you a starting line. Logic, math, a community of confused peers who share your ambition. That matters more than the cynics admit.

But the finish line is somewhere else entirely.

The skills that make data engineers valuable are built in the space between academia and industry. Pipelines, product thinking, stakeholder conversations, judgment under ambiguity.

No curriculum covers this. No lecture hall prepares you for the moment a stakeholder asks you to scope something that has never been done before and expects an answer by Thursday.

The people who figure this out early have an unfair advantage. They show up at conferences while still in their second year. They build side projects that prove curiosity, not mastery. They surround themselves with people who are ahead of them and ask questions until the vocabulary clicks.

The people who wait for a curriculum to teach them what the job demands graduate with a degree and a gap where their career momentum should be.

University is a foundation. Treat it as one. Build everything else yourself.

—

Yordan

More on the Topic

• The Surprising Trait That Beats Experience

• Level-up Data Engineering

• Why the Hard Skills Obsession Is Misleading Every Aspiring Data Engineer

Subscribe now

I walked into a performance review expecting a conversation about growth. My manager told me he saw me staying close to data architecture, going deeper into technical design, owning more of the platform layer.

I wanted the opposite. Ten years in tech at that point, and the purely technical challenges had stopped being the draw. I enjoyed talking to people, managing people, making decisions across teams. I had spent the past year doing more of it informally and assumed everyone noticed.

My manager had no idea. His response stuck with me: “You should have told me this months ago. We needed time to set the right goals and show results before this review“.

He was right. I had been skipping 1:1s, walking in with “I don’t have anything to share,” or spending the full 30 minutes listing what my team shipped that sprint. The one meeting designed for me to shape my own direction, and I treated it like overhead.

Your 1:1 is the single most strategic meeting on your calendar, and chances are you prepare for it the least.

“I Don’t Have Anything to Share”

Five words that cost more than any failed project.

I used to say this all the time. The week felt routine, nothing blew up, the team was heads down on planned work. Cancel the meeting, reclaim the 30 minutes, get back to real work.

The Hidden Assumption

The assumption behind this is that 1:1s exist for problems. If nothing is broken, the meeting has no purpose.

Data professionals fall into this trap faster than most because the work feels self-evident. The pipeline runs or it doesn’t. Progress is visible in commits and deployments, so reporting it out loud feels redundant.

But your manager is not tracking your commits. They are building a mental model of who you are, what you want, and where you are headed.

Every time you cancel, every time you show up with nothing, you hand them a blank canvas and let them paint whatever picture makes sense from their vantage point.

Silence Gets Filled With Assumptions

My manager painted a data architect, but I wanted to lead people.

That gap did not form in one bad conversation. It formed across years of silence I chose because I thought the work spoke for itself.

The work says what got done. It says nothing about where you want to go, what is frustrating you, or what you need next. Those things come out when you say them, and the 1:1 is the meeting built for exactly that.

If you keep canceling it, your manager will fill the gap with assumptions. And you will sit in a performance review one day hearing a version of your career you never signed up for.

Give the Update, Skip the Details

People hear “stop giving status updates“ and swing to the other extreme.

They walk in talking about career goals and strategic direction while their manager sits there wondering if the migration is still on track.

The Update Serves a Purpose

Your manager needs a status check. They need to know you are spending time on the right things and nothing is stuck. The update gives your manager confidence that the team is pointed in the right direction and no one is burning cycles on low-leverage work.

Give them the high-level overview. Five minutes, tops. What is in progress, what shipped, what is blocked.

If your manager is technical, they will follow fast and ask sharp questions. If they are not technical, they do not care about the implementation details anyway.

The Mistake Is Filling 30 Minutes With Five Minutes of Content

The real problem is stretchingthe update to fill the entire slot because you did not prepare anything else to talk about.

The update is the warmup. It earns you the right to spend the remaining 25 minutes on the things that shape your role, your trajectory, and your relationship with your manager.

What comes after the update is where the 1:1 becomes a strategic meeting instead of an administrative one. And that is where most data professionals leave the biggest gap.

Now let me break down exactly what to bring to that remaining time, how to prepare for it in ten minutes, and how to reset the dynamic if you have been running empty 1:1s for months.

Read more

This article is part of the Lead & Grow playlist. Click here to explore the full series.

I spent three years shipping some of the best engineering work of my career and getting almost zero recognition for it. I built solid pipelines, clean models and efficient infrastructure. And nobody outside my immediate team had any idea what I had done or why it mattered.

I thought the work would speak for itself. It didn’t. Work never does.

The problem was the language. I described everything in the terms I built it in, and those terms meant nothing to the people who controlled my career trajectory. I had a translation problem, and I didn’t even know it existed.

The Language Gap That Keeps You Invisible

Here is the thing about technical work: The better you do it, the more invisible it becomes.

A pipeline that runs flawlessly at 3am generates no Slack alerts, no urgent emails, no visible crisis for anyone to notice. Your success looks like nothing happened.

So when someone asks what you have been working on, you say something like “I migrated the ingestion layer from ETL to ELT“ or “I refactored the dbt models to use incremental materialization“. And the person in the Zoom square nods politely and moves on, because those words carry zero weight in their world.

The people deciding your promotion, your project funding, your headcount don’t think in pipelines and materializations. They think in revenue, cost, and risk.

They won’t. They have their own problems to think about.

So the credit goes to whoever already speaks their language. The PM who presents the dashboard you built, the analyst who shares the insight your pipeline made possible get the visibility because they describe the outcome, and you described the plumbing.

This pattern is career poison, and it compounds. One invisible quarter becomes a year of “solid contributor“ ratings that never break through to “high impact“. The frustrating part is that you are already doing high-impact work. You are describing it in a way that buries the impact.

The Three-Column Translation

I use a framework with my direct reports that fixes this. It forces you to take any piece of technical work and push it through three columns until it lands in language a CFO would repeat back to you.

Column 1: Technical Input. What did you change? Eight words max. Use past tense. “Migrated ETL to ELT“. “Refactored the ingestion layer“. This is where most engineers stop, but that’s the least important column.

Column 2: Enablement Outcome. What became faster, more reliable, or self-serve as a result? Put a number on it. Percentage improvement, hours saved, or direct dollar impact. “Analysts build and modify transformations without engineering tickets“ or “Query runtime dropped from 90 to 18 minutes“.

Column 3: Business Driver. Which business bucket does this fall into: revenue, cost, or risk? Name the specific stakeholder behavior or decision that moves the number. This is the column that matters, and this is the column most engineers skip entirely.

Here is what the full translation looks like in practice.

An ETL to ELT migration:

• Column 1: “Migrated ETL to ELT”

• Column 2: “Analysts build and modify their own transformations directly in the warehouse without filing engineering tickets“

• Column 3: “Product team runs independent analyses, cutting 2 weeks from the launch decision cycle“

A dbt refactor:

• Column 1: “Added daily partitions“

• Column 2: “Runtime dropped from 90 to 18 minutes, 80% reduction“

• Column 3: “72 analyst hours freed per month, equivalent to $1.2M in retention analysis capacity“

Look at the difference between Column 1 and Column 3. Column 1 is what you did. Column 3 is why anyone should care.

Read your Column 3 out loud and ask yourself if the CFO would repeat it in a sentence. If it still sounds technical, it hasn’t crossed the line yet.

Stories Beat Updates

Translation is half the problem. Delivery is the other half.

Most data engineers send updates. You share what you did, what changed, and what comes next. The information is accurate and nobody acts on it, because updates are passive. You inform, but you don’t move anyone toward a decision.

The fix is a four-part structure that turns any update into something people act on:

• Context: what problem were you solving?

• Insight: what does the work show?

• Impact: why does this matter to the business?

• Action: what should happen next?

That last step is the one everyone leaves out, and it is the most important one. Send an update without a recommended next step and your audience will do nothing. They will read it, nod, and go back to their inbox. End with a clear recommendation or a specific decision you need from them, and the message becomes a conversation.

The language matters too. Drop the engineering voice when you talk to business stakeholders. Lead with what surprised you and what changed. Use phrases like “what stood out to me“ and “the implication for the team is“ because they signal there is a human being behind the update with a point of view, not a system generating a status report.

A story that leaves someone thinking the same way they started is unfinished. If your update doesn’t change what someone believes or what they do next, it was noise.

Final Thoughts

The data engineers I see get promoted are rarely the best builders on the team. They are the ones who learned to describe their work in terms that register with the people holding the budget, the headcount, and the promotion decisions.

This is a skill. It’s easy to learn, and most people never develop it because nobody tells them it exists.

Everything I covered here is a piece of a larger system I built and use with my own team. The Business Translation Kit, including the interactive worksheet, the KPI mapping framework, and the executive framing templates, is available as a free resource.

Shut up and give me the kit

More on the Topic

• Talk Techie to Me: Translating Data Complexity for Business Leads

• Business value mapping playbook: How to translate data work into executive language

• 4 awkward stakeholder conversations you have to master if you want more money and respect

This newsletter is funded by paid subscriptions from readers like yourself.

If you aren’t already, consider becoming a paid subscriber to receive the full experience!

See what people say about working with me.

You are more than welcome to find whatever interests you here and try it out in your particular case. Let me know how it went!

Upgrade Now

Data-driven the way subscription businesses actually operate it. Numbers connected to decisions, decisions connected to outcomes.

Below are the ideas she shared. The full conversation is in the video above.

Stakeholders Make Emotional Decisions First (and They Have Good Reason To)

The standard data professional complaint is that stakeholders ignore data and run on gut. Yana flips that framing.

Her view:

Most people are wired to make emotional decisions first and look for data that confirms them second.

From her background in neurolinguistic programming, she puts the share of people who genuinely process the world through numbers at around 15%. Everyone else needs a different approach.

Showing up with a dashboard and expecting it to land is wishful thinking. Data needs a story around it. A number without context gets interpreted differently by every person in the room. One sees growth, another sees underperformance, a third questions whether the metric tracks the right thing at all.

Yana learned this early. Every conversation with engineering teams felt like friction. She was asking questions they thought she should already know. They were building things that missed what she actually needed. The gap cost both sides time.

What closed it was her decision to learn enough about how systems work to ask better questions. To stop being helpless in technical conversations.

The One Move That Ends Arguments in Any Meeting

Yana has one technique she uses across every organization. It works in telecom. It works in banking. She has a perfect record with it.

Before any contentious conversation, find the data that links your ask to what the organization actually cares about. Customers, revenue, profit. Those three. Then add competitive benchmarks if you can find them.

In her current bank, she came in and asked how many successful app registrations they were getting. She pulled the internal rate, found competitor data, and discovered some competitors were achieving double or triple the registration conversion rate. She put that in front of the team.

The conversation changed.

People stop defending their position when the data shows the position is costing them. Go into the meeting with the numbers already pulled. Link them to something the people in the room care about. Then let the data do the work.

This applies directly to data teams. When a project needs prioritization, funding, or unblocking, skip the technical justification. Show the business number it connects to, then show where a competitor is already ahead.

What Yana Actually Tracks on Substack

Substack’s native analytics fall short. Yana built her own spreadsheet and tracks the following every month:

• Total subscribers

• New subscribers

• Total paid subscribers

• New paid subscribers

• Churned paid subscribers

• New revenue

• Average revenue per subscriber per month

• Average revenue per subscriber per year

• Free-to-paid conversion rate

• Net Promoter Score

The benchmark Yana uses for revenue per subscriber is the email marketing industry average, ranging roughly from $1 to $12 per subscriber per year depending on niche. Above average means the business is healthy. Below average means something needs auditing.

The NPS tracking is the piece most Substackers skip. She uses Substack’s survey feature to ask paid subscribers one question: on a scale from 1 to 10, how likely are you to recommend this publication to someone you know. That one question, run through the standard NPS formula, tells her whether her paid tier delivers real value. Her target is above 70.

Yana’s reasoning:

NPS is the only reliable signal for whether paid subscribers are satisfied or are simply on their way out.

How She Runs a $10K/Month Business in the Hours Left Over From a Full-Time Job

Yana works five days a week, eight hours a day, in an office. She can answer Substack comments during lunch or late after work. That’s the window she operates in.

AI handles everything operational. Formatting, research, scheduling, the mechanical parts. For writing, she trained a system on her own content. She extracted all her published articles into a database, connected it to an MCP server, and built what she calls a second brain. She can query it for content gaps, understand what paid subscribers want more of, and run analysis on her own business data.

She writes in her voice, informed by her own data, with AI executing the work that requires no judgment.

Her warning about AI:

It pulls you into rabbit holes when you let it.

Every answer suggests the next step, and the next step, and an hour later you have built something with no clear business purpose. Use it to execute a decision. Know exactly what you want before you open it.

Yana also made the point that the 9-to-5 built the Substack. The telecom career gave her subscription business knowledge, customer journey expertise, conversion rate benchmarks, and an understanding of which metrics actually predict business health.

All of it went directly into how she runs the newsletter. The side business grew because of the day job.

Final Thoughts

Yana has 15 years of subscription business experience, a disciplined tracking system, and a clear rule: link your ask to numbers that matter, then find the benchmark that shows where you stand.

That’s the part worth sitting with. Data professionals spend time arguing that stakeholders ignore data. Yana learned enough about systems to have real conversations with engineers. She built her own analytics because the platform’s fell short. She tracks NPS because her industry background taught her it mattered.

The gap between business and data closes when people on both sides decide to move. Yana moved. The engineers still telling business people to figure it out on their own are the ones falling behind.

Interested into growing your Substack? Subscribe to Yana’s publication.

This newsletter is funded by paid subscriptions from readers like yourself.

If you aren’t already, consider becoming a paid subscriber to receive the full experience!

See what people say about working with me.

You are more than welcome to find whatever interests you here and try it out in your particular case. Let me know how it went!

Subscribe now

I got this wrong. I read the counterarguments. I talked to people who genuinely believe in the credential economy. And I arrived at an uncomfortable conclusion: the market has spoken, and the market wants badges.

So I built one.

If you have been waiting for a certification that actually reflects what the modern data industry rewards, your wait is over. You can get certified today, and it will take you less time than your next standup.

What You Can Get Certified In

There is only one credential that matters right now, and it is the Licensed AI-Powered Insights Leverage Specialist™.

It validates your enterprise-grade proficiency in leveraging cross-functional data synergies across the modern AI-powered stack. If you have ever sat in a meeting where someone said something like that and nodded along, you are already overqualified.

The exam is five questions. Multiple choice. The pass rate is 100%, which puts it well ahead of most vendor certifications in terms of respecting your time.

Take It For FREE

Your official certificate is issued immediately upon completion by the Data Gibberish Institute for Enterprise Excellence & Advanced AI-Driven Thought Leadership™. No waiting. No retake fees. No two-year recertification treadmill.

Why You Should Stop Waiting

You have seen the LinkedIn posts. Someone announces their fourteenth certification. Three thousand likes. Fire emojis from people who have never shipped a pipeline in their lives. The comments say “congrats, you inspire me“.

Fourteen certifications. At some point you have to stop fighting the current.

The market does not reward taste. It rewards signals. And if a badge is the signal that gets you hired, promoted, or taken seriously in a room full of people who have no idea what you actually do, then the question is not whether to get certified. The question is why you have waited this long.

Your peers are out there collecting credentials. Your LinkedIn profile is sitting there, badge-free, quietly judging you. This is your chance to fix that.

Get Certified Today

Final Thoughts

Go get certified.

When you are done, you will hold exactly as much proof of competence as most of the certifications currently listed on your colleagues’ LinkedIn profiles. The difference is that this one is honest about what it is.

Take The Exam

Happy April 1st.

More on the Topic

• The Certifications Scam

• 5 Exclusive Soft Skill Lessons You Can Learn From Senior Data Engineers

• Beyond Senior Data Engineer: Skills & Strategies for the Next Big Step

This newsletter is funded by paid subscriptions from readers like yourself.

If you aren’t already, consider becoming a paid subscriber to receive the full experience!

See what people say about working with me.

You are more than welcome to find whatever interests you here and try it out in your particular case. Let me know how it went!

Subscribe now

Doing the same thing with fewer people is not a transformation. It is a managed decline with better PR. And the people who pay for it are the ones who showed up every day trying to do good work inside a company that had already stopped believing in itself.

If that’s you, keep reading.

AI Expands What You Can Build

I want to be specific here, because vague encouragement is BS.

Since the newest Claude models dropped, I have been committing code to GitHub every single day. Including weekends and evenings. Not only for work. I want to build things and I finally have the tools to do it fast enough to stay interested.

I am also hiring all the time. This week I am interviewing for another analytics engineer on my team. The work is growing. The scope of what we can do in a week now would have taken a month two years ago.

That is what AI does to a team with a direction. It multiplies output. It raises the ceiling on what one person or one small team can ship. It turns “we don’t have the bandwidth“ into a much harder excuse to make.

So when a company looks at that same capability and decides the right move is to cut 15% of its workforce to maintain current output, pay attention to what they are telling you.

They are not optimizing or transforming. They looked at a bigger ceiling and decided they had no interest in reaching it.

That is a product and strategy problem. It’s a leadership failure.

AI handed every company in the world a way to do more. Some took it and used it to do less, cheaper. The ones doing less cheaper already knew they were out of moves.

The Real Leadership Problem

“Doing more with less“ sounds responsible. It gets nodded through in board meetings. It shows up in earnings calls as discipline.

It is a tell for leaders who stopped asking what’s next.

Growth requires a thesis. You need a belief about where the market is going, what your customers need in two years, what you can build that nobody else can. That thesis is what turns a capability like AI into a hiring reason instead of a cutting reason.

When that thesis is missing, every new tool becomes a cost lever. AI shows up and the question leadership asks is “how many people can we remove from the org?“

That question is a symptom. It means the people at the top are managing a position are trying to hold margin on a product they stopped believing in. The layoffs are a controlled retreat dressed up in transformation language.

And the people who get cut pay for that failure.

I have worked with leaders who had a thesis and leaders who didn’t. The difference is obvious inside six months. One type makes you feel like the work is expanding. The other makes you feel like every quarter is a negotiation over who stays.

If you have been in that second environment, you already know the feeling. It sits in your chest in every all-hands.

You Got Pushed Out of a Dead End

I know it hurts. You might have a family to feed, bills that do not pause for your job search, a mortgage that does not care about market conditions. I am not going to tell you it does not sting.

But here is what is also true.

You were working for people who had already given up on their product. The trajectory you were on was going nowhere. The ceiling in that company was coming down slowly, quarterly, in ways that got explained away in town halls.

Getting pushed out of that is an exit from a room that was getting smaller and smaller over time.

The thing that matters now is what you do with the tools in front of you. AI is not going anywhere. The capability is real. I built more things in the last two months than in the two years before them. Some were useful, some were experimental and some were pure jokes. All of them taught me something.

The knowledge you have took years to build. The context you carry about real problems in real businesses is something most people building products right now are guessing at.

Stop waiting for someone to hand you a roadmap. Build something, ship something small. Get your hands moving.

The people who come out of this period ahead treated the disruption as an opening. You got freed from a dead end. That is a brutal way for it to happen. It is still a door.

Final Thoughts

The data professionals who come out of this period ahead made a decision early. They looked at AI and asked what they could build with it. They did not wait for permission or for a company to hand them a strategy, but picked something and started.

The ones who struggled treated AI as a threat to manage. They watched the news cycle, worried about their job, hoped their company would figure it out.

WTF were those companies thinking? Honestly? A capability leap like this comes along once in a generation and the best move some leadership teams could come up with was a headcount reduction.

The tools are cheap and he knowledge you have is expensive to replicate. The gap between what a skilled person with AI can ship today versus two years ago is wide enough to build a product in, start a service on, grow a team around.

I am not special. I am just moving. If you start moving too, you will be surprised how fast the last few months start to feel like the best thing that happened to your career.

More on the Topic

• AI for Data Engineers

• What Charles Darwin Teaches You About Being A Kick-Ass DataOps Professional

• Data Engineers Are Becoming MetadataOps Engineers (And You Don’t Even Know It Yet)

This newsletter is funded by paid subscriptions from readers like yourself.

If you aren’t already, consider becoming a paid subscriber to receive the full experience!

See what people say about working with me.

You are more than welcome to find whatever interests you here and try it out in your particular case. Let me know how it went!

Upgrade now

I’ve run hundreds of interviews over 16 years. When I ask “do you have any questions?“ and someone says no, I already know how this ends.

It’s not about the answers they gave. It’s about what that moment reveals: they came to survive the interview, not to think about the role.

The market is brutal. Hundreds of applicants per position. Everyone has the technical skills. The only thing left to separate yourself is how you think, and nothing shows that faster than the questions you ask.

At the senior level, the questions you ask are the interview.

The Market Is Brutal and That’s Exactly Why You Can’t Afford to Be Passive

The market is brutal right now. When I post about a new role, I get hundreds of applicants in just a few hours. Everyone has the technical skills, everyone has a decent CV, and everyone has prepared roughly the same answers to roughly the same questions.

So when everything else is equal, what decides it?

How you think. And nothing exposes that faster than what you ask.

A candidate who asks sharp, specific questions about the role signals something a polished answer never can: That you’ve already thought about the problems, the team, the constraints. You are not just trying to get hiredm but trying to figure out if this is worth their time.

That posture changes everything about how an interviewer perceives you. You stop being evaluated and start being considered.

Why Asking Questions Is Not Just Due Diligence

Most people treat the “do you have any questions?“ moment as a formality. You ask something polite, you show you’re interested, you wrap up. That’s the wrong way to think about it.

When you ask a sharp question, you stop being evaluated and start being considered. The interviewer shifts from “can this person do the job“ to “is this the kind of person we want making decisions here“. That is a completely different conversation.

The question you ask also tells the interviewer what you care about. Ask about tech stack and you signal you’re still thinking like a junior.

Ask about how data requests get prioritized and you signal you understand that the hardest part of a senior data role has nothing to do with the code. Ask about what happened to the last person in the role and you signal you’ve been around long enough to know that’s where the real story lives.

You ask questions to demonstrate you’ve already thought about the problems this role will throw at you before you’ve taken the job.

Here’s what to do:

Read more

Finance Year 2027 at my company starts in May. I always know company’s plans upfront. I thought FY26 was the same. By Q2 everything had shifted and what we called a roadmap was just a wishlist.

My team complained and I decided to change my tactic.

This year I grabbed my CPTO. Before any announcements, before any all-hands, before the business had even finalized anything. I asked what was coming, so I can take my team’s wishlist and map it against the company roadmap. That way we can build a plan that was realistic and OKRs we can actually hit.

The difference between a team that spends the year firefighting and a team that spends it delivering is just one conversation.

Every surprise project, urgent pivot, and “nobody told us” moment are your damn fault. That’s a business alignment problem, and it starts with you. Here’s what I mean.

The Drama Is the Point

There is a version of you that secretly enjoys the chaos. Think about it: The urgent Slack at 11pm, the all-hands pivot that wrecks your sprint, and he stakeholder who drops a critical request with a two-day deadline. You complain about all of it, and then you solve it anyway. And for a moment, you feel indispensable.

Reactivity performs well in the short term.

You look responsive, like someone who gets things done under pressure. Your manager sees you handling fires and assumes you’re good at your job. What they don’t see is that you lit half those fires yourself by skipping the data team planning conversations that would have changed everything.

The senior engineer who thrives in chaos is someone who never learned that prevention is harder to see but worth ten times more. Heroes get celebrated, so you keep choosing the version of the job that gets you noticed, even if it’s burning your team out.

Here’s what that costs you. Every urgent pivot your team absorbs without warning is two weeks of actual work gone. Retooling, re-scoping, re-explaining priorities to people who had different priorities yesterday.

Nobody tracks that cost. It just disappears into your team’s capacity and you fall behind on everything else.

Nobody Tells Anyone Anything

Here is the excuse I hear most:

The business doesn’t communicate with us. We find out about things last minute. We’re not in the meeting when decisions are made.

I’ve said versions of this myself. It’s true, but it’s also irrelevant.

PMs find out about things late. Finance gets surprised by headcount freezes. Sales hears about product changes from customers before internal announcements. Nobody in a fast-moving company has perfect information.

The difference is some functions treat that as a reason to push harder for clarity, and data teams treat it as a reason to complain.

Nobody owes you a seat at the planning table. Stakeholder alignment is not a gift. You earn it by showing up before the table is set. That means walking over to the PMs in April and asking what Q3 looks like.

It means grabbing your CPO before the roadmap is finalized and asking what bets the business is making. It means having those conversations when nothing is urgent, so you’re not having them when everything is.

The business will always move faster than it communicates. That is how companies work. Your job is to close that gap yourself, and not wait for someone to close it for you.

Every team that gets blindsided had months of opportunity to ask. But they didn’t, and when the surprise landed, they opened Slack and started typing about how chaotic everything is.

That’s your fucking choice. Own it!

Alignment Is Just a Conversation

I think data people overcomplicate this. They hear “alignment” and picture a formal process:

• A recurring meeting with an agenda.

• A shared OKR document that everyone updates religiously.

• A stakeholder map with color-coded relationship scores.

None of that is what I’m talking about.

Alignment is you walking over to your PM and asking “what’s keeping you up at night for the next quarter?”. That’s it. That’s the whole thing.

All you need is ten minutes and the willingness to ask a direct question before the pressure is on. Most data people never do it because it feels uncomfortable to show up without a specific ask.

But you’re not there to solve anything yet. You’re there to listen. That feels unproductive until the moment it saves your team three weeks of rework.

Do it early enough and you find out about the product release that was going to wreck your pipeline before it wrecks your pipeline. You find out the business is doubling down on a market segment your data model doesn’t support yet.

None of that information was secret and nobody was hiding it from you. But you never bothered to ask.

One business alignment conversation in March is worth more than any amount of sprint planning in September. By September the decisions are made, the timelines are set, and your team is already behind. In March you still have room to shape what’s coming.

Ask.

Final Thoughts

There is a ceiling reactive data professionals hit and never understand. They execute well, handle pressure, and solve hard problems fast. But they stay exactly where they are for years.

Because nobody above them sees a leader. They see someone who is very good at cleaning up messes.

The career cost is not the missed deadlines or burnt out team. It’s that you built a reputation for surviving chaos instead of preventing it. And that reputation follows you.

The people who move up are the ones who made their team predictable. The one who walked into planning conversations early. Those who stopped being surprised because they stopped waiting.

Every year you spend firefighting is a year you didn’t spend building that. Reputation compounds. The person known for calm, prepared delivery gets pulled into bigger problems. The person known for handling chaos gets handed more chaos.

You get to decide which one you are. But you have to decide in March, not in September when everything is already on fire.

Until next time,

Yordan

PS: Want to earn trust, respect and bigger budget for you and your data team? I have all the right playbooks in Premium Content Library.

Subscribe now

More on the Topic

• Your Stakeholder is Wrong (and it’s Your Fault)

• Crush Scope Creep: Data Engineer’s Blueprint for Bulletproof Data Product Plans

• The Signs Your Data Team Is Becoming a Cost Center

The further you go in your career, the more you run into problems that don’t live inside a single domain. A pipeline that nobody uses because the data model underneath it was wrong. A dashboard that answers the wrong question because nobody talked to the business before building it. A data strategy that looks good on paper and falls apart when it hits the actual architecture.

I sat down with , data strategist, consultant, and author of newsletter, to talk about what it actually means to think across the system. Dylan came from business consulting before moving into data, which means he sees this from both sides.

The conversation covered how companies actually operate, what separates the people who get ahead from the ones who stay stuck, and one habit that changes how you work without requiring you to become an expert in everything.

Here is what I took away.

The Data Ecosystem Is Not Your Tech Stack

What most people call the data ecosystem is really a landscape, a snapshot of tools at a point in time. It changes occasionally, but it’s static.

The ecosystem is something different. Think of a pond. Everything inside it is connected: the soil, the water, the microorganisms, the plants, the animals. Remove one element and the whole system shifts.

Data works the same way. Your data model affects your engineering. Your engineering affects your analytics. Your governance affects your ML. Your business model shapes all of it. These aren’t separate domains you visit when a ticket arrives, but a system, and if you only know your corner of it, you’re operating blind.

The problem is that most people work reactively. They handle what’s in front of them. They talk to the governance team when there’s a compliance issue, not because they thought about how governance connects to what they’re building. They don’t think about the relationships between domains. They just navigate them when forced to.

Dylan’s point is that the people who think proactively about those connections, who anticipate how one decision ripples into another domain, are the ones who produce better work and get trusted with bigger problems.

That’s a systems skill. And most data training never touches it.

Most Companies Are Still a Mess

Dylan has worked with a lot of billion-dollar organisations. His read: mostly chaos.

Data strategy documents exist. They talk about vision and use cases. They don’t connect to the architecture underneath, the governance layer, or the engineering reality on the ground. Nobody in the building knows how all the pieces fit together, not even the CDO, who spends most of their time managing upward.

The result is piecemeal data. Companies ahead in one domain are five steps behind in another. They buy great BI tools on top of poor data models and wonder why the dashboards don’t deliver. They bolt AI onto existing business processes and get some ROI but miss the structural shift entirely.

What’s almost always missing is the link between business strategy and how data actually enables the organisation. And the people who see that link, who say “we need to think about this before we build that,” are the ones who get trusted, get promoted, and get asked into the room where decisions happen.

Dylan’s observation from client work: the best performers in every organisation he’s walked into are the ones thinking across domains. LinkedIn will have you believe most companies are running cutting-edge data stacks. The reality for 95% of them is legacy chaos, and that gap is an opportunity for anyone willing to think holistically about it.

Data People Need Business Literacy

We obsess over data literacy. Get the business stakeholders to understand data, read the dashboards, ask better questions. Dylan flips it.

Data people need business literacy just as much, and most of them don’t have it. They don’t know how the business model works, what the real KPIs are, why the marketing team cares about a number that looks meaningless from an engineering seat. They build things nobody uses because they never asked what outcome was actually needed.

The gap is generational too. Five or ten years ago, most people who moved into data migrated from somewhere else, finance, operations, product. They understood business from the inside.

Now whole cohorts start their careers as data people. They’ve never sat on the other side of the table.

Dylan’s framing is direct: if you want to influence the business, you have to speak its language. That starts with understanding the business model, the KPIs, what each stakeholder is actually measured on.

Once you have that, you stop presenting data and start presenting decisions. Stakeholders notice. They start saying you’re different from the engineers who only think about the technical problem. That’s when trust builds and trust is what gets you the work that matters.

The “So What” Frame

Dylan had a consulting partner early in his career who rejected every slide that didn’t answer one question: so what?

Not “is this technically correct“, or “did you finish it“ So what. What changes because of this? What’s the action out of it? Dylan hated it at the time. It’s second nature to him now.

It’s the fastest way to pull yourself out of ticket mode and into strategic thinking. You don’t need to study the whole data ecosystem to start thinking more holistically. You need one habit applied every time you deliver something:

• Does this matter?

• To whom?

• What do they do next?

If you can’t answer that, you’re producing output. The business is paying for impact.

That question also protects you from the trap Dylan sees in most data teams, the reactive, firefighting, ticket-based mode that makes it almost impossible to step back. When every day is about closing tickets, you never ask whether the tickets matter.

The “so what“ frame forces that pause. Fifteen minutes of strategic thinking before you build something is worth more than three days of building the wrong thing fast.

What AI Changes and What It Doesn’t

AI is taking over individual technical tasks faster than most people are comfortable admitting. Routine work, boilerplate code and first-draft outputs are largely handled.

What’s left is context. AI doesn’t know your organisation. It doesn’t know why a specific stakeholder cares about a specific number, or whether a proposed solution fits your actual business model, or what you’ve already tried and why it failed. That’s your job.

Domain expertise plus the ability to connect domains, to say “this decision over here affects that system over there“, is what makes AI output useful rather than generic. The people who will matter in the next five years are the ones who can think across the system.

Dylan is also working on the next layer of the problem. How business models fundamentally change in an AI-first world. Most companies are bolting AI onto existing processes and getting some ROI.

The shift is rethinking how the organisation operates at the root. It has implications for data modelling, governance, architecture, strategy.

Nobody has solved that yet. It’s the open problem, and it’s where the real opportunity sits for anyone thinking holistically right now.

Where to Find Dylan

Dylan writes the Data Ecosystem newsletter on Substack. Weekly deep dives on every domain in the data space and how they connect. If you want to build the holistic view we talked about, start there.

He also posts daily on LinkedIn. If you want to connect, include a note. He reads them and responds.

And this summer he’s releasing a LinkedIn Learning course on insights-led thinking in the age of AI. Worth watching for.

Final Thoughts

The data industry has spent years rewarding depth. Know your tool, know your domain, go deep. That got a lot of people to senior. It stops working somewhere around there.

The problems that matter at the senior level and beyond don’t live inside a single domain. They live in the connections between domains.

Between the data model and the engineering. Between the engineering and the business outcome. Between what the team is building and what the organisation actually needs.

The people who see those connections are the ones who get trusted with the hard problems.

Business literacy, holistic thinking, asking “so what“ before you ship anything. None of this is complicated. It’s just not what most data training covers, so most people never build it deliberately.

You can wait until your career stalls to figure that out. Or you can start now, while the gap between you and everyone else who only knows their lane is still wide enough to matter.

Until next time,

Yordan

PS: Many paid members tell me they got promoted after working through the resources inside. If that sounds relevant to where you are right now, here’s what you get with a paid membership.

I spent years running meetings where everyone left happy and nothing got decided. I would invite the full project team, present the options, and then spend the next hour trying to find a solution that offended nobody.

The scope would balloon, the edge cases would multiply, and I would walk away calling it collaboration when it was, in fact, avoidance.

The moment I stopped doing that, something shifted. People started treating me differently. Not because I became more aggressive or stopped listening, but because I started owning the call.

I made the decision, explained my reasoning, and moved on. Turns out, that is what leadership looks like from the outside.

Most leaders I talk to know this problem exists. They sit in their own consensus meetings, watching the scope grow and the timeline slip, and they tell themselves they are being inclusive.

They are not. They are distributing the blame before the project even starts, and everyone in that call can see it.

This is how you fix it.

The Group Is Too Big and You Know It

I have seen this play out enough times to recognize the pattern immediately. Someone schedules a decision meeting and the invite goes to everyone who has ever touched the project.

Eight people on a call. Ten people on a call. And within five minutes, everyone is talking, everyone has an opinion, and the person who called the meeting is furiously trying to synthesize it all into something that makes everyone happy.

That is not a decision meeting, but negotiation where the wrong person is doing the negotiating.

The problem starts with who you invite.

When you pull in everyone connected to a project, you are not gathering input, you are gathering stakeholders. And stakeholders do not make decisions, they protect their interests. So every person you add brings their own priorities, their own edge cases, their own version of what the project should be.

You think you are being thorough. What you are actually doing is outsourcing your judgment to a group of people who were never supposed to own the call.

I watched this sink a multimillion project in 2025. Not because the team was incompetent, not because the technology was wrong, but because every decision went back to the group, every edge case got discussed, and every opinion got baked in.

The scope grew meeting by meeting until the project collapsed under its own weight. Nobody made the call, but everybody contributed to the failure.

Read more

Product drops a ticket asking for a churn model. Marketing wants a pipeline for campaign attribution. Someone from finance needs a new metric in the warehouse. And the engineer opens their editor and starts building.

No questions asked. No pushback. Just execution.

I get it. You want to be helpful. You want to look productive. You don’t want to be the person who complicates things with twenty questions when someone just wants a model.

But here’s the problem: the request is almost never the real problem.

Stakeholders Are Guessing

When someone asks you for a churn model, they are not a data scientist who has diagnosed the problem and prescribed the solution. They are a business person who has a problem they cannot solve and has translated it into something they think you can build.

That translation is almost always wrong.

A product manager who wants a churn model is usually trying to answer a much simpler question: who should the retention team call next week? Marketing asking for attribution is not an analytics project. They want to know which channels to cut and which to double down on before the next budget cycle.

The model is not the answer. It is their guess at what the answer might look like.

When you accept that guess without questioning it, you start building the wrong thing. You ship it, nobody uses it quite right, and three months later there is another request that is a slightly different wrong guess.

This is the customer service trap. You close tickets instead of solving problems. You get very efficient at producing outputs that do not change anything.

Why You Keep Doing It

You already know this happens. You have seen it. You have built the model that sat on a dashboard nobody opened.

So why do you still start building the moment the request lands?

Because slowing down feels uncomfortable. The second you ask “what decision does this actually support?” the conversation gets heavier. The stakeholder gets defensive. The request that seemed simple suddenly has no clear answer.

Subscribe now

Most engineers avoid that moment. It is easier to just start building.

And your organization rewards you for it. Closed tickets are visible. Pipelines deployed are visible. The conversation where you figured out that nobody actually needed a model, that conversation is invisible. Nobody puts it in a sprint review.

So you keep executing. And the ambiguity moves downstream, buried inside a system that runs perfectly and helps nobody.

The One Conversation That Changes Everything

The difference between an order taker and someone who actually moves the business forward almost always comes down to a single moment: the first conversation after the request arrives.

Order takers treat the request as instructions.

I am not saying you need to interrogate every stakeholder for an hour before writing a line of code. I am saying you need to ask two questions before you start: what decision does this output support, and what changes after you have it?

That is it. Two questions.

When you ask them, something interesting happens. The request starts to fall apart or change shape. The churn model becomes a prioritization spreadsheet. The attribution pipeline becomes a simple analysis of three channels. The activation metric becomes a conversation about what the product team is actually trying to decide.

Sometimes the real problem has nothing to do with data. Sometimes it is a broken process or a misaligned team or a goal that nobody has defined clearly. You find that out fast when you ask what changes after the output exists.

And once you know the actual problem, the work gets smaller. Faster. More useful.

What This Does for Your Career

Here is the part nobody talks about.

When you keep executing requests without questioning them, you train your stakeholders to treat you like a vending machine. They put in a ticket, they get an output. They never think to bring you in earlier, when the problem is still being defined, because they have learned that is not what you do.

You become a very skilled, very expensive order taker.

The people who get asked to sit in on strategy calls are not the ones who are fastest at closing tickets. They are the ones who have shown they can reframe a bad question into a useful one. They have made someone’s problem smaller and cleaner before any work started.

That is the reputation worth building.

Once you start doing this consistently, you will also notice the work gets better fast. And you will realize that the conversation itself needs structure, or the same ambiguity comes back with the next request.

Talk soon,

Yordan

PS: A lot of paid members tell me they got promoted after working through the resources inside. If that sounds relevant to where you are right now, grab your membership now.

Subscribe now

The tech world moves fast. New tools, new frameworks, new cloud features, and now AI on top of everything. Many data engineers worry that if they slow down, they will quickly fall behind or even become obsolete.

The common advice is simple: read more, follow more experts, subscribe to more newsletters. But that approach creates a different problem. Instead of helping you stay informed, it often floods you with noise.

The real challenge in tech today is not lack of information. It is learning how to filter it. That is where an information diet becomes useful.

The Information Problem In Tech

If you work in data engineering, you probably feel constant pressure to keep learning. The field changes quickly, and the amount of new information keeps growing. New tools appear, cloud platforms add features, new architectures become popular, and a large number of blogs, talks, and newsletters try to explain it all.

Trying to follow everything can easily become a job on its own.

High velocity knowledge production

One reason this feels overwhelming is the speed at which new knowledge is produced. The modern data ecosystem evolves very quickly. New orchestration tools, storage systems, query engines, and AI platforms appear regularly. Each release is followed by documentation, tutorials, benchmarks, and opinionated articles about why the tool matters.

If you try to keep up with even a small part of the ecosystem, the amount of material adds up quickly. You might start reading about a new framework, only to see another one appear a few weeks later. Over time, the stream of content grows much faster than you can realistically absorb.

Platform amplification

The way information spreads online makes the situation worse. Platforms like LinkedIn, X, and YouTube are built to reward engagement. Their algorithms promote content that attracts attention, which often means content that is new, surprising, or controversial.

Because of this, your feed is often filled with the newest ideas rather than the most useful ones. A recently released tool can suddenly appear everywhere, even before anyone has real experience using it in production. When this happens repeatedly, it becomes difficult to judge what actually deserves your attention.

Social pressure

There is also a social effect that shapes how you see the field. When you scroll through social media or attend conferences, you mostly see what other engineers choose to share. People post about the new tools they are exploring, the architectures they are building, and the research they are reading.

What you rarely see is what they decide not to follow. Because of that, it can easily feel like everyone else is keeping up with everything new in data and AI.

In reality, no one is doing that. The engineers who try often spend so much time consuming information that they struggle to extract real value from it.

The Paradox of Staying Current

When you feel the pressure to stay current, the natural reaction is simple: you try to follow more sources. You subscribe to more newsletters, follow more people on social media and bookmark more blogs and podcasts.

If you consume more information, you should become more informed, right?

Wrong!

When you increase the number of inputs, you reduce understanding. Every new source competes for your attention. The more information you consume, the harder it becomes to identify what actually matters.

Over time you start collecting fragments instead of knowledge. You read headlines, quick takes, and summaries of tools you may never use. You recognize names and trends, but you rarely spend enough time with an idea to truly understand it.

You can see this pattern everywhere in tech. One week your feed is full of a new database, the next week everyone talks about a different orchestration tool. A month later it is a new AI framework. Each trend replaces the previous one before anyone has had the time to learn it properly.

This creates the illusion of learning. You feel informed because you see many ideas pass through your feed, but most of them leave as quickly as they arrive.

At the same time, the cognitive cost grows. When your attention is constantly divided across many inputs, it becomes harder to think deeply. Your decisions get worse, and your focus weakens. Even though you spend more time consuming information, you extract less value from it.

That is the paradox. When you try to follow everything, you actually become less informed.

Staying current is not about increasing the amount of information you consume. It is about learning how to protect your attention.

The Information Diet

If attention is your most limited resource, then the real goal becomes clear. You need a way to convert attention into useful insight as efficiently as possible.

This is where the idea of an information diet becomes useful.

Most engineers try to increase the amount of information they consume. They subscribe to more newsletters, follow more experts, and add more blogs to their reading list. It feels productive, but it rarely improves understanding.

The real objective is different. You want to maximize insight per unit of attention.

You can think about it like this:

Read more

In this conversation, Erfan Hesami and I dismantle the myth that technical depth is the only path to the top. You spend years mastering Python and SQL only to reach a level where pipelines fail because requirements were wrong or stakeholders changed their minds.

This is the ceiling where most data careers stall. Erfan, drawing from his experience in fintech and manufacturing, confirms that the jump to senior is not about a tenth tool. It is about shifting focus from the immediate task to the broader system.

You must stop being a ticket-taker and start being an operator who designs for long-term reliability. Erfan highlights that the senior mindset requires moving beyond the IDE to master the non-code half of the job.

This involves making hard calls on “good enough“ solutions and building trust through translation. We discuss how to bridge the gap between technical skill and leadership so your career does not stall.

This session covers the specific mindset shifts and operational boundaries Erfan uses to cross the senior threshold.

Subscribe now

The Senior Operator Writes Less Code

Erfan notes that mid-level engineers often measure value by the complexity of the systems they build. You want the newest framework because it feels like progress. Senior engineers realize every line of code is a liability. You provide more value by preventing a project than by building it poorly.

When you operate at senior level, your primary job is making technical decisions aligned with business problems. You prioritize projects based on impact rather than technical interest.

If a simple SQL view solves the problem, you do not build a custom microservice. The consequence of over-engineering is a maintenance debt that eventually prevents you from shipping anything new.

Reframe your role from a builder to a decision-maker. You are there to ensure data flows reliably, not just to move bytes.

Erfan suggests that a good week is not one where you closed the most tickets, but where you simplified an architecture or spent time upskilling to prevent future technical debt.

Ownership Is Moving Without Permission

Waiting for a manager to tell you what to do is a junior behavior. Erfan observes that many competent engineers stay mid-level because they wait for a formal promotion before acting like a leader. They expect the title to grant them the right to speak up. In reality, the title follows the behavior.

You must take ownership of the end-to-end lifecycle before anyone asks. Erfan emphasizes raising your hand when you see a flaw in database design or a better way to handle upstream sources. If you wait for permission, you are a resource.

The tradeoff is that ownership comes with accountability. When a project fails, you cannot blame the requirements. Erfan argues that unclear requirements are an engineering problem. You bridge this gap by networking across departments like CRM or accounting to understand pain points before you ever open your IDE.

Zoom Out to See the System

Juniors focus on the feature in front of them to move a ticket from left to right. This narrow focus leads to local optimizations that break the global system. Erfan warns against building a fast pipeline that produces garbage data because you did not check the upstream quality.

As a senior, you must zoom out to see the broader system. Erfan explains that you must ask why a feature was requested and how it affects downstream consumers. You focus on the foundation that makes the platform scalable and easier for others to use over time. This shift in perspective allows you to catch errors in assumptions before they become code.

This systemic thinking is your true differentiator. Erfan points out that AI can write boilerplate code, but it cannot navigate the politics of conflicting requirements.

The Resources

Here are the links to all the resources we mentioned:

• Pipeline To Insights: Erfan’s Substack newsletter where he shares his learning experiences and data engineering insights.

• Erfan’s on LinkedIn: Here you can connect with Erfan and read his new

• The GitHub Portfolio Article: A specific article mentioned that was written for Pipeline 2 Insights regarding how to build a GitHub portfolio.

• Article on Certifications: A “very nice” article written by Jordan discussing the role and value of certifications in a data career.

Final Thoughts

The shift Erfan describes is a move from the certain world of syntax to the ambiguous world of people and systems. You must be willing to ask the questions that feel stupid but reveal deep flaws in a plan. You must be comfortable sharing opinions even when they are not explicitly requested.

True seniority is not a reward for time served. It is a status earned by those who take responsibility for outcomes, not just outputs. Erfan’s approach shows that you stop being a person who writes code and start being a person who solves business problems using data.

Focus on building trust through consistency and ownership. Become the expert who can navigate different areas and adapt to new challenges. When you master the non-code half of the job, your career will stop stalling.

Until next time,

Yordan

You are drowning in a backlog that only grows. Last quarter you missed the migration deadline because your lead engineer spent six weeks fixing a broken ingestion pipeline. Now the business wants a real-time attribution model by July, but your team is still manually patching SQL scripts.

You know you need more hands, but simply asking for a generic data engineer is a rookie move that leads to budget friction.

Last week, I showed you how I request headcount increases and consistently get a Yes from my CFO. But before doing that, you must know exactly what profile you are buying.

Hiring the wrong type of help creates more management overhead than it removes. You need a framework to decide if you are building a permanent foundation, performing a surgical extraction, or seeking a map for a new territory.

Today, I will show you how to distinguish between employees, contractors, and consultants so you can spend your budget on the right outcomes.

When Do You Hire Somebody Full Time

Full-time employees are more than resource. They are an investment in the long-term compounding of your data platform.

You choose an FTE (full-time employee) when the value of the work comes from its duration rather than its immediate completion. If the role requires negotiating with the Head of Sales every Tuesday or maintaining a core system for the next three years, you need an employee who owns the outcome.

Strategic Context and Ownership

FTEs specialize in the nuance of your specific business logic. While a contractor can build a perfect pipeline, an full-time employee understands why your CRM data has had edge cases since 2019. This deep institutional knowledge allows them to make decisions that prevent technical debt before it happens.

You hire an employee when you need someone to own a domain and live with the consequences of their architectural choices for years.

Cultural Stability and Mentorship

High-performing data teams are built on trust and consistent mentorship. You cannot effectively outsource the professional development of your junior analysts or the soul of your engineering culture.

Employees provide the social glue that keeps a team functioning through shifting priorities. When the roadmap changes or a project is killed, an FTE has the institutional skin in the game to pivot without looking for their next gig.

Long-term ROI and Availability

Hiring a full-time employee is the correct move when the roadmap shows a permanent need for a specific skill set for at least twelve months. You avoid the “switching costs“ of offboarding and onboarding every quarter.

While the upfront cost of hiring is higher, the daily cost of an employee is significantly lower for baseline work. You are buying consistent, predictable capacity for the core functions that keep the lights on and the data flowing.

When Do You Hire a Contractor

Contractors are surgical tools. You bring them in to solve a specific problem, fill a temporary gap, or provide a niche skill you do not need forty hours a week. Unlike an employee, you buy a result instead of a career.

Project-Based Work

If the work has a clear start and end date, a contractor is the logical choice. Building a ClickHouse cluster or setting up Kafka for real-time event tracking are intensive, one-time lifts. You need the expert who has done this ten times before to handle the heavy lifting of the initial setup.

Once the infrastructure is laid and the system is stable, your internal team can handle the day-to-day maintenance. You pay a premium for their speed and specialized experience.

Specialized Expertise

Say you need a world-class performance tuner for a three-month audit of your Snowflake spend. In that case, you do not need that level of seniority or that specific salary on your payroll forever.

Contractors allow you to access “over-qualified“ talent for short bursts. This keeps your internal team focused on the business logic while the specialist handles the complex plumbing that only needs to be touched once a year.

Speed to Hire and Testing the Waters

Onboarding a contractor is often faster than the months-long gauntlet of interviewing for a full-time role. If a critical project is stalled because you lack a specific pair of hands, a contractor can be on-site in two weeks.

This also serves as a low-risk trial. A “contract-to-hire“ arrangement lets you see if the role is actually necessary or if the person fits the team before you commit to the massive overhead of a salary and benefits package.

The Trade-off of Context

The risk with a contractor is the loss of context when they leave. You mitigate this by treating them as a resource for your FTEs, not a silo. Their job is to build the tool and train your team on how to use it. If a contractor builds a foundation and leaves without a knowledge transfer, you hired a liability.

When Do You Hire a Consultant

A consultant is not a pair of hands. You hire a consultant when you have a problem but are not quite sure how to solve it, or when you need a high-level strategy to reach the next level. They fill a knowledge gap, not a capacity gap.

Strategy and Direction

Consultants manage the process of discovery. While a contractor asks for a Jira ticket, a consultant asks why the ticket exists in the first place.

You hire them to analyze your situation and tell you what your goals should be. Whether it is diagnosing why your churn rate is high or designing a roadmap to transition your entire team to a new tech stack, you are paying for their perspective and their past experience with similar failures.

The “Objective Outsider” Effect

Internal teams often suffer from tunnel vision or political fatigue. A consultant provides a “culture neutral“ assessment of your architecture or team structure. They can say the things to your leadership that you cannot say without risking your social capital.

If you need to build a data team from scratch and need someone to design the hiring rubric and initial tech stack, a consultant provides the blueprint so you don’t have to guess.

Transformation and Handover

The goal of a consultant is to provide a solution or a transformation, then leave. They are the architects, not the construction crew. A successful engagement ends with your team feeling empowered to execute the plan they laid out. If a consultant becomes a permanent fixture in your weekly standups, you have accidentally hired an expensive, long-term contractor.

The “Golden Rule” Test

If you are still on the fence about which path to take, ask yourself one question:

If this person left in six months, would it be a minor inconvenience or a total disaster?

A minor inconvenience suggests you only need somebody interim. If you can document their work, hand it off, or hire another specialist to pick up the pieces without the project collapsing, you are looking for a tactical capacity play. Contractors are built for this. They come in, execute the “how“, and move on once the documentation is signed off.

A total disaster indicates you need somebody full-time. This signals that the role is critical to your daily operations and requires a “memory“ that shouldn’t walk out the door when a contract ends.

If the departure of a single person would halt your roadmap or leave the team unable to explain how the core engine works, you have a structural gap. You need a permanent owner who is incentivized to ensure the system survives their eventual departure.

A Few Myths To Bust

Choosing a hiring model based on gut feeling leads to bloated budgets or burnt-out teams. You have to look past the surface-level costs to see the actual impact on your operations.

1. Interim employees are way too expensive

The daily rate of a contractor is higher, but the total cost of ownership is often lower. When you hire a full-time employee, the salary is only the baseline. You also pay for benefits, payroll taxes, equity, and the 4-6 months it takes for them to become fully productive.

An interim professional is a plug-and-play solution. You pay for the days they work, and when the project ends, the cost drops to zero instantly.

2. Contractors aren’t as loyal as permanent staff

Their loyalty is to the result, not the hierarchy. An employee might stay because they like the health insurance or are waiting for a vest date. A contractor’s survival depends on their reputation and immediate delivery.

They are often more focused on hitting your KPIs because they don’t have the luxury of “coasting“ through a performance review cycle.

3. Bringing in an outsider will disrupt our culture

Interims usually provide a culture-neutral perspective that internal teams desperately need. Because they aren’t trying to climb the ladder or win a popularity contest, they can be more honest about technical debt or broken processes. Instead of disrupting your culture, they act as a stabilizing force that allows your FTEs to focus on high-value, long-term work.

4. If I hire a contractor, I don’t have to manage them

Outsourcing work does not mean outsourcing leadership. A common mistake is assuming a high hourly rate replaces the need for context and alignment.

If you don’t provide a clear definition of success and regular feedback, even the best contractor will build the wrong thing. You are still the architect, and they are just the specialist helping you execute the plan.

Final Thoughts

Hiring is a strategic decision, even when it feels like a tactical fix for a capacity gap. Who you bring onto the team, and under what terms, signals your true priorities.

If you hire an FTE for a temporary migration, you are over-indexing on stability for a one-time event. If you hire a contractor for a core system, you are risking your long-term operational memory for a short-term budget save.

You cannot spray and pray with your headcount. You must define success criteria before the first interview or the first SOW (scope of work document) is signed. Know what you want these people to achieve and how you will measure if they are doing it.

Without a plan, you aren’t scaling your team; you are just increasing your management debt. Every hire should solve a specific friction point in your delivery.

Thanks for reading,

Yordan

The problem is that AI agents are the most demanding stakeholders you have ever had. They have zero intuition. They do not understand your tribal knowledge or your hidden business logic. If you give them a table without a deep layer of metadata, they will guess. When they guess, they hallucinate. When they hallucinate on your watch, you lose your seat at the table.

This conversation with Ale highlighted that the job has changed. You are no longer just shipping rows, but shipping the instructions that keep the machines from lying.

The Death Of The Static Catalog

Traditional data catalogs are where documentation goes to die. You spend a month tagging columns and then nobody looks at them. Ale calls the new approach MetadataOps.

In this model, your agents provide data back to the system. They help you understand how they use the information. If an agent struggles to identify a primary key, that is a production bug.

Documentation is now the fuel for your production AI. If you treat metadata as an afterthought, your AI solutions will stay in the playground. Real operators build systems that document themselves through use.

Managing The Unstructured Mess

Unstructured, human generated data like images, audio or markdown documentation is a real mess. You must build guardrails that expect the mess. This means using regular expressions and specific skills to pull structure out of chaos. You are acting as a bridge between human habits and machine requirements.

This requires deciding when a data source is “good enough” to ship. You have to weigh the risk of a hallucination against the speed of the business. These are leadership calls, not technical ones.

The AI Stakeholder Shift

Your stakeholders think everything is possible now because they saw a demo on Twitter. This creates a massive gap in expectations. People expect you to ship features ten times faster. They do not see the infrastructure required to make those features reliable.

You must translate the technical risk of AI into business terms. If you fail to manage the context, you fail to manage the trust.

You are a translator now. You explain why a simple prompt is not a production strategy. You show them that the quality of the output depends on the quality of the metadata you have been screaming about for years.

Links

We mentioned a few resources. Here are the links:

1. Alejandro’s publication: Make sure you subscribe.

2. Substack Recapped: A must try if you write on Substack.

Final Thoughts

The gap between a senior individual contributor and a data leader is the ability to manage ambiguity. Ale’s shift from dbt to agent protocols is a map for your own career.

You cannot hide in the IDE and hope the data stays clean. You have to lean into the mess of unstructured files and vague stakeholder requests. Metadata is the only tool that bridges that gap.

You need to start thinking about it as managing the context of your entire company. If you do not own the metadata, you do not own the outcome. Seniority is about taking ownership of the results, not just the scripts. Use your metadata to build a system that thinks.

Until next time,

Yordan from Data Gibberish

PS: We live stream these conversations. Join and chat with us next time.

Your backlog can cover twelve months of non stop work and your team is drowning in ad-hoc requests. You tell your manager you are underwater and need help. They tell you to prioritize better or wait until next quarter. You stay stuck in a cycle of burnout while technical debt anchors your throughput to the floor.

You need to work on your pitch. You are likely framing headcount as a solution to your own stress, but your CFO does not fund stress reduction. They fund risk mitigation and growth capacity. If you want a new engineer, you must prove that your current team structure creates a specific, unacceptable bottleneck for the company’s bottom line.

You need to shift from counting tickets to mapping risks and aligning with the business roadmap. This manual shows you how to audit your team and write a proposal that treats headcount as a mandatory dependency for the company’s own goals.

The Volume Trap

You likely think the sheer volume of Jira tickets is your strongest argument for a new hire. You present a mountain of “In Progress“ tasks and a stagnant “Backlog“ as evidence of a resource gap, expecting your manager to see the pile and offer help.

This approach backfires because it signals a lack of leadership rather than a lack of engineers. When you lead with busyness, you invite a lecture on time management instead of a budget approval.

Your stakeholders do not see a capacity problem when you show them a crowded board. They assume you lack the backbone to say No to low-value requests or the technical skill to automate the noise.

By using stress as your primary metric, you position yourself as a victim of your own inability to manage your domain. This framing makes you look like an operator who cannot prioritize, which is the exact opposite of the leader they want to give more resources to.

This obsession with “doing more“2 often leads to the mistake of treating technical debt as a standalone crisis. Your CTO might sympathize with your spaghetti code, but your CFO sees debt as a baseline cost of doing business. They expect you to manage that debt within your existing headcount as part of your daily chores.

Unless you can prove that this debt is the specific physical barrier preventing a high-priority launch, your complaints about “refactoring“ will be dismissed as engineering perfectionism.

So, you must stop asking for relief and start documenting the specific business opportunities that are dying on the vine.

When you ask for a hire because you are tired, you are asking for a favor that can be denied. When you shift the focus to how your current architecture physically cannot support the Q3 revenue goals, you move from complaining to providing a solution for the company’s growth.

The Strategic Headcount Framework

You cannot solve a structural capacity problem with a tactical conversation about your workload. To secure a new hire in this climate, you need a methodology that moves the debate from your team’s burnout to the company’s risk profile.

This is all about providing the business with a clear choice between investment and stagnation.

The framework I use to bridge this gap is built on three pillars: structural audit, strategic alignment, and the risk-based business case.

You start by mapping the hidden vulnerabilities in your current org chart. This moves the narrative from “we are busy” to “we are fragile,” which is a much more potent lever for a CFO.

Once the internal risks are exposed, you align your team’s output with the next twelve months of the company’s roadmap. Because if your request for a new engineer is not tied to a specific, high-priority launch, it will be categorized as a luxury.

Read along to learn more about my framework. But before that, get the full manual with all the scripts and templates that you can use to expand your team.

Read more