This article is part of the Data Project Management playlist. Click here to explore the full series.
You know which model or DAG costs your team hours every week. The ticket that says so has been in the backlog for months, and you’ve started fixing it in pieces, on a Friday afternoon or inside the estimate for the next pipeline that touches it, without telling anyone.
I did the same. I cleaned up months of tech debt and got a “thank you” from the people close to the work, but outside the team I got only silence, and when reviews came up, I struggled to fill the doc. (The whole story is in No one gets hired for clean pipelines)
The story of that work never made it out of the team, and nobody outside it ever saw what the debt had been costing them. Your ticket has the same problem before you’ve written a line of the fix. It says how the pipeline makes you feel and nothing about what it costs anyone, which makes it a complaint, and complaints don’t get sprint time.
Your debt ticket says the pipeline is fragile and stops there
Open your last debt ticket and count the words like “fragile“, “flaky“ and “nobody wants to touch it“. Then count the words that say what it costs somebody, and who that somebody is.
Whoever owns the sprint ranks your ticket against new sources and models that somebody asked for and is waiting on, so each of those comes with a requester and a cost of waiting. Your ticket has neither, so it loses every ranking on arithmetic, and nobody has to be against you for that to happen.
You’ve heard that “tech debt” means nothing to the person deciding. I said it in 4 awkward stakeholder conversations you have to master, and half the internet says it with me. Swapping the word for a nicer one leaves the ticket asking for the same thing.
Chelsea Troy describes that thing from the other side:
It sounds to the business like the engineers are asking for three weeks free from the obligation to release any features
Now compare it with how Rex Morgan words a request:
We currently spend 8 hours a week on these manual tasks we neglected to automate.
His sentence tells the person deciding what they pay every week to leave things as they are, which is the one thing your ticket leaves out.
A repayment date keeps moving until the debt has a price
A lead system test engineer posted a rule on r/ExperiencedDevs after 7 years at one org. The rule is that anything logged as debt “needs a Repayment Date“.
It’s half the fix, and somebody in the same thread found the hole:
What happens when they don’t meet it? Give you another?
For a long time I ran debt tickets with cost and date as two equal fields. It didn’t work.
My read on why is that a date you set yourself costs nobody a damn thing to move, so it moves. A schema change upstream eats the sprint, the date slides, and the only thing spent is your credibility, a bit more of it with every date you give after that one.
A cost that comes back every month keeps running whatever happens to the date. Put it on the ticket as hours a month, plus compute a month when it’s on the bill, and every time the date moves the ticket records who moved it and adds another month to a running total. Moving the date stops being free, because somebody has to agree, out loud, to pay for another month.
The price changes what “not now“ means, too. Today it’s what happens by default to anything without a price, with nobody’s name on it, and with a monthly price on the ticket it becomes a choice the sprint owner makes and owns.
Price the debt in what it costs every month, and let the date follow from the price.
Subscribe if you want your team out of constant urgent fixes and your work seen by the people who fund it, and I’ll send you the two pieces a week I write on how to do both.
Pipeline debt charges you and everyone downstream every time it runs
Martin Fowler’s definition of the interest comes from software, where you pay it when you change the code:
The extra effort that it takes to add new features is the interest paid on the debt.
A pipeline charges you every time it runs, whether anyone touches the code or not.
The interest is the re-run when a job times out, the backfill when data goes missing, the bad load that reaches whoever reads the table first because nobody wrote the test, and the warehouse that runs bigger than it needs to because the model under it was never tuned.
Your team pays most of it as on-call or “keeping the lights on”, it disappears into the week, and that’s how a pipeline that sucks survives for years with nothing on any ticket to say what it costs.
Reema Kuvadia has the reason nobody counts it:
The debt rarely gets addressed, because it never appears as a single, dramatic incident anyone gets paged for.
The people who use the data pay as well, and they never see your ticket.
I consulted for a company where the data team ran ad-hoc queries for the marketing team, all the time. Our pipelines were timing out and we were occasionally missing data.
The queries were different every month, and nobody trusted the data, so the marketing team kept their own spreadsheets, and keeping them took 4 to 6 hours at the end of every month.
Marketing paid interest on a debt that was ours, and it didn’t get fixed while I was there.
Whenever the person paying the interest and the person who owns the pipeline are different people, nobody prices it, because you don’t feel the cost and they don’t know a ticket exists.
So count their hours as well as yours, and bring them with you when you ask. Whoever pays the interest is the one who wants the date, and the sprint owner hears it faster from them than from you.
Count people’s time in hours, because the salaries get paid anyway
I almost never put a dollar figure on people’s time. The people whose hours the debt eats are on a salary, and the salary gets paid whether the debt is fixed or not, so a dollar figure on their time promises a saving no bill will ever show. What the debt takes is hours they could spend on something else.
Count the hours from the last few weeks, and you’ll find most of them are already written down somewhere.
Give them as a range you had before anyone asked, the same rule as A confident guess is a lie, even when it’s right. It also fits Stop talking about tables, where cost means “Can we save money or time?“. Time is the half of that you can defend.
Revenue figures on debt tickets are mostly bullshit, and the first person who checks one stops trusting your next number as well.
Risk goes on the ticket only as an incident that already happened, named and dated.
Warehouse debt is the one you can price in money
Money goes on the ticket in one case, when the interest is on an invoice. Warehouse spend that exists only because the debt does is read off the bill, and nobody has to take your word for it.
I’ve carried that kind of debt on purpose. We ran bigger warehouses, our modelling wasn’t great, and we had no proper clustering, because we wanted to move fast. We knew the Snowflake cost would be high, so we budgeted for the increase in favour of releasing the projects, and once we’d delivered them, we set time aside to close the gap.
All three show up on the warehouse bill every month they stay, so the increase was a number we could budget for before we shipped. A debt like that can’t turn into a complaint, because somebody decided to pay for it with the cost already known.
Most of yours is the other kind, taken on by somebody, at some point, with no number written down anywhere, and the ticket is where you put the number back. Start with the warehouse bill, because it’s the one number on your ticket nobody can call a guess.
The fix, backfill and migration included, goes on the ticket as its own field. It’s a one-off, so it doesn’t grow while the ticket waits, and it can’t keep a date honest.
For your own case, the ready-to-use version is my digital twin, and it comes with the annual subscription.
Paying it down in secret costs you twice
When the ticket doesn’t move, you pay for the fix yourself, on a Friday or hidden in a feature estimate. It’s a crap deal, and you’re the one offering it. You pay once with your own hours and again at review, because nobody saw the price fall, so as far as anyone deciding your raise knows, the repayment never happened.
This is how I put that same silence in No one gets hired for clean pipelines:
You’re not failing because your work isn’t good enough. You’re failing because it’s silent.
That piece fixes the silence after the work is done, by asking what changed, and a priced ticket gets there before you start, because it has a before number and an after number with your name next to both.
Ron Jeffries would tell you not to ask at all, and to clean what you touch inside the feature, because
Often, that time is not granted: we’re asking for time to fix what we screwed up in the past.
He’s right about anything that fits inside work you’re already doing, like tidying a model you’re changing anyway. The ticket is for the debt that doesn’t fit, like moving a model to incremental loads, which needs a backfill and a stretch of running old and new side by side.
Priced and funded, the repayment happens in work hours, on the sprint board, with your name on it, and your Fridays go back to being yours.
Final thoughts
I’m not sure every team needs a price on every item. Alex Ewerlöf’s team won a protected debt day with no price on any item. If you already have protected debt time and nobody gatekeeps it, pricing every ticket is ceremony, and the better question is the one Ewerlöf asks: “you need to ask why the gatekeeper exists at all”.
Most of you have a gatekeeper, and the price is for them.
Some debt costs nothing to carry. A messy model that runs cheaply, breaks nobody’s week and never needs changing has a monthly price near zero, and a price near zero doesn’t need a date at all.
The case I can’t close is the one where the price is on the ticket, the person paying it is in the conversation, and the answer is still “not now”. At that point somebody has chosen to keep paying every month, knowing what it costs. Maybe that’s the right call. I still don’t know what you do next, except keep adding to the total.
Until next time,
Yordan
PS: If you want my answer to your own version of this before you have to give yours, grab the yearly subscription and ask my digital twin, which answers the way I would.