This article is part of the Show & Tell playlist. Click here to explore the full series.
If you’ve started owning your own deadlines, the date you give is the day you’d finish if nothing got in the way. The project matters to the people asking, you want them happy with you, and you genuinely want to make it.
I gave dates like that for years, and somebody had to say it to my face before I did anything about it.
My boss doubled every estimate I gave him
10, maybe 12 years ago, I was a software engineer at a video streaming company, sitting right next to Deyan. He asked me how long a new feature would take, a live chat, and I told him it was easy and I’d be done in two weeks.
He didn’t even blink. “So, one month”.
Then he told me, “You haven’t hit any of your estimates in a while”. He’d learned that doubling whatever I said was right most of the time.
It didn’t feel good, and it wasn’t a massive surprise either, because I’d heard I wasn’t great with estimates a few times before, and I’d grown up with it. Whenever I asked my dad when something would be done, he gave me a date, it was always his best case, and we never hit it.
So I knew I had a problem, and now somebody at work had told me I suck at estimating. It took me a few years of experimenting and reading to fix it, and I even tried Six Sigma, which I knew wasn’t for me.
Part of my problem was the interruptions eating my week, and part was that my two weeks was the best case, the time the feature would take if every hour of those two weeks went on it.
Why your best case is the date you give
You give the best case because you’re trying your best. You probably don’t say the assumptions out loud, but the date you give is the one you’d hit working uninterrupted with everything going right, and you give it because you want to deliver by then.
That date assumes a perfect world, where every task is as clear as it looks, nobody needs your help or a second pair of eyes, and nothing breaks.
None of that happens, because things break all the time, and the more experienced you get, the more calls you get pulled into.
A missed date costs you trust
When you miss your own dates, the cost is trust. Once your boss says you haven’t hit any of your estimates in a while, they’ve stopped trusting you, and it doesn’t stay with the estimates. The ambitious projects, the promotion and the raise you’re chasing start slipping with it.
After enough optimistic dates my manager stopped believing me, and the math was on his side.
There’s a weird thing I keep seeing, too, where you miss a day, the next project starts a day late and ends up a week late, and then everything slides like an avalanche. I don’t know why it happens, it’s just what I’ve seen.
And it gets worse with size, because on a small project you miss by a day or a few hours, and I’ve had a month’s estimate take three months, a whole quarter.
Never estimate on perfect conditions
You must never estimate on perfect conditions, because they don’t exist, least of all once you’re chasing a senior or lead title. The question I ask is how long it’ll take given the reality of my week.
Your week has three things in it that the best case ignores, the meetings, the downtime and the unknowns.
Nobody gets 8 hours of work out of an 8-hour day. There’s the standup, the planning, the Slack thread you got pulled into, the call that was meant to be quick, and lunch, which I forgot myself when I listed these on the stream. Open your calendar and count the hours of real work you got last week, because your contract hours won’t tell you.
On my first day at my second job, my boss told me, “In an 8-hour day, if you manage to pull 6 hours of productive work, that’s amazing. That’s the best I can dream of”. These days I try to stay hands-on, and if I get 2 to 3 hours of focused work a day, that’s a great day. For most people it’s 4 to 6.
Then there are the days you won’t touch the work at all. Everybody on my team has one rota day a week, when they take the incidents and the questions so everybody else can focus on their tasks. Add holidays and planning days on top. When my team goes to our London office for 3 days of planning and team activities, those are 3 days out.
And then the unknowns, which is where most of the damage is. Access you don’t have yet, a tool nobody on the team has used, a decision somebody else has to make, and another team you’re waiting on, where you’re not their priority.
None of those show up in a best case, so each one needs a number.
How to estimate data engineering projects
Cut the work into tasks of 2 days or less and size each one. Multiply each task’s hours by an ambiguity factor, add 10%, divide by your real focus hours a day, round up and add the days you won’t work. Without the factors and the 10% you get your earliest date, and with them, the date you’d bet on.
It’s the method I’ve used for years, plus what I’ve added since. The first version ended on one single number, which is a funny way to run a method whose whole point is the gap between two numbers.
Big projects get estimated one stage at a time
The formula only works on something you can see the end of. So a big project gets split into stages, you estimate the next one, and the stages after it get their dates when they’re next.
I do it for four reasons.
If something goes wrong or priorities switch, you miss one stage.
Your stakeholders get something they can see, touch and play with, more often.
In six months you might have much better ideas, and the project might look different by then.
Nobody can plan tiny tasks across a three-month project, and with 100 of them on the list you lose sight of what the project is for.
Send both dates and promise the second
You end up with two dates, the earliest you could be done and the date you’d bet on. I always send both and say it could be done between this date and that one.
Your stakeholders will still hear the first one and hope for it, and I haven’t found a way to stop that, so I promise the second one anyway.
An estimate is a promise. Only promise the date you’d bet on.
My personal philosophy is that I want to be trustworthy, so I do what I said I’d do, and an estimate is something I said I’d do.
It goes beyond you, because if you promise the marketing team their data by a date, they plan a campaign around it, and when you miss, you’ve blocked their work as well.
I under-promise and over-deliver, so if I’m done three days before the date I promised, people are happy. If you give them the optimistic date instead, those three days of stress are yours, and you put them there yourself.
