👷 Why data projects die when you buy the stack everyone uses

and the six questions to think through before you sign a contract

I just came back from a trip to London, where I did some workshops, made some announcements and, more importantly, said bye to Plamena, my manager and our CFO.

So, her new place is a small company without a data team, and she told us she has no idea what they’ll need to do when they start hiring one and building a data stack.

One of my Data Engineers was very surprised and reacted on the spot.

What do you mean, you don’t know what tools to use? Snowflake, and everything we use here.

I told her I’d probably just use PostgreSQL and Excel.

Plamena may not be a technical person (she openly admits she knows nothing about the technicalities around data), but as an experienced finance person, she knows that a small company can’t afford to spend over a quarter of a million dollars just on the warehouses. Add a BI tool, reverse ETL, data observability, and a couple more, and we’re talking about way over half a million.

I’d start with Postgres and Excel because at a company that size, your tool budget is your hiring budget. The stack everyone uses also eats the time you need for feedback, and while you integrate it, you’re stuck in fake productivity.

You also get the six questions I ask myself before buying any tool. They’re a way of thinking about the problem, and you answer them for your own company.

Say you’re the first data person in the company, and you’re given a budget to get started. Like most strong technical people, you decide to run a platform MVP first.

So, you get Snowflake for $100k/year, Fivetran with an estimated cost of $25k/year, and Looker for $75k/year. At this point, you are already looking at $200k/year just on the critical tooling.

In a small company, that’s the budget for your first few team members, and you’ve already spent it. This immediately turns you into a team lead without a team, which means you’re responsible for:

• tool integration

• pipeline implementation

• reporting

• stakeholder management

• strategising

And you’re supposed to do everything alone. Isn’t that a setup for failure?

If you want to check your own company before you sign for a tool or a stack, the five questions I answer before I decide on any tool are in the resource hub, each with the reading behind it, and the hub comes with a paid subscription.

The stack eats your time too

But the problem is not only financial. It’s also a project management issue.

Implementing a complex amalgam of big tools requires time. Even tools that promise you can get started in hours actually need time to move from a PoC to production. Not to mention the learning curve, legalities, and locking yourself into a contract for what could be the wrong tool for the job.

With all of this, you find yourself fighting the stack far more than doing what you’re there to do.

While you integrate, the business works around you in spreadsheets anyway. That’s the obvious cost, and I won’t repeat how the business moves without you.

The less obvious one is fake productivity. You keep thinking, “just that small bit and we’re done”, instead of delivering what you already have and collecting the feedback. It’s the reason I’d rather scope a minimum viable data product that proves value fast than build a bold one.

But the thing is, you never run out of small bits. There’s always one more source to connect, one more model to clean up, one more permission to set. Then one more workaround to finish before you show it to anyone (it’s almost ready anyway). Each of them is real work, and each of them feels like progress. So you keep saying you are almost done.

Meanwhile, nobody outside the data team sees anything. Nobody has the chance to tell you that you are building the wrong solution, that what they need changes while you’re integrating, or that they don’t need it in the first place.

That’s why I call it fake productivity. You are busy every single day, and you still learn nothing about whether anyone needs what you are building. Every small bit you add pushes the feedback further away and makes “this is not what I need” more expensive when it finally comes.

I give myself five working days from the day I start building to put something ugly in front of the person who asked for it. With Postgres and Excel, that’s realistic. While you are still integrating Snowflake, Fivetran and Looker, it isn’t, and the small bits keep coming.

I’ve seen this many times, mostly with people I’ve coached who came to me after a failure like this.

It doesn’t matter what they bought, and they don’t come to me for help picking tools. They come to learn data strategy and to think like a leader who connects tech and business.

What I told Plamena

I obviously don’t know enough about Plamena’s new place (even she doesn’t know enough yet), but my answer was based on what I already knew, and it was only for the first couple of months. I mean, spreadsheets were one of the options for our own People Analytics project.

But, when do you move off PostgreSQL and Excel?

Once you have clarity about the demand in terms of ingestion and query patterns, workloads, and all sorts of technical and business requirements. You only get there through feedback, and that’s why it matters more than anything.