I've Been Using Meltano for 4 Years: Here's My Full Review
An honest look at the pros, cons, and everything in between
Looking for an open-source tool to streamline your data workflows?
Meltano is a developer-friendly platform that leverages standards like Singer and Airflow. It empowers data professionals like you to simplify data integration and orchestration.
I've been using Meltano since 2020. In this post, I'll share my experience with the platform. We'll look at its features, how easy it is to use, and how well it performs. By the end, you'll better know if Meltano is the right choice for your data projects.
But that's not all! Want to know if Meltano is right for you? Take this short quiz to find out!
Reading time: 6 minutes
⭐ User Experience
In my four-year journey with Meltano, I've found the user experience exceptional. The tool's code-first approach makes navigating easy, even for beginners.
By leveraging industry standards like Singer and Airflow, Meltano ensures smooth-running data pipelines. With a no-fluff interface and many plugins available, customisations are abundant.
Meltano is an excellent fit for tech-savvy users, but it might have a steeper learning curve for those new to data engineering. This is due to its heavy reliance on Airflow and the complexity of running it in a cloud environment. Considering these factors when evaluating Meltano for your data engineering needs is essential.
Meltano is a powerful tool for those looking for tried-and-true data solutions. But it does require some technical skills and may need tweaking to fit your needs.
⚙️ YAML
As with most modern data/infrastructure tools, Meltano relies on YAML and environment variables. It turns you from a data engineer into a YAML engineer.
Do you leverage different environments like dev, staging and prod? Meltano's configuration has built-in support through hard-coded config groups and environment variables. This turns your Meltano platform into an actual Twelve-factor app.
If you prefer buttons and dropdowns, this code-first approach may be a problem, but it is perfect for me.
🗣️ Singer
As I mentioned, Meltano relies on Singer to run integration jobs. As you may already know, I am a massive fan of Singer. Singer is a powerful and flexible standard for building data integration pipelines.
More than 100 extractors and 40 loaders are available in Meltano Hub. If you can't find what you want, leverage the Singer SDK and build your integration.
But it's not all rainbows and unicorns. Most plugins in Meltano have different variants supported by independent companies. This variety can be a double-edged sword, as finding the variant that best fits your needs requires time. It's essential to consider your requirements and each variant's capabilities before deciding.
Now, the Meltano team does a great job by trying all these and making the best variant, the default one. Yet another variant might better fit your needs.
Take any random SaaS plugin that pulls data via an API. Most plugin maintainers would only add the streams (API endpoints) they need. You must fork the plugin and add the stream if you need something different.
Another example is with PostgreSQL plugins. Say you choose to use the log-based replication for better reliability. Yet, all variants confirm the slot right after they consume the logs. So, you lose this data if something fails before you save it.
Now, these are not Meltano problems or Singer problems. These are problems. Yet, you need to take your timeframes and Python expertise into account.
🎭 Airflow
If you have even slight experience with Airflow, you have a love-hate relationship with the orchestrator.
First, Airflow is one of the most versatile tools in the data space. It allows you to run and orchestrate all sorts of tasks. But Airflow is rather stubborn and hard to work with. Yet, the Meltano team did a great job providing a good DAG generation template.
While reinventing the wheel doesn't make sense, Meltano relies too much on Airflow (or Dagster), which comes with some challenges. Let me tell you about a rare use case I had:
By default, Meltano stores logs on the disk. This can lead to challenges like disk overflow or log loss. We decided to leverage the company's ELK stack for the logs. The organisation has a strict schema to store and access the logs for easier multi-product handling.
Everything sounds great until you realise it has hard-coded the path to read logs. But this path is too simple and won't conform to your strategy. So you have two options:
🫠 Make an exception to the rule and allow other apps to do so
😵💫 Use Kibana to view your logs and spend much more time than if logs were accessible in Airflow
Again, this is not a strict Meltano problem, but you should know your needs and if Airflow meets them.
🧩 Other Plugins
As a comprehensive DataOps OS, Meltano can handle all other aspects of your data platform. You can expand your data platform's capabilities with tools like dbt, Elementary, Evidence, and Metabase.
Running Synq and other external tools is also straightforward. Again, some of those may need knowledge of Docker/Kubernetes.
The only downside is how hard running many dbt projects took a lot of work. I don't have any complaints here.
🏎️ Performance
With Singer, you can rely on speed, efficiency, and reliable data processing capabilities. It allowed us to handle increasing data volumes without compromising performance.
Meltano's ability to streamline data operations and automate processes has improved workflow efficiency. As Stéphane from Potloc says:
Whether you need to sync 500 or 5,000,000 rows, Meltano will perform the task with ease
However, running Meltano in a cloud environment can be challenging. Proper deployment requires extra technical expertise to overcome. Here's what you need to do to make Meltano super scalable:
Build a Docker image with your setup;
Create separate Kubernetes pods for the scheduler and the webserver;
Leverage Airflow's Kubernetes Executor.
These sound like three easy steps but can be challenging without a decent cloud-native experience.
📞 Support and Community
If you are looking to pay for 24/7 customer support, ready to answer any question, Meltano is not the tool for you. As an open-source tool, Meltano lacks paid support.
While Meltano may not offer paid support, it fosters a vibrant Slack community with nearly 5,000 members. They are always happy to help you with troubleshooting, best practices and new ideas.
You may wait for an answer, but there is a considerable chance someone else has the same issue and is willing to help you.
🏁 Conclusion
Meltano is a robust DataOps OS with a code-first approach, ease of use, and adherence to industry standards.
Meltano might be tough for non-techies and has some limits in the cloud. But its huge plugin library and customisations still make it a great choice for data pros.
Meltano is ideal for data engineers looking for a powerful tool based on industry standards. Yet, there may be better fits for those seeking a more user-friendly alternative like Fivetran.
Is Meltano a good fit for you? Check this quick quiz if you need more help.
📚 Picks of the Week
Build it or buy it? This article from
feels close to what you just read here. (link)I haven’t used PySpark in a long time. This
article reminded me of how we used to test our code. (link)This podcast episode from
is a real gem if you want to hear some unspoken truths about data leadership. (link)
😍 How Am I Doing?
I love hearing from readers and am always looking for feedback. How am I doing with Data Gibberish? Is there anything you’d like to see more or less of? Which aspects of the newsletter do you enjoy the most?
Hit reply and say hello. I’d love to hear from you!
Cool, do dlt next, it's the successor to singer for data people https://dlthub.com/docs/intro