Apache Airflow Reviews

What I like best about Apache Airflow is its flexibility in building and managing complex workflows using code. The DAG-based approach makes it easy to define dependencies, schedule jobs, and monitor pipeline execution in a centralized way.

Debugging failed workflows and managing dependencies across multiple tags can become challenging over time. Also, initial setup and maintenance can be complex, especially for large-scale deployments if it is not a managed service. Basic local setup is manageable, but production-grade deployment with scaling, monitoring, and high availability requires significant expertise.

Apache Airflow solves our workflow orchestration and scheduling challenges, automating workflows, triggering dependent jobs, managing retries, and coordinating data movement. This reduces manual intervention and improves pipeline reliability.

I use Apache Airflow to schedule and manage data pipelines, and I appreciate how it automates ETL pipelines easily and enables monitoring of tasks. I love its flexibility to build and manage complex workflows effortlessly using code, along with features like Operators and sensors. The UI is very helpful for tracking pipelines and quickly debugging failures. It helps me create, automate, and schedule complex workflows by reducing manual efforts, making pipeline monitoring and error handling much easier. I love most of the features that Airflow provides, and I rate it 9/10 as I love this platform.

For beginners, Apache Airflow can be complex to setup, especially with Docker. At times when a DAG is updated, the UI can feel slow.

I use Apache Airflow to automate ETL pipelines, reducing manual efforts and making pipeline monitoring and error handling easier.

It is easiy to deploy with docker. Provide secure authentication. A better UI in airlfow3.x. There. is many method, operator, hooks are added. easily to add dependecy. A better workflow monitoring tool. Easily schedule , activate and deactivate jobs. Good performance when used in any cloud platform like aws in serverless mode. easily scaleable. Easy to integrate third part library. Apache airflow is free of cost but you need to pay compute cost if you are using any cloud. A wide support community

When I create a task in expansion mode, it doesn’t display the graphical flow properly. I also noticed there’s too much logging. And when setting it up on AWS MWAA in a private VPC, it requires additional configuration.

I have thousands of jobs that I need to monitor daily. In the past, we created a custom Python script, but later we found Airflow and started using it. Now, in one place, we can monitor all running jobs along with their status, and we can easily notify the developer or the respective person to check any alerts. It also helps me provide a better orchestration flow.

I love using Apache Airflow for creating ETL pipelines, especially with its UI-driven DAG visualization that makes understanding the workflow and dependencies so much easier. The graph and stage view are amazing, and identifying errors is straightforward. The task monitoring feature is really useful, letting me see which jobs are running, and the retry and failure handling are essential for smooth operations. Setup was straightforward, particularly on AWS, and it's incredibly helpful when integrated with Acceldata for pipeline observability.

The only thing I noticed, when there are more DAGs, is that the UI can sometimes feel slow when dealing with many tasks.

Apache Airflow helps us create a robust ETL pipeline, automates Spark job flows, and simplifies error identification with its UI. The DAG visualization aids in understanding workflows and dependencies.

What I like most about Apache Airflow is how it makes managing workflows feel organised and predictable. You can clearly define tasks, set dependencies, and see everything laid out in one place, which makes complex pipelines much easier to understand.

I also like how flexible it is. You can build and customise workflows to fit pretty much any data pipeline, and the scheduling just works once it is set up. The UI is another big plus, it makes it easy to monitor runs, debug issues, and quickly see where something has failed without digging through logs for ages.

One thing I don’t like about Apache Airflow is that it can feel quite heavy and complex, especially at the start. Setting it up properly takes time, and there are quite a few moving parts to understand before everything runs smoothly.

the whole process is very extensible, since there are so many users, we have a lot of built in operators/plugins. the UI is very clean and intuitive.

i dont think there is a feature where we can watch the logs of all the jobs in one place, lets say we have 20-30 steps, so we have to go to each step and check the logs, so i would all logs to be consolidated

solves the problem of cron jobs and removes manual dependency.

For me, the standout feature is definitely the Web UI. As a data engineer, I often find myself troubleshooting, and the Grid view in Airflow makes it remarkably simple to identify exactly where a pipeline has failed. I can quickly access the logs for any specific task and determine what went wrong within seconds. This level of transparency is something that traditional cron jobs or basic scripts simply don't offer. Having a central dashboard for all your workflows truly provides peace of mind.

The main challenge is the ease of implementation for beginners. Setting up the infrastructure (like the webserver, scheduler, and database) requires a good bit of DevOps knowledge, which can be a hurdle for a small team.

Because it’s open-source, you don’t have traditional customer support, so you rely heavily on the community. While the community is active, the documentation can sometimes be a bit overwhelming when you are trying to troubleshoot a very specific configuration issue. It’s a powerful tool, but the ease of use from a setup perspective could definitely be improved.

Airflow has capabilities that overcomes traditional CRON jobs sscheduling, I have been using it since last 6+ years and it is helping me a lot to built robust pipelines with ease in backfilling tasks when any issue comes in. Also it has a good customer support when it comes to its upgradation

I do not dislike anything in terms of Airflow as it is already a best tool for data engineers and AI data engineers.It provides seamless integration.

I had nuerous problems in data pipelines scheduling, earlier there were overheads and manual config configuration referencing issues in python code. The xcom had helped a lot with python and bash operators to schedule jobs with each and push details to downstream data pipeline tasks.

I like Apache Airflow's clear DAGs since they make workflows easy to understand and maintain. The scheduling feature ensures pipelines run automatically without manual effort, which is really helpful. I also appreciate the retries and monitoring, as they help quickly detect and recover from failures. Additionally, its scalability is a significant advantage, allowing me to handle growing data workloads reliably, making Airflow dependable for production pipelines. Overall, these features really enhance my experience with Apache Airflow.

Some things in Apache Airflow not work very well for me. Setup and initial configuration is little complex and takes time. UI sometimes feels slow when lot of DAGs are running. Debugging failed tasks is not always clear, logs are scattered. Also version upgrades can break existing DAGs, backward compatibility should be better.

Apache Airflow manages and automates complex data workflows. It solves manual job running, tracks dependencies, and handles failures across systems. I can define task dependencies, schedule workflows, monitor executions, and fix failures, providing structure, visibility, and reliability to data orchestration.

Airflow enhances pipeline observability - both process and data - to the highest level.

It enables the distribution of pipeline execution among a team of stakeholders with varying technical backgrounds in a safe and user-friendly environment.

The installation, setup, and running are not straightforward, and some fine-tuning is necessary.