11,974 questions
-1
votes
1
answer
27
views
How to dynamically create an unknown number of tasks in Airflow based on output of a previous task? [closed]
I have an Airflow DAG where the number of downstream tasks is unknown until an upstream task finishes. The workflow looks like this:
|---> Task B.1 --|
|---> Task B.2 --|
...
0
votes
0
answers
110
views
Local unit testing apache airflow
Cannot Run Apache Airflow Unit Tests: sqlalchemy.exc.OperationalError: unable to open database file
I am attempting to run a single unit test within the Apache Airflow repository (airflow-core/tests/...
- 21
1
vote
1
answer
57
views
XCom limit of 1024 in airflow
I am hitting the 1024 item limit in the size of a returned list in my task in airflow. I am currently on airflow v2.11. A simple bit of code that reproduces the issue I see is below.
import os, ...
-1
votes
1
answer
48
views
What cause this Airflow EmailOperator Error? [closed]
I am using Airflow 3.x for a personal task and faced this error
Failure caused by [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1010) source=task.stdout
What I want to do is send an email ...
- 1
-1
votes
2
answers
73
views
How to trigger a downstream Airflow DAG only when specific tasks in another DAG succeed? [closed]
I’m using Apache Airflow 2.x and I want to trigger a downstream DAG only when specific tasks in an upstream DAG succeed.
Here’s the situation:
I have an upstream DAG: dag_A
Inside dag_A, I have ...
1
vote
1
answer
69
views
Depedency Hell airflow+dbt
Hello I'm new to airflow, lately I'm struggling on a project with dbt+airflow+docker. My problem 1) I pip install dbt-core, dbt-duckdb adapter, 2) I try to install airflow with:
pip install "...
- 11
0
votes
1
answer
53
views
Add jinja template value with DateTime value
In an Airflow Dag file, I'm trying to handle following jinja template variables as DateTime values.
Specifically {{ data_interval_start }} and {{ data_interval_end }},
which (according to https://...
- 1,434
0
votes
0
answers
44
views
What does "state attribute is queued" mean for airflow 3?
I try to install the new airflow 3 with docker on a VM with this conf:
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE}
restart: unless-stopped
env_file: .env
environment:
# --...
- 2,704
Best practices
0
votes
0
replies
48
views
How to manage (Google Cloud Composer) Airflow roles with infrastructure as code?
Are there any best practices for handling roles in code?
Most of our setup is managed via Terraform. It would be great if there was a possibility.
There is a Terraform provider, however using the ...
- 3,731
-1
votes
0
answers
29
views
Great Expectations 1.8.0 – InvalidDataContextConfigError: “Error while processing DataContextConfig: anonymous_usage_statistics validations_store_name
I’m using Great Expectations 1.8.0 inside an Apache Airflow DAG (Python 3.12), and I keep getting this error whenever my task tries to initialize the FileDataContext:
InvalidDataContextConfigError: ...
0
votes
0
answers
53
views
KubernetesPodOperator - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
I’m running Apache Airflow inside a Docker container and trying to use the KubernetesPodOperator to run a simple “hello world” pod in an external Kubernetes cluster (not the same one where Airflow ...
Tooling
1
vote
0
replies
69
views
masking secrets airflow 2.4.3
I’m using Apache Airflow 2.4.3 and trying to securely store a Snowflake connection with a private key inside the connection’s extras JSON field.
I want to mask the sensitive private_key_content field ...
0
votes
0
answers
38
views
Airflow 3 - Run LocalExecutor with unlimited Parallelism
Airflow Docs say that the LocalExecutor supports an "unlimited Parallelism". Source
They do not specify exactly how to run LocalExecutor in this mode, but I assume it is through the core....
- 1,255
0
votes
0
answers
79
views
MWAA Airflow task_instance_mutation_hook not working
I have added a airflow_local_setting.py file to my mwaa s3 bucket at the root of dags folder. I have a dag_policy defined which is being picked up and works as expected.
In the same file I have added ...
- 13
0
votes
1
answer
123
views
Unable to use Airflow variable with function in Jinja template
I'm trying to pass Airflow logical_date to dbt model so that I can use it in the model (sql). I'm using Airflow 2.11.0. I'm doing below, but the dag couldn't get constructed, with an error says:
'...
- 367
