(like CSVs or DataFrames); these should be stored in S3 or GCS instead. Database Bloat
Click the tab. You will see the key, value, and timestamp. Conclusion
The recommended approach for large data transfer is to and pass only the reference via XCom:
def push_task(**context): return "key": "value", "id": 123 airflow xcom exclusive
"Airflow XCom exclusive" refers to the practice of pushing XCom data targeted specifically for one or more downstream tasks, ensuring no other tasks mistakenly consume or rely on that data. It is a best practice for maintaining modularity and preventing unintended dependencies between tasks.
If a Python function task ends with a return statement used purely for local debugging, Airflow will still push it to the database. If the return value is unnecessary, remove the return statement or explicitly set do_xcom_push=False in your operator configurations.
Airflow 2.0 introduced the TaskFlow API, which completely abstracts explicit XCom calling syntax. Understanding how this builds upon underlying XCom networks gives data engineers an edge in writing clean pipelines. Example: Seamless Data Passing (like CSVs or DataFrames); these should be stored
Using Custom XCom Backends to store sensitive data in Vault or encrypted S3 buckets.
Use these strategies depending on your requirement:
To maintain database hygiene, implement a dedicated maintenance DAG that runs weekly to purge old XCom records: Conclusion The recommended approach for large data transfer
Since Airflow 2.0, the makes handling data between tasks much cleaner. When you return a value from a @task decorated function, it is automatically pushed as an XCom.
XComs solve this by acting as a centralized state-sharing mechanism. They are explicitly defined by a targeted trio of identifiers: : The pipeline the task belongs to. task_id : The specific task that generated the data.
+--------------------+ Implicit/Explicit Return +----------------------+ | | -----------------------------------> | | | Upstream Task A | | Airflow Metadata DB | | | <----------------------------------- | (xcom table) | +--------------------+ .xcom_pull(task_ids) +----------------------+ | | .xcom_pull() v +----------------------+ | | | Downstream Task B | | | +----------------------+ Implicit vs. Explicit XComs
Now the metadata DB stores only a URI reference; the exclusive payload lives in external storage.
In this example, task1 pushes a greeting message to XCom using xcom_push_key . task2 then pulls that message from XCom using xcom_pull and prints it.