Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename execution_date to logical_date across codebase #43902

Merged
merged 15 commits into from
Nov 15, 2024

Conversation

sunank200
Copy link
Collaborator

@sunank200 sunank200 commented Nov 11, 2024

Motivation

This PR renames execution_date to logical_date across the codebase. The shift towards logical_date helps move away from the limitations of execution_date, particularly with dynamic DAG runs and cases where multiple runs occur at the same time.

Key Changes

  • Replaced all instances of execution_date with logical_date including models.
  • Updated references in database models, templates, and functions.
  • Removed unique constraints on execution_date in the database to allow multiple DAG runs with the same logical time.

How execution_date and logical_date Work

  1. Logical date is equivalent to execution date: The two are just different names for the same value.
  2. Timetable controls logical date: The logical date can be set to any value, not necessarily tied to the data interval's start or end.
  3. Schedules dictate behavior: For value-based schedules (like cron), the logical date is set by the timetable class used.

Transitioning from execution_date

  • Airflow 3.0 will fully remove execution_date.
  • For uniquely identifying runs:
    • Use run_id for human-readable identifiers.
    • Use data_interval_start/data_interval_end for data partitioning in future.
    • id (auto-incremented) is recommended for ordering DAG runs.
  • If your existing logic relies on execution_date, switch to using data_interval_start or data_interval_end for identifying time ranges.

Testing

  • Updated unit tests to reflect the changes.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:Scheduler including HA (high availability) scheduler provider:cncf-kubernetes Kubernetes provider related issues labels Nov 11, 2024
@sunank200 sunank200 added legacy api Whether legacy API changes should be allowed in PR legacy ui Whether legacy UI change should be allowed in PR labels Nov 11, 2024
@sunank200 sunank200 force-pushed the rename-execution-date-to-logical-date branch 9 times, most recently from ea36b6c to 65f9b8e Compare November 12, 2024 02:54
@uranusjr
Copy link
Member

This needs a news fragment. Rewording the above summary would work fine. See existing files in newsfragments (the significant ones) for some examples.

@uranusjr
Copy link
Member

uranusjr commented Nov 12, 2024

For templates, replace {{ ds }} with {{ data_interval_start | ds }}.

This should probably use logical_date for max compatibility. Also should mention ts (same rewrite).

Oh wait, we’re not removing those in this PR yet, so this should not be mentioned at all.

airflow/models/taskreschedule.py Show resolved Hide resolved
airflow/models/xcom.py Show resolved Hide resolved
airflow/utils/context.py Outdated Show resolved Hide resolved
airflow/utils/context.pyi Outdated Show resolved Hide resolved
@sunank200 sunank200 force-pushed the rename-execution-date-to-logical-date branch 2 times, most recently from 14a4cff to 16b27e8 Compare November 15, 2024 03:01
@sunank200 sunank200 force-pushed the rename-execution-date-to-logical-date branch from 16b27e8 to 96075b1 Compare November 15, 2024 03:01
@uranusjr uranusjr added the airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes label Nov 15, 2024
@uranusjr uranusjr merged commit 123dadd into apache:main Nov 15, 2024
83 checks passed
@uranusjr uranusjr deleted the rename-execution-date-to-logical-date branch November 15, 2024 09:15
hardeybisey pushed a commit to hardeybisey/airflow that referenced this pull request Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes area:API Airflow's REST/HTTP API area:CLI area:Scheduler including HA (high availability) scheduler legacy api Whether legacy API changes should be allowed in PR legacy ui Whether legacy UI change should be allowed in PR provider:cncf-kubernetes Kubernetes provider related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants