Replies: 1 comment 1 reply
-
To the best of what I know, there is no such limit or configuration item to restrict how many DAG files you can have in a single ZIP. May you share a bit more information, like the log or traceback? |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
In order to manage DAG dependencies on python custom modules, I am using the Packaged DAGs feature of Airflow i.e. putting the DAG.python file in root and putting custom_modules.py in dependencies folder with empty init.py file. I make a zip of this and copy it to the Airflow DAG folder location. This works as expected upto a limit. If my DAG files are within 6-8 count, all the DAGs get imported successfully on the UI and function properly. When the count increases to above 8-10, the DAGs get imported to the UI, but upon clicking them, Airflow throws an error - does not exist. When I split the DAGs into two zips, they get imported properly and can be accessed as well. Is there a upper limit to how many files can the packaged zip contain? Has anyone else faced this?
I am running the zip packaging through a CI/CD workflow and hence, was zipping all the files in my repo together. With this limitation, I will have to zip the same dependencies again in individual zip files causing redundancy. Also, the limit is unknown. Don't know when it may break again in production.
Beta Was this translation helpful? Give feedback.
All reactions