forked from apache/airflow
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGELOG.txt
3188 lines (3124 loc) · 184 KB
/
CHANGELOG.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Airflow 1.10.6, 2019-10-28
--------------------------
New Features
""""""""""""
- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546)
- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407)
- [AIRFLOW-4939] Add default_task_retries config (#5570)
- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130)
- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789)
- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743)
Improvements
""""""""""""
- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738)
- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737)
- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736)
- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330)
- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034)
- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194)
- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334)
- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891)
- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233)
- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303)
- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302)
- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304)
- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309)
- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153)
- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) (#6163)
- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012)
- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062)
- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059)
- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948)
- [AIRFLOW-3888] HA for Hive metastore connection (#4708)
- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873)
- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768)
- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214)
- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696)
- [AIRFLOW-2891] allow configurable docker_operator container name (#5689)
- [AIRFLOW-4285] Update task dependency context definition and usage (#5079)
- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758)
- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825)
- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589)
- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339)
- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195)
- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879)
- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159)
- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154)
- [AIRFLOW-4835] Refactor operator render_template (#5461)
Bug Fixes
"""""""""
- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078)
- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635)
- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747)
- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314)
- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance (#5908)
- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308)
- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102)
- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100)
- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106)
- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108)
- [AIRFLOW-5492] added missing docstrings (#6107)
- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125)
- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101)
- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678)
- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909)
- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185)
- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904)
- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841)
- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237)
- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307)
- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration (#5943)
- [AIRFLOW-5408] Fix env variable name in Kubernetes template (#6016)
- [AIRFLOW-5102] Worker jobs should terminate themselves if they can't heartbeat (#6284)
- [AIRFLOW-5572] Clear task reschedules when clearing task instances (#6217)
- [AIRFLOW-5543] Fix tooltip disappears in tree and graph view (RBAC UI) (#6174)
- [AIRFLOW-5444] Fix action_logging so that request.form for POST is logged (#6064)
- [AIRFLOW-5484] fix PigCliHook has incorrect named parameter (#6112)
- [AIRFLOW-5342] Fix MSSQL breaking task_instance db migration (#6014)
- [AIRFLOW-5556] Add separate config for timeout from scheduler dag processing (#6186)
- [AIRFLOW-4858] Deprecate "Historical convenience functions" in airflow.configuration (#5495) (#6144)
- [AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics (#6157)
- [AIRFLOW-5419] Use ``sudo`` to kill cleared tasks when running with impersonation (#6026) (#6176)
- [AIRFLOW-5537] Yamllint is not needed as dependency on host
- [AIRFLOW-5536] Better handling of temporary output files
- [AIRFLOW-5535] Fix name of VERBOSE parameter
- [AIRFLOW-5519] Fix sql_to_gcs operator missing multi-level default args by adding apply_defaults decorator (#6146)
- [AIRFLOW-5210] Make finding template files more efficient (#5815)
- [AIRFLOW-5447] Scheduler stalls because second watcher thread in default args (#6129)
Doc-only changes
""""""""""""""""
- [AIRFLOW-5574] Fix Google Analytics script loading (#6218)
- [AIRFLOW-5588] Add Celery's architecture diagram (#6247)
- [AIRFLOW-5521] Fix link to GCP documentation (#6150)
- [AIRFLOW-5398] Update contrib example DAGs to context manager (#5998)
- [AIRFLOW-5268] Apply same DAG naming conventions as in literature (#5874)
- [AIRFLOW-5101] Fix inconsistent owner value in examples (#5712)
- [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319)
- [AIRFLOW-XXX] Fix Documentation for adding extra Operator Links (#6301)
- [AIRFLOW-XXX] Add section on task lifecycle & correct casing in docs (#4681)
- [AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted from UPDATING.md (#6240)
- [AIRFLOW-XXX] Improve format in code-block directives (#6242)
- [AIRFLOW-XXX] Format Sendgrid docs (#6245)
- [AIRFLOW-XXX] Update to new logo (#6066)
- [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291)
- [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances in 1.10.4 (#6226)
- [AIRFLOW-XXX] Fix incorrect units in docs for metrics using Timers (#6152)
- [AIRFLOW-XXX] Fix backtick issues in .rst files & Add Precommit hook (#6162)
- [AIRFLOW-XXX] Update documentation about variables forcing answer (#6158)
- [AIRFLOW-XXX] Add a third way to configure authorization (#6134)
- [AIRFLOW-XXX] Add example of running pre-commit hooks on single file (#6143)
- [AIRFLOW-XXX] Add information about default pool to docs (#6019)
- [AIRFLOW-XXX] Make Breeze The default integration test environment (#6001)
Misc/Internal
"""""""""""""
- [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline (#6358) (#6361)
- [AIRFLOW-5533] Fixed failing CRON build (#6167)
- [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant from library (#5744)
- [AIRFLOW-5369] Adds interactivity to pre-commits (#5976)
- [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165)
- [AIRFLOW-4686] Make dags Pylint compatible (#5753)
- [AIRFLOW-4864] Remove calls to load_test_config (#5502)
- [AIRFLOW-XXX] Pin version of mypy so we are stable over time (#6198)
- [AIRFLOW-XXX] Add tests that got missed from #5127
- [AIRFLOW-4928] Move config parses to class properties inside DagBag (#5557)
- [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627)
- [AIRFLOW-5580] Add base class for system test (#6229)
Airflow 1.10.5, 2019-09-04
--------------------------
New Features
""""""""""""
- [AIRFLOW-1498] Add feature for users to add Google Analytics to Airflow UI (#5850)
- [AIRFLOW-4074] Add option to add labels to Dataproc jobs (#5606)
- [AIRFLOW-4846] Allow specification of an existing secret containing git credentials for init containers (#5475)
Improvements
""""""""""""
- [AIRFLOW-5335] Update GCSHook methods so they need min IAM perms (#5939)
- [AIRFLOW-2692] Allow AWS Batch Operator to use templates in job_name parameter (#3557)
- [AIRFLOW-4768] Add Timeout parameter in example_gcp_video_intelligence (#5862)
- [AIRFLOW-5165] Make Dataproc highly available (#5781)
- [AIRFLOW-5139] Allow custom ES configs (#5760)
- [AIRFLOW-5340] Fix GCP DLP example (#594)
- [AIRFLOW-5211] Add pass_value to template_fields BigQueryValueCheckOperator (#5816)
- [AIRFLOW-5113] Support icon url in slack web hook (#5724)
- [AIRFLOW-4230] bigquery schema update options should be a list (#5766)
- [AIRFLOW-1523] Clicking on Graph View should display related DAG run (#5866)
- [AIRFLOW-5027] Generalized CloudWatch log grabbing for ECS and SageMaker operators (#5645)
- [AIRFLOW-5244] Add all possible themes to default_webserver_config.py (#5849)
- [AIRFLOW-5245] Add more metrics around the scheduler (#5853)
- [AIRFLOW-5048] Improve display of Kubernetes resources (#5665)
- [AIRFLOW-5284] Replace deprecated log.warn by log.warning (#5881)
- [AIRFLOW-5276] Remove unused helpers from airflow.utils.helpers (#5878)
- [AIRFLOW-4316] Support setting kubernetes_environment_variables config section from env var (#5668)
Bug fixes
"""""""""
- [AIRFLOW-5168] Fix Dataproc operators that failed in 1.10.4 (#5928)
- [AIRFLOW-5136] Fix Bug with Incorrect template_fields in DataProc{*} Operators (#5751)
- [AIRFLOW-5169] Pass GCP Project ID explicitly to StorageClient in GCSHook (#5783)
- [AIRFLOW-5302] Fix bug in none_skipped Trigger Rule (#5902)
- [AIRFLOW-5350] Fix bug in the num_retires field in BigQueryHook (#5955)
- [AIRFLOW-5145] Fix rbac ui presents false choice to encrypt or not encrypt variable values (#5761)
- [AIRFLOW-5104] Set default schedule for GCP Transfer operators (#5726)
- [AIRFLOW-4462] Use datetime2 column types when using MSSQL backend (#5707)
- [AIRFLOW-5282] Add default timeout on kubeclient & catch HTTPError (#5880)
- [AIRFLOW-5315] TaskInstance not updating from DB when user changes executor_config (#5926)
- [AIRFLOW-4013] Mark success/failed is picking all execution date (#5616)
- [AIRFLOW-5152] Fix autodetect default value in GoogleCloudStorageToBigQueryOperator(#5771)
- [AIRFLOW-5100] Airflow scheduler does not respect safe mode setting (#5757)
- [AIRFLOW-4763] Allow list in DockerOperator.command (#5408)
- [AIRFLOW-5260] Allow empty uri arguments in connection strings (#5855)
- [AIRFLOW-5257] Fix ElasticSearch log handler errors when attemping to close logs (#5863)
- [AIRFLOW-1772] Google Updated Sensor doesnt work with CRON expressions (#5730)
- [AIRFLOW-5085] When you run kubernetes git-sync test from TAG, it fails (#5699)
- [AIRFLOW-5258] ElasticSearch log handler, has 2 times of hours (%H and %I) in _clean_execution_dat (#5864)
- [AIRFLOW-5348] Escape Label in deprecated chart view when set via JS (#5952)
- [AIRFLOW-5357] Fix Content-Type for exported variables.json file (#5963)
- [AIRFLOW-5109] Fix process races when killing processes (#5721)
- [AIRFLOW-5240] Latest version of Kombu is breaking airflow for py2
Misc/Internal
"""""""""""""
- [AIRFLOW-5111] Remove apt-get upgrade from the Dockerfile (#5722)
- [AIRFLOW-5209] Fix Documentation build (#5814)
- [AIRFLOW-5083] Check licence image building can be faster and moved to before-install (#5695)
- [AIRFLOW-5119] Cron job should always rebuild everything from scratch (#5733)
- [AIRFLOW-5108] In the CI local environment long-running kerberos might fail sometimes (#5719)
- [AIRFLOW-5092] Latest python image should be pulled locally in force_pull_and_build (#5705)
- [AIRFLOW-5225] Consistent licences can be added automatically for all JS files (#5827)
- [AIRFLOW-5229] Add licence to all other file types (#5831)
- [AIRFLOW-5227] Consistent licences for all .sql files (#5829)
- [AIRFLOW-5161] Add pre-commit hooks to run static checks for only changed files (#5777)
- [AIRFLOW-5159] Optimise checklicence image build (do not build if not needed) (#5774)
- [AIRFLOW-5263] Show diff on failure of pre-commit checks (#5869)
- [AIRFLOW-5204] Shell files should be checked with shellcheck and have identical licence (#5807)
- [AIRFLOW-5233] Check for consistency in whitespace (tabs/eols) and common problems (#5835)
- [AIRFLOW-5247] Getting all dependencies from NPM can be moved up in Dockerfile (#5870)
- [AIRFLOW-5143] Corrupted rat.jar became part of the Docker image (#5759)
- [AIRFLOW-5226] Consistent licences for all html JINJA templates (#5828)
- [AIRFLOW-5051] Coverage is not properly reported in the new CI system (#5732)
- [AIRFLOW-5239] Small typo and incorrect tests in CONTRIBUTING.md (#5844)
- [AIRFLOW-5287] Checklicence base image is not pulled (#5886)
- [AIRFLOW-5301] Some not-yet-available files from breeze are committed to master (#5901)
- [AIRFLOW-5285] Pre-commit pylint runs over todo files (#5884)
- [AIRFLOW-5288] Temporary container for static checks should be auto-removed (#5887)
- [AIRFLOW-5326] Fix teething proglems for Airflow breeze (#5933)
- [AIRFLOW-5206] All .md files should have all common licence, TOC (where applicable) (#5809)
- [AIRFLOW-5329] Easy way to add local files to docker (#5933)
- [AIRFLOW-4027] Make experimental api tests more stateless (#4854)
Doc-only changes
""""""""""""""""
- [AIRFLOW-XXX] Fixed Azkaban link (#5865)
- [AIRFLOW-XXX] Remove duplicate lines from CONTRIBUTING.md (#5830)
- [AIRFLOW-XXX] Fix incorrect docstring parameter in SchedulerJob (#5729)
Airflow 1.10.4, 2019-08-06
--------------------------
New Features
""""""""""""
- [AIRFLOW-4811] Implement GCP Data Loss Prevention Hook and Operators (#5539)
- [AIRFLOW-5035] Replace multiprocessing.Manager with a golang-"channel" style (#5615)
- [AIRFLOW-4883] Kill hung file process managers (#5605)
- [AIRFLOW-4929] Pretty print JSON Variables in UI (#5573)
- [AIRFLOW-4884] Roll up import_errors in RBAC UI (#5516)
- [AIRFLOW-4871] Allow creating DagRuns via RBAC UI (#5507)
- [AIRFLOW-4591] Make default_pool a real pool (#5349)
- [AIRFLOW-4844] Add optional is_paused_upon_creation argument to DAG (#5473)
- [AIRFLOW-4456] Add sub-classable BaseBranchOperator (#5231)
- [AIRFLOW-4343] Show warning in UI if scheduler is not running (#5127)
- [AIRFLOW-4739] Add ability to arbitrarily define kubernetes worker pod labels (#5376)
- [AIRFLOW-4348] Add GCP console link in BigQueryOperator (#5195)
- [AIRFLOW-4306] Global operator extra links (#5094)
- [AIRFLOW-4812] Add batch images annotation (#5433)
- [AIRFLOW-4135] Add Google Cloud Build operator and hook (#5251)
- [AIRFLOW-4781] Add the ability to specify ports in kubernetesOperator (#5410)
- [AIRFLOW-4521] Pause dag also pause its subdags (#5283)
- [AIRFLOW-4738] Enforce exampleinclude for example DAGs (#5375)
- [AIRFLOW-4326] Airflow AWS SQS Operator (#5110)
- [AIRFLOW-3729] Support "DownwardAPI" in env variables for KubernetesPodOperator (#4554)
- [AIRFLOW-4585] Implement Kubernetes Pod Mutation Hook (#5359)
- [AIRFLOW-161] New redirect route and extra links (#5059)
- [AIRFLOW-4420] Backfill respects task_concurrency (#5221)
- [AIRFLOW-4147] Add Operator to publish event to Redis (#4967)
- [AIRFLOW-3359] Add option to pass customer encryption keys to Dataproc (#4200)
- [AIRFLOW-4318] Create Google Cloud Translate Speech Operator (#5102)
- [AIRFLOW-3960] Adds Google Cloud Speech operators (#4780)
- [AIRFLOW-1501] Add GoogleCloudStorageDeleteOperator (#5230)
- [AIRFLOW-3672] Add support for Mongo DB DNS Seedlist Connection Format (#4481)
- [AIRFLOW-4397] add integrations docs manually for gcs sensors (#5204)
- [AIRFLOW-4251] Instrument DagRun schedule delay (#5050)
- [AIRFLOW-4118] instrument DagRun duration (#4946)
- [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure (#5182)
- [AIRFLOW-4361] Fix flaky test_integration_run_dag_with_scheduler_failure (#5140)
- [AIRFLOW-4168] Create Google Cloud Video Intelligence Operators (#4985)
- [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166)
- [AIRFLOW-4335] Add default num_retries to GCP connection (#5117)
- [AIRFLOW-3808] Add cluster_fields to BigQueryHook's create_empty_table (#4654)
- [AIRFLOW-4362] Fix test_execution_limited_parallelism (#5141)
- [AIRFLOW-4307] Backfill respects concurrency limit (#5128)
- [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator (#5077)
- [AIRFLOW-4169] Add Google Cloud Vision Detect Operators (#4986)
- [AIRFLOW-XXX] Fix WS-2019-0032 (#5384)
- [AIRFLOW-XXX] Fix CVE-2019-11358 (#5197)
- [AIRFLOW-XXX] Change allowed version of Jinja2 to fix CVE-2019-10906 (#5075)
Improvement
"""""""""""
- [AIRFLOW-5022] Fix DockerHook for registries with port numbers (#5644)
- [AIRFLOW-4961] Insert TaskFail.duration as int match DB schema column type (#5593)
- [AIRFLOW-5038] skip pod deleted log message when pod deletion is disabled (#5656)
- [AIRFLOW-5067] Update pagination symbols (#5682)
- [AIRFLOW-4981][AIRFLOW-4788] Always use pendulum DateTimes in task instance context (#5654)
- [AIRFLOW-4880] Add success, failure and fail_on_empty params to SqlSensor (#5488)
- [AIRFLOW-3617] Add gpu limits option in configurations for Kube executor and pod (#5643)
- [AIRFLOW-4998] Run multiple queries in BigQueryOperator (#5619)
- [AIRFLOW-4929] Improve display of JSON Variables in UI (#5641)
- [AIRFLOW-4959] Add .hql support for the DataProcHiveOperator (#5591)
- [AIRFLOW-4962] Fix Werkzeug v0.15 deprecation notice for DispatcherMiddleware import (#5595)
- [AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)
- [AIRFLOW-4911] Silence the FORBIDDEN errors from the KubernetesExecutor (#5547)
- [AIRFLOW-3495] Validate one of query and query_uri passed to DataProcSparkSqlOperator (#5510)
- [AIRFLOW-4925] Improve css style for Variables Import file field (#5552)
- [AIRFLOW-4906] Improve debugging for the SparkSubmitHook (#5542)
- [AIRFLOW-4904] unittest.cfg name and path can be overriden by setting $AIRFLOW_TEST_CONFIG (#5540)
- [AIRFLOW-4920] Use html.escape instead of cgi.escape to fix DeprecationWarning (#5551)
- [AIRFLOW-4919] DataProcJobBaseOperator dataproc_*_properties templated (#5555)
- [AIRFLOW-4478] Lazily instantiate default resources objects. (#5259)
- [AIRFLOW-4564] AzureContainerInstance bugfixes and improvements (#5319)
- [AIRFLOW-4237] Including Try Number of Task in Gantt Chart (#5037)
- [AIRFLOW-4862] Allow directly using IP address as hostname for webserver logs (#5501)
- [AIRFLOW-4857] Add templated fields to SlackWebhookOperator (#5490)
- [AIRFLOW-3502] Add celery config option for setting "pool" (#4308)
- [AIRFLOW-3217] Button to toggle line wrapping in log and code views (#4277)
- [AIRFLOW-4491] Add a "Jump to end" button for logs (#5266)
- [AIRFLOW-4422] Pool utilization stats (#5453)
- [AIRFLOW-4805] Add py_file as templated field in DataflowPythonOperator (#5451)
- [AIRFLOW-4838] Surface Athena errors in AWSAthenaOperator (#5467)
- [AIRFLOW-4831] conf.has_option no longer throws if section is missing. (#5455)
- [AIRFLOW-4829] More descriptive exceptions for EMR sensors (#5452)
- [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom (#5276)
- [AIRFLOW-4791] add "schema" keyword arg to SnowflakeOperator (#5415)
- [AIRFLOW-4759] Don't error when marking sucessful run as failed (#5435)
- [AIRFLOW-4716] Instrument dag loading time duration (#5350)
- [AIRFLOW-3958] Support list tasks as upstream in chain (#4779)
- [AIRFLOW-4409] Prevent task duration break by null value (#5178)
- [AIRFLOW-4418] Add "failed only" option to task modal (#5193)
- [AIRFLOW-4740] Accept string ``end_date`` in DAG default_args (#5381)
- [AIRFLOW-4423] Improve date handling in mysql to gcs operator. (#5196)
- [AIRFLOW-4447] Display task duration as human friendly format in UI (#5218)
- [AIRFLOW-4377] Remove needless object conversion in DAG.owner() (#5144)
- [AIRFLOW-4766] Add autoscaling option for DataprocClusterCreateOperator (#5425)
- [AIRFLOW-4795] Upgrade alembic to latest release. (#5411)
- [AIRFLOW-4793] Add signature_name to mlengine operator (#5417)
- [AIRFLOW-3211] Reattach to GCP Dataproc jobs upon Airflow restart (#4083)
- [AIRFLOW-4750] Log identified zombie task instances (#5389)
- [AIRFLOW-3870] STFPOperator: Update log level and return value (#4355)
- [AIRFLOW-4759] Batch queries in set_state API. (#5403)
- [AIRFLOW-2737] Restore original license header to airflow.api.auth.backend.kerberos_auth
- [AIRFLOW-3635] Fix incorrect logic in detele_dag (introduced in PR#4406) (#4445)
- [AIRFLOW-3599] Removed Dagbag from delete dag (#4406)
- [AIRFLOW-4737] Increase and document celery queue name limit (#5383)
- [AIRFLOW-4505] Correct Tag ALL for PY3 (#5275)
- [AIRFLOW-4743] Add environment variables support to SSHOperator (#5385)
- [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency (#5363)
- [AIRFLOW-3370] Add stdout output options to Elasticsearch task log handler (#5048)
- [AIRFLOW-4396] Provide a link to external Elasticsearch logs in UI. (#5164)
- [AIRFLOW-1381] Allow setting host temporary directory in DockerOperator (#5369)
- [AIRFLOW-4598] Task retries are not exhausted for K8s executor (#5347)
- [AIRFLOW-4218] Support to Provide http args to K8executor while calling k8 python client lib apis (#5060)
- [AIRFLOW-4159] Add support for additional static pod labels for K8sExecutor (#5134)
- [AIRFLOW-4720] Allow comments in .airflowignore files. (#5355)
- [AIRFLOW-4486] Add AWS IAM authenication in MySqlHook (#5334)
- [AIRFLOW-4417] Add AWS IAM authenication for PostgresHook (#5223)
- [AIRFLOW-3990] Compile regular expressions. (#4813)
- [AIRFLOW-4572] Rename prepare_classpath() to prepare_syspath() (#5328)
- [AIRFLOW-3869] Raise consistent exception in AirflowConfigParser.getboolean (#4692)
- [AIRFLOW-4571] Add headers to templated field for SimpleHttpOperator (#5326)
- [AIRFLOW-3867] Rename GCP's subpackage (#4690)
- [AIRFLOW-3725] Add private_key to bigquery_hook get_pandas_df (#4549)
- [AIRFLOW-4546] Upgrade google-cloud-bigtable. (#5307)
- [AIRFLOW-4519] Optimise operator classname sorting in views (#5282)
- [AIRFLOW-4503] Support fully pig options (#5271)
- [AIRFLOW-4468] add sql_alchemy_max_overflow parameter (#5249)
- [AIRFLOW-4467] Add dataproc_jars to templated fields in Dataproc oper… (#5248)
- [AIRFLOW-4381] Use get_direct_relative_ids get task relatives (#5147)
- [AIRFLOW-3624] Add masterType parameter to MLEngineTrainingOperator (#4428)
- [AIRFLOW-3143] Support Auto-Zone in DataprocClusterCreateOperator (#5169)
- [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support (#4695)
- [AIRFLOW-4399] Avoid duplicated os.path.isfile() check in models.dagbag (#5165)
- [AIRFLOW-4031] Allow for key pair auth in snowflake hook (#4875)
- [AIRFLOW-3901] add role as optional config parameter for SnowflakeHook (#4721)
- [AIRFLOW-3455] add region in snowflake connector (#4285)
- [AIRFLOW-4073] add template_ext for AWS Athena operator (#4907)
- [AIRFLOW-4093] AWSAthenaOperator: Throw exception if job failed/cancelled/reach max retries (#4919)
- [AIRFLOW-4356] Add extra RuntimeEnvironment keys to DataFlowHook (#5149)
- [AIRFLOW-4337] Fix docker-compose deprecation warning in CI (#5119)
- [AIRFLOW-3603] QuboleOperator: Remove SQLCommand from SparkCmd documentation (#4411)
- [AIRFLOW-4328] Fix link to task instances from Pool page (#5124)
- [AIRFLOW-4255] Make GCS Hook Backwards compatible (#5089)
- [AIRFLOW-4103] Allow uppercase letters in dataflow job names (#4925)
- [AIRFLOW-4255] Replace Discovery based api with client based for GCS (#5054)
- [AIRFLOW-4311] Remove sleep in localexecutor (#5096)
- [AIRFLOW-2836] Minor improvement-contrib.sensors.FileSensor (#3674)
- [AIRFLOW-4104] Add type annotations to common classes. (#4926)
- [AIRFLOW-3910] Raise exception explicitly in Connection.get_hook() (#4728)
- [AIRFLOW-3322] Update QuboleHook to fetch args dynamically from qds_sdk (#4165)
- [AIRFLOW-4565] instrument celery executor (#5321)
- [AIRFLOW-4573] Import airflow_local_settings after prepare_classpath (#5330)
- [AIRFLOW-4448] Don't bake ENV and _cmd into tmp config for non-sudo (#4050)
- [AIRFLOW-4295] Make ``method`` attribute case insensitive in HttpHook (#5313)
- [AIRFLOW-3703] Add dnsPolicy option for KubernetesPodOperator (#4520)
- [AIRFLOW-3057] add prev_*_date_success to template context (#5372)
- [AIRFLOW-4336] Stop showing entire GCS files bytes in log for gcs_download_operator (#5151)
- [AIRFLOW-4528] Cancel DataProc task on timeout (#5293)
Bug fixes
"""""""""
- [AIRFLOW-5089] Change version requirement on google-cloudp-spanner to work around version incompatibility (#5703)
- [AIRFLOW-4289] fix spark_binary argument being ignored in SparkSubmitHook (#5564)
- [AIRFLOW-5075] Let HttpHook handle connections with empty host fields (#5686)
- [AIRFLOW-4822] Fix bug where parent-dag task instances are wrongly cleared when using subdags (#5444)
- [AIRFLOW-5050] Correctly delete FAB permission m2m objects in ``airflow sync_perms`` (#5679)
- [AIRFLOW-5030] fix env var expansion for config key contains __ (#5650)
- [AIRFLOW-4590] changing log level to be proper library to suppress warning in WinRM (#5337)
- [AIRFLOW-4451] Allow named tuples to be templated (#5673)
- [AIRFLOW-XXX] Fix bug where Kube pod limts were not applied (requests were, but not limits) (#5657)
- [AIRFLOW-4775] Fix incorrect parameter order in GceHook (#5613)
- [AIRFLOW-4995] Fix DB initialisation on MySQL >=8.0.16 (#5614)
- [AIRFLOW-4934] Fix ProxyFix due to Werkzeug upgrade (#5563) (#5571)
- [AIRFLOW-4136] fix key_file of hook is overwritten by SSHHook connection (#5558)
- [AIRFLOW-4587] Replace self.conn with self.get_conn() in AWSAthenaHook (#5545)
- [AIRFLOW-1740] Fix xcom creation and update via UI (#5530) (#5531)
- [AIRFLOW-4900] Resolve incompatible version of Werkzeug (#5535)
- [AIRFLOW-4510] Don't mutate default_args during DAG initialization (#5277)
- [AIRFLOW-3360] Make the DAGs search respect other querystring parameters with url-search-params-polyfill for IE support (#5503)
- [AIRFLOW-4896] Make KubernetesExecutorConfig's default args immutable (#5534)
- [AIRFLOW-4494] Remove ``shell=True`` in DaskExecutor (#5273)
- [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC (#5525)
- [AIRFLOW-4892] Fix connection creation via UIs (#5527)
- [AIRFLOW-4406] Fix a method name typo: NullFernet.decrpyt to decrypt (#5509)
- [AIRFLOW-4849] Add gcp_conn_id to cloudsqldatabehook class to use correctly CloudSqlProxyRunner class (#5478)
- [AIRFLOW-4769] Pass gcp_conn_id to BigtableHook (#5445)
- [AIRFLOW-4524] Fix incorrect field names in view for Mark Success/Failure (#5486)
- [AIRFLOW-3671] Remove arg ``replace`` of MongoToS3Operator from ``kwargs`` (#4480)
- [AIRFLOW-4845] Fix bug where runAsUser 0 doesn't get set in k8s security context (#5474)
- [AIRFLOW-4354] Fix exception in "between" date filter in classic UI (#5480)
- [AIRFLOW-4587] Replace self.conn with self.get_conn() in AWSAthenaHook (#5462)
- [AIRFLOW-4516] K8s runAsUser and fsGroup cannot be strings (#5429)
- [AIRFLOW-4298] Stop Scheduler repeatedly warning "connection invalidated" (#5470)
- [AIRFLOW-4559] JenkinsJobTriggerOperator bugfix (#5318)
- [AIRFLOW-4841] Pin Sphinx AutoApi to 1.0.0 (#5468)
- [AIRFLOW-4479] Include s3_overwrite kwarg in load_bytes method (#5312)
- [AIRFLOW-3746] Fix DockerOperator missing container exit (#4583)
- [AIRFLOW-4233] Remove Template Extension from Bq to GCS Operator (#5456)
- [AIRFLOW-2141][AIRFLOW-3157][AIRFLOW-4170] Serialize non-str value by JSON when importing Variables (#4991)
- [AIRFLOW-4826] Remove warning from ``airflow resetdb`` command (#5447)
- [AIRFLOW-4148] Fix editing DagRuns when clicking state column (#5436)
- [AIRFLOW-4455] dag_details broken for subdags in RBAC UI (#5234)
- [AIRFLOW-2955] Fix kubernetes pod operator to set requests and limits on task pods (#4551)
- [AIRFLOW-4459] Fix wrong DAG count in /home page when DAG count is zero (#5235)
- [AIRFLOW-3876] AttributeError: module 'distutils' has no attribute 'util'
- [AIRFLOW-4146] Fix CgroupTaskRunner errors (#5224)
- [AIRFLOW-4524] Fix bug with "Ignore \*" toggles in RBAC mode (#5378)
- [AIRFLOW-4765] Fix DataProcPigOperator execute method (#5426)
- [AIRFLOW-4798] obviate interdependencies for dagbag and TI tests (#5422)
- [AIRFLOW-4800] fix GKEClusterHook ctor calls (#5424)
- [AIRFLOW-4799] don't mutate self.env in BashOperator execute method (#5421)
- [AIRFLOW-4393] Add retry logic when fetching pod status and/or logs in KubernetesPodOperator (#5284)
- [AIRFLOW-4174] Fix HttpHook run with backoff (#5213)
- [AIRFLOW-4463] Handle divide-by-zero errors in short retry intervals (#5243)
- [AIRFLOW-2614] Speed up trigger_dag API call when lots of DAGs in system
- [AIRFLOW-4756] add ti.state to ti.start_date as criteria for gantt (#5399)
- [AIRFLOW-4760] Fix zip-packaged DAGs disappearing from DagBag when reloaded (#5404)
- [AIRFLOW-4731] Fix GCS hook with google-storage-client 1.16 (#5368)
- [AIRFLOW-3506] use match_phrase to query log_id in elasticsearch (#4342)
- [AIRFLOW-4084] fix ElasticSearch log download (#5177)
- [AIRFLOW-4501] Register pendulum datetime converter for sqla+pymysql (#5190)
- [AIRFLOW-986] HiveCliHook ignores 'proxy_user' value in a connection's extra parameter (#5305)
- [AIRFLOW-4442] fix hive_tblproperties in HiveToDruidTransfer (#5211)
- [AIRFLOW-4557] Add gcp_conn_id parameter to get_sqlproxy_runner() of CloudSqlDatabaseHook (#5314)
- [AIRFLOW-4545] Upgrade FAB to latest version (#4955)
- [AIRFLOW-4492] Change Dataproc Cluster operators to poll Operations (#5269)
- [AIRFLOW-4452] Webserver and Scheduler keep crashing because of slackclient update (#5225)
- [AIRFLOW-4450] Fix request arguments in has_dag_access (#5220)
- [AIRFLOW-4434] Support Impala with the HiveServer2Hook (#5206)
- [AIRFLOW-3449] Write local dag parsing logs when remote logging enabled. (#5175)
- [AIRFLOW-4300] Fix graph modal call when DAG has not yet run (#5185)
- [AIRFLOW-4401] Use managers for Queue synchronization (#5200)
- [AIRFLOW-3626] Fixed triggering DAGs contained within zip files (#4439)
- [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766)
- [AIRFLOW-4403] search by ``dag_id`` or ``owners`` in UI (#5184)
- [AIRFLOW-4308] Fix TZ-loop around DST on python 3.6+ (#5095)
- [AIRFLOW-4324] fix DAG fuzzy search in RBAC UI (#5131)
- [AIRFLOW-4297] Temporary hot fix on manage_slas() for 1.10.4 release (#5150)
- [AIRFLOW-4299] Upgrade to Celery 4.3.0 to fix crashing workers (#5116)
- [AIRFLOW-4291] Correctly render doc_md in DAG graph page (#5121)
- [AIRFLOW-4310] Fix incorrect link on Dag Details page (#5122)
- [AIRFLOW-4331] Correct filter for Null-state runs from Dag Detail page (#5123)
- [AIRFLOW-4294] Fix missing dag & task runs in UI dag_id contains a dot (#5111)
- [AIRFLOW-4332] Upgrade sqlalchemy to remove security Vulnerability (#5113)
- [AIRFLOW-4312] Add template_fields & template_ext to BigQueryCheckO… (#5097)
- [AIRFLOW-4293] Fix downgrade in d4ecb8fbee3_add_schedule_interval_to_dag.py (#5086)
- [AIRFLOW-4267] Fix TI duration in Graph View (#5071)
- [AIRFLOW-4163] IntervalCheckOperator supports relative diff and not ignore 0 (#4983)
- [AIRFLOW-3938] QuboleOperator Fixes and Support for SqlCommand (#4832)
- [AIRFLOW-2903] Change default owner to ``airflow`` (#4151)
- [aIRFLOW-4136] Fix overwrite of key_file by constructor (#5155)
- [AIRFLOW-3241] Remove Invalid template ext in GCS Sensors (#4076)
Misc/Internal
"""""""""""""
- [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load (#5120)
- [AIRFLOW-4869] Reorganize sql to gcs operators. (#5504)
- [AIRFLOW-5021] move gitpython into setup_requires (#5640)
- [AIRFLOW-4583] Fixes type error in GKEPodOperator (#5612)
- [AIRFLOW-4116] Dockerfile now supports CI image build on DockerHub (#4937)
- [AIRFLOW-4115] Multi-staging Aiflow Docker image (#4936)
- [AIRFLOW-4963] Avoid recreating task context (#5596)
- [AIRFLOW-4865] Add context manager to set temporary config values in tests. (#5569)
- [AIRFLOW-4937] Fix lodash security issue with version below 4.17.13 (#5572) (used only in build-pipeline, not runtime)
- [AIRFLOW-4868] Fix typo in kubernetes/docker/build.sh (#5505)
- [AIRFLOW-4211] Add tests for WebHDFSHook (#5015)
- [AIRFLOW-4320] Add tests for SegmentTrackEventOperator (#5104)
- [AIRFLOW-4319] Add tests for Bigquery related Operators (#5101)
- [AIRFLOW-4014] Change DatastoreHook and add tests (#4842)
- [AIRFLOW-4322] Add test for VerticaOperator (#5107)
- [AIRFLOW-4323] Add 2 tests for WinRMOperator (#5108)
- [AIRFLOW-3677] Improve CheckOperator test coverage (#4756)
- [AIRFLOW-4659] Fix pylint problems for api module (#5398)
- [AIRFLOW-4358] Speed up test_jobs by not running tasks (#5162)
- [AIRFLOW-4394] Don't test behaviour of BackfillJob from CLI tests (#5160)
- [AIRFLOW-3471] Move XCom out of models.py (#4629)
- [AIRFLOW-4379] Remove duplicate code & Add validation in gcs_to_gcs.py (#5145)
- [AIRFLOW-4259] Move models out of models.py (#5056)
- [AIRFLOW-XXX] Speed up building of Cassanda module on Travis (#5233)
- [AIRFLOW-4535] Break jobs.py into multiple files (#5303)
- [AIRFLOW-1464] Batch update task_instance state (#5323)
- [AIRFLOW-4554] Test for sudo command, add some other test docs (#5310)
- [AIRFLOW-4419] Refine concurrency check in scheduler (#5194)
- [AIRFLOW-4269] Minor acceleration of jobs._process_task_instances() (#5076)
- [AIRFLOW-4341] Remove ``View.render()`` already exists in fab.BaseView (#5125)
- [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time (#5126)
- [AIRFLOW-4256] Remove noqa from migrations (#5055)
- [AIRFLOW-4034] Remove unnecessary string formatting with ``**locals()`` (#4861)
- [AIRFLOW-3944] Remove code smells (#4762)
Doc-only changes
""""""""""""""""
- [AIRFLOW-XXX] Add missing doc for annotations param of KubernetesPodOperator (#5666)
- [AIRFLOW-XXX] Fix typos in CONTRIBUTING.md (#5626)
- [AIRFLOW-XXX] Correct BaseSensorOperator docs (#5562)
- [AIRFLOW-4926] Fix example dags where its start_date is datetime.utcnow() (#5553)
- [AIRFLOW-4860] Remove Redundant Information in Example Dags (#5497)
- [AIRFLOW-4767] Fix errors in the documentation of Dataproc Operator (#5487)
- [AIRFLOW-1684] Branching based on XCom variable (Docs) (#4365)
- [AIRFLOW-3341] FAQ return DAG object example (#4605)
- [AIRFLOW-4433] Add missing type in DockerOperator doc string (#5205)
- [AIRFLOW-4321] Replace incorrect info of Max Size limit of GCS Object Size (#5106)
- [AIRFLOW-XXX] Add information about user list (#5341)
- [AIRFLOW-XXX] Clarify documentation related to autodetect parameter in GCS_to_BQ Op (#5294)
- [AIRFLOW-XXX] Remove mention of pytz compatibility from timezone documentation (#5316)
- [AIRFLOW-XXX] Add missing docs for GoogleCloudStorageDeleteOperator (#5274)
- [AIRFLOW-XXX] Remove incorrect note about Scopes of GCP connection (#5242)
- [AIRFLOW-XXX] Fix mistakes in docs of Dataproc operators (#5192)
- [AIRFLOW-XXX] Link to correct class for timedelta in macros.rst (#5226)
- [AIRFLOW-XXX] Add Kamil as committer (#5216)
- [AIRFLOW-XXX] Add Joshua and Kevin as committer (#5207)
- [AIRFLOW-XXX] Reduce log spam in tests (#5174)
- [AIRFLOW-XXX] Speed up tests for PythonSensor (#5158)
- [AIRFLOW-XXX] Add Bas Harenslak to committer list (#5157)
- [AIRFLOW-XXX] Add Jarek Potiuk to commiter list (#5132)
- [AIRFLOW-XXX] Update docstring for SchedulerJob (#5105)
- [AIRFLOW-XXX] Fix docstrings for CassandraToGoogleCloudStorageOperator (#5103)
- [AIRFLOW-XXX] update SlackWebhookHook and SlackWebhookOperator docstring (#5074)
- [AIRFLOW-XXX] Ignore python files under node_modules in docs (#5063)
- [AIRFLOW-XXX] Build a universal wheel with LICNESE files (#5052)
- [AIRFLOW-XXX] Fix docstrings of SQSHook (#5099)
- [AIRFLOW-XXX] Use Py3.7 on readthedocs
- [AIRFLOW-4446] Fix typos (#5217)
Airflow 1.10.3, 2019-04-09
--------------------------
New Feature
"""""""""""
- [AIRFLOW-4232] Add ``none_skipped`` trigger rule (#5032)
- [AIRFLOW-3971] Add Google Cloud Natural Language operators (#4980)
- [AIRFLOW-4069] Add Opsgenie Alert Hook and Operator (#4903)
- [AIRFLOW-3552] Fix encoding issue in ImapAttachmentToS3Operator (#5040)
- [AIRFLOW-3552] Add ImapAttachmentToS3Operator (#4476)
- [AIRFLOW-1526] Add dingding hook and operator (#4895)
- [AIRFLOW-3490] Add BigQueryHook's Ability to Patch Table/View (#4299)
- [AIRFLOW-3918] Add ssh private-key support to git-sync for KubernetesExecutor (#4777)
- [AIRFLOW-3659] Create Google Cloud Transfer Service Operators (#4792)
- [AIRFLOW-3939] Add Google Cloud Translate operator (#4755)
- [AIRFLOW-3541] Add Avro logical type conversion to bigquery hook (#4553)
- [AIRFLOW-4106] instrument staving tasks in pool (#4927)
- [AIRFLOW-2568] Azure Container Instances operator (#4121)
- [AIRFLOW-4107] instrument executor (#4928)
- [AIRFLOW-4033] record stats of task duration (#4858)
- [AIRFLOW-3892] Create Redis pub sub sensor (#4712)
- [AIRFLOW-4124] add get_table and get_table_location in aws_glue_hook and tests (#4942)
- [AIRFLOW-1262] Adds missing docs for email configuration (#4557)
- [AIRFLOW-3701] Add Google Cloud Vision Product Search operators (#4665)
- [AIRFLOW-3766] Add support for kubernetes annotations (#4589)
- [AIRFLOW-3741] Add extra config to Oracle hook (#4584)
- [AIRFLOW-1262] Allow configuration of email alert subject and body (#2338)
- [AIRFLOW-2985] Operators for S3 object copying/deleting (#3823)
- [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators (#3828)
- [AIRFLOW-3799] Add compose method to GoogleCloudStorageHook (#4641)
- [AIRFLOW-3218] add support for poking a whole DAG (#4058)
- [AIRFLOW-3315] Add ImapAttachmentSensor (#4161)
- [AIRFLOW-3556] Add cross join set dependency function (#4356)
Improvement
"""""""""""
- [AIRFLOW-3823] Exclude branch's downstream tasks from the tasks to skip (#4666)
- [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes (#4648)
- [AIRFLOW-4247] Template Region on the DataprocOperators (#5046)
- [AIRFLOW-4008] add envFrom for Kubernetes Executor (#4952)
- [AIRFLOW-3947] Flash msg for no DAG-level access error (#4767)
- [AIRFLOW-3287] Moving database clean-up code into the CoreTest.tearDown() (#4122)
- [AIRFLOW-4058] Name models test file to get automatically picked up (#4901)
- [AIRFLOW-3830] Remove DagBag from /dag_details (#4831)
- [AIRFLOW-3596] Clean up undefined template variables. (#4401)
- [AIRFLOW-3573] Remove DagStat table (#4378)
- [AIRFLOW-3623] Fix bugs in Download task logs (#5005)
- [AIRFLOW-4173] Improve SchedulerJob.process_file() (#4993)
- [AIRFLOW-3540] Warn if old airflow.cfg file is found (#5006)
- [AIRFLOW-4000] Return response when no file (#4822)
- [AIRFLOW-3383] Rotate fernet keys. (#4225)
- [AIRFLOW-3003] Pull the krb5 image instead of building (#3844)
- [AIRFLOW-3862] Check types with mypy. (#4685)
- [AIRFLOW-251] Add option SQL_ALCHEMY_SCHEMA parameter to specify schema for metadata (#4199)
- [AIRFLOW-1814] Temple PythonOperator {op_args,op_kwargs} fields (#4691)
- [AIRFLOW-3730] Standarization use of logs mechanisms (#4556)
- [AIRFLOW-3770] Validation of documentation on CI] (#4593)
- [AIRFLOW-3866] Run docker-compose pull silently in CI (#4688)
- [AIRFLOW-3685] Move licence header check (#4497)
- [AIRFLOW-3670] Add stages to Travis build (#4477)
- [AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef and secretRef (#4772)
- [AIRFLOW-3408] Remove outdated info from Systemd Instructions (#4269)
- [AIRFLOW-3202] add missing documentation for AWS hooks/operator (#4048)
- [AIRFLOW-3908] Add more Google Cloud Vision operators (#4791)
- [AIRFLOW-2915] Add example DAG for GoogleCloudStorageToBigQueryOperator (#3763)
- [AIRFLOW-3062] Add Qubole in integration docs (#3946)
- [AIRFLOW-3288] Add SNS integration (#4123)
- [AIRFLOW-3148] Remove unnecessary arg "parameters" in RedshiftToS3Transfer (#3995)
- [AIRFLOW-3049] Add extra operations for Mongo hook (#3890)
- [AIRFLOW-3559] Add missing options to DatadogHook. (#4362)
- [AIRFLOW-1191] Simplify override of spark submit command. (#4360)
- [AIRFLOW-3155] Add ability to filter by a last modified time in GCS Operator (#4008)
- [AIRFLOW-2864] Fix docstrings for SubDagOperator (#3712)
- [AIRFLOW-4062] Improve docs on install extra package commands (#4966)
- [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME (#4705)
- [AIRFLOW-4002] Option to open debugger on errors in `airflow test`. (#4828)
- [AIRFLOW-3997] Extend Variable.get so it can return None when var not found (#4819)
- [AIRFLOW-4009] Fix docstring issue in GCSToBQOperator (#4836)
- [AIRFLOW-3980] Unify logger (#4804)
- [AIRFLOW-4076] Correct port type of beeline_default in init_db (#4908)
- [AIRFLOW-4046] Add validations for poke_interval & timeout for Sensor (#4878)
- [AIRFLOW-3744] Abandon the use of obsolete aliases of methods (#4568)
- [AIRFLOW-3865] Add API endpoint to get python code of dag by id (#4687)
- [AIRFLOW-3516] Support to create k8 worker pods in batches (#4434)
- [AIRFLOW-2843] Add flag in ExternalTaskSensor to check if external DAG/task exists (#4547)
- [AIRFLOW-2224] Add support CSV files in MySqlToGoogleCloudStorageOperator (#4738)
- [AIRFLOW-3895] GoogleCloudStorageHook/Op create_bucket takes optional resource params (#4717)
- [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view (#4774)
- [AIRFLOW-4006] Make better use of Set in AirflowSecurityManager (#4833)
- [AIRFLOW-3917] Specify alternate kube config file/context when running out of cluster (#4859)
- [AIRFLOW-3911] Change Harvesting DAG parsing results to DEBUG log level (#4729)
- [AIRFLOW-3584] Use ORM DAGs for index view. (#4390)
- [AIRFLOW-2821] Refine Doc "Plugins" (#3664)
- [AIRFLOW-3561] Improve queries (#4368)
- [AIRFLOW-3600] Remove dagbag from trigger (#4407)
- [AIRFLOW-3713] Updated documentation for GCP optional project_id (#4541)
- [AIRFLOW-2767] Upgrade gunicorn to 19.5.0 to avoid moderate-severity CVE (#4795)
- [AIRFLOW-3795] provide_context param is now used (#4735)
- [AIRFLOW-4012] Upgrade tabulate to 0.8.3 (#4838)
- [AIRFLOW-3623] Support download logs by attempts from UI (#4425)
- [AIRFLOW-2715] Use region setting when launching Dataflow templates (#4139)
- [AIRFLOW-3932] Update unit tests and documentation for safe mode flag. (#4760)
- [AIRFLOW-3932] Optionally skip dag discovery heuristic. (#4746)
- [AIRFLOW-3258] K8S executor environment variables section. (#4627)
- [AIRFLOW-3931] set network, subnetwork when launching dataflow template (#4744)
- [AIRFLOW-4095] Add template_fields for S3CopyObjectOperator & S3DeleteObjectsOperator (#4920)
- [AIRFLOW-2798] Remove needless code from models.py
- [AIRFLOW-3731] Constrain mysqlclient to <1.4 (#4558)
- [AIRFLOW-3139] include parameters into log.info in SQL operators, if any (#3986)
- [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks (#4043)
- [AIRFLOW-3933] Fix various typos (#4747)
- [AIRFLOW-3905] Allow using "parameters" in SqlSensor (#4723)
- [AIRFLOW-2761] Parallelize enqueue in celery executor (#4234)
- [AIRFLOW-3540] Respect environment config when looking up config file. (#4340)
- [AIRFLOW-2156] Parallelize Celery Executor task state fetching (#3830)
- [AIRFLOW-3702] Add backfill option to run backwards (#4676)
- [AIRFLOW-3821] Add replicas logic to GCP SQL example DAG (#4662)
- [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator (#4347)
- [AIRFLOW-3647] Add archives config option to SparkSubmitOperator (#4467)
- [AIRFLOW-3802] Updated documentation for HiveServer2Hook (#4647)
- [AIRFLOW-3817] Corrected task ids returned by BranchPythonOperator to match the dummy operator ids (#4659)
- [AIRFLOW-3782] Clarify docs around celery worker_autoscale in default_airflow.cfg (#4609)
- [AIRFLOW-1945] Add Autoscale config for Celery workers (#3989)
- [AIRFLOW-3590] Change log message of executor exit status (#4616)
- [AIRFLOW-3591] Fix start date, end date, duration for rescheduled tasks (#4502)
- [AIRFLOW-3709] Validate `allowed_states` for ExternalTaskSensor (#4536)
- [AIRFLOW-3522] Add support for sending Slack attachments (#4332)
- [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www only) (#4373)
- [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www_rbac only) (#4373)
- [AIRFLOW-3044] Dataflow operators accept templated job_name param (#3887)
- [AIRFLOW-3023] Fix docstring datatypes
- [AIRFLOW-2928] Use uuid4 instead of uuid1 (#3779)
- [AIRFLOW-2988] Run specifically python2 for dataflow (#3826)
- [AIRFLOW-3697] Vendorize nvd3 and slugify (#4513)
- [AIRFLOW-3692] Remove ENV variables to avoid GPL (#4506)
- [AIRFLOW-3907] Upgrade flask and set cookie security flags. (#4725)
- [AIRFLOW-3698] Add documentation for AWS Connection (#4514)
- [AIRFLOW-3616][AIRFLOW-1215] Add aliases for schema with underscore (#4523)
- [AIRFLOW-3375] Support returning multiple tasks with BranchPythonOperator (#4215)
- [AIRFLOW-3742] Fix handling of "fallback" for AirflowConfigParsxer.getint/boolean (#4674)
- [AIRFLOW-3742] Respect the `fallback` arg in airflow.configuration.get (#4567)
- [AIRFLOW-3789] Fix flake8 3.7 errors. (#4617)
- [AIRFLOW-3602] Improve ImapHook handling of retrieving no attachments (#4475)
- [AIRFLOW-3631] Update flake8 and fix lint. (#4436)
Bug fixes
"""""""""
- [AIRFLOW-4248] Fix 'FileExistsError' makedirs race in file_processor_handler (#5047)
- [AIRFLOW-4240] State-changing actions should be POST requests (#5039)
- [AIRFLOW-4246] Flask-Oauthlib needs downstream dependencies pinning due to breaking changes (#5045)
- [AIRFLOW-3887] Downgrade dagre-d3 to 0.4.18 (#4713)
- [AIRFLOW-3419] Fix S3Hook.select_key on Python3 (#4970)
- [AIRFLOW-4127] Correct AzureContainerInstanceHook._get_instance_view's return (#4945)
- [AIRFLOW-4172] Fix changes for driver class path option in Spark Submit (#4992)
- [AIRFLOW-3615] Preserve case of UNIX socket paths in Connections (#4591)
- [AIRFLOW-3417] ECSOperator: pass platformVersion only for FARGATE launch type (#4256)
- [AIRFLOW-3884] Fixing doc checker, no warnings allowed anymore and fixed the current… (#4702)
- [AIRFLOW-2652] implement / enhance baseOperator deepcopy
- [AIRFLOW-4001] Update docs about how to run tests (#4826)
- [AIRFLOW-3699] Speed up Flake8 (#4515)
- [AIRFLOW-4160] Fix redirecting of 'Trigger Dag' Button in DAG Page (#4982)
- [AIRFLOW-3650] Skip running on mysql for the flaky test (#4457)
- [AIRFLOW-3423] Fix mongo hook to work with anonymous access (#4258)
- [AIRFLOW-3982] Fix race condition in CI test (#4968)
- [AIRFLOW-3982] Update DagRun state based on its own tasks (#4808)
- [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names (#4636)
- [AIRFLOW-3945] Stop inserting row when permission views unchanged (#4764)
- [AIRFLOW-4123] Add Exception handling for _change_state method in K8 Executor (#4941)
- [AIRFLOW-3771] Minor refactor securityManager (#4594)
- [AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos.run() (#4238)
- [AIRFLOW-3736] Allow int value in SqoopOperator.extra_import_options(#4906)
- [AIRFLOW-4063] Fix exception string in BigQueryHook [2/2] (#4902)
- [AIRFLOW-4063] Fix exception string in BigQueryHook (#4899)
- [AIRFLOW-4037] Log response in SimpleHttpOperator even if the response check fails
- [AIRFLOW-4044] The documentation of `query_params` in `BigQueryOperator` is wrong. (#4876)
- [AIRFLOW-4015] Make missing API endpoints available in classic mode
- [AIRFLOW-3153] Send DAG processing stats to statsd (#4748)
- [AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#4209)
- [AIRFLOW-4129] Escape HTML in generated tooltips (#4950)
- [AIRFLOW-4070] AirflowException -> log.warning for duplicate task dependencies (#4904)
- [AIRFLOW-4054] Fix assertEqualIgnoreMultipleSpaces util & add tests (#4886)
- [AIRFLOW-3239] Fix test recovery further (#4074)
- [AIRFLOW-4053] Fix KubePodOperator Xcom on Kube 1.13.0 (#4883)
- [AIRFLOW-2961] Refactor tests.BackfillJobTest.test_backfill_examples test (#3811)
- [AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was broken (#4415)
- [AIRFLOW-3543] Fix deletion of DAG with rescheduled tasks (#4646)
- [AIRFLOW-2548] Output plugin import errors to web UI (#3930)
- [AIRFLOW-4019] Fix AWS Athena Sensor object has no attribute 'mode' (#4844)
- [AIRFLOW-3758] Fix circular import in WasbTaskHandler (#4601)
- [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files (#4947)
- [AIRFLOW-4100] Correctly JSON escape data for tree/graph views (#4921)
- [AIRFLOW-3636] Fix a test introduced in #4425 (#4446)
- [AIRFLOW-3977] Add examples of trigger rules in doc (#4805)
- [AIRFLOW-2511] Fix improper failed session commit handling causing deadlocks (#4769)
- [AIRFLOW-3962] Added graceful handling for creation of dag_run of a dag which doesn't have any task (#4781)
- [AIRFLOW-3881] Correct to_csv row number (#4699)
- [AIRFLOW-3875] Simplify SlackWebhookHook code and change docstring (#4696)
- [AIRFLOW-3733] Don't raise NameError in HQL hook to_csv when no rows returned (#4560)
- [AIRFLOW-3734] Fix hql not run when partition is None (#4561)
- [AIRFLOW-3767] Correct bulk insert function (#4773)
- [AIRFLOW-4087] remove sudo in basetaskrunner on_finish (#4916)
- [AIRFLOW-3768] Escape search parameter in pagination controls (#4911)
- [AIRFLOW-4045] Fix hard-coded URLs in FAB-based UI (#4914)
- [AIRFLOW-3123] Use a stack for DAG context management (#3956)
- [AIRFLOW-3060] DAG context manager fails to exit properly in certain circumstances
- [AIRFLOW-3924] Fix try number in alert emails (#4741)
- [AIRFLOW-4083] Add tests for link generation utils (#4912)
- [AIRFLOW-2190] Send correct HTTP status for base_url not found (#4910)
- [AIRFLOW-4015] Add get_dag_runs GET endpoint to "classic" API (#4884)
- [AIRFLOW-3239] Enable existing CI tests (#4131)
- [AIRFLOW-1390] Update Alembic to 0.9 (#3935)
- [AIRFLOW-3885] Fix race condition in scheduler test (#4737)
- [AIRFLOW-3885] ~10x speed-up of SchedulerJobTest suite (#4730)
- [AIRFLOW-3780] Fix some incorrect when base_url is used (#4643)
- [AIRFLOW-3807] Fix Graph View Highlighting of Tasks (#4653)
- [AIRFLOW-3009] Import Hashable from collection.abc to fix Python 3.7 deprecation warning (#3849)
- [AIRFLOW-2231] Fix relativedelta DAG schedule_interval (#3174)
- [AIRFLOW-2641] Fix MySqlToHiveTransfer to handle MySQL DECIMAL correctly
- [AIRFLOW-3751] Option to allow malformed schemas for LDAP authentication (#4574)
- [AIRFLOW-2888] Add deprecation path for task_runner config change (#4851)
- [AIRFLOW-2930] Fix celery excecutor scheduler crash (#3784)
- [AIRFLOW-2888] Remove shell=True and bash from task launch (#3740)
- [AIRFLOW-3885] ~2.5x speed-up for backfill tests (#4731)
- [AIRFLOW-3885] ~20x speed-up of slowest unit test (#4726)
- [AIRFLOW-2508] Handle non string types in Operators templatized fields (#4292)
- [AIRFLOW-3792] Fix validation in BQ for useLegacySQL & queryParameters (#4626)
- [AIRFLOW-3749] Fix Edit Dag Run page when using RBAC (#4613)
- [AIRFLOW-3801] Fix DagBag collect dags invocation to prevent examples to be loaded (#4677)
- [AIRFLOW-3774] Register blueprints with RBAC web app (#4598)
- [AIRFLOW-3719] Handle StopIteration in CloudWatch logs retrieval (#4516)
- [AIRFLOW-3108] Define get_autocommit method for MsSqlHook (#4525)
- [AIRFLOW-3074] Add relevant ECS options to ECS operator. (#3908)
- [AIRFLOW-3353] Upgrade Redis client (#4834)
- [AIRFLOW-3250] Fix for Redis Hook for not authorised connection calls (#4090)
- [AIRFLOW-2009] Fix dataflow hook connection-id (#4563)
- [AIRFLOW-2190] Fix TypeError when returning 404 (#4596)
- [AIRFLOW-2876] Update Tenacity to 4.12 (#3723)
- [AIRFLOW-3923] Update flask-admin dependency to 1.5.3 to resolve security vulnerabilities from safety (#4739)
- [AIRFLOW-3683] Fix formatting of error message for invalid TriggerRule (#4490)
- [AIRFLOW-2787] Allow is_backfill to handle NULL DagRun.run_id (#3629)
- [AIRFLOW-3780] Fix some incorrect when base_url is used (#4643)
- [AIRFLOW-3639] Fix request creation in Jenkins Operator (#4450)
- [AIRFLOW-3779] Don't install enum34 backport when not needed (#4620)
- [AIRFLOW-3079] Improve migration scripts to support MSSQL Server (#3964)
- [AIRFLOW-2735] Use equality, not identity, check for detecting AWS Batch failures[]
- [AIRFLOW-2706] AWS Batch Operator should use top-level job state to determine status
- [AIRFLOW-XXX] Fix typo in http_operator.py
- [AIRFLOW-XXX] Solve lodash security warning (#4820)
- [AIRFLOW-XXX] Pin version of tornado pulled in by Celery. (#4815)
- [AIRFLOW-XXX] Upgrade FAB to 1.12.3 (#4694)
- [AIRFLOW-XXX] Pin pinodb dependency (#4704)
- [AIRFLOW-XXX] Pin version of Pip in tests to work around pypa/pip#6163 (#4576)
- [AIRFLOW-XXX] Fix spark submit hook KeyError (#4578)
- [AIRFLOW-XXX] Pin psycopg2 due to breaking change (#5036)
- [AIRFLOW-XXX] Pin Sendgrid dep. (#5031)
- [AIRFLOW-XXX] Fix flaky test - test_execution_unlimited_parallelism (#4988)
Misc/Internal
"""""""""""""
- [AIRFLOW-4144] add description of is_delete_operator_pod (#4943)
- [AIRFLOW-3476][AIRFLOW-3477] Move Kube classes out of models.py (#4443)
- [AIRFLOW-3464] Move SkipMixin out of models.py (#4386)
- [AIRFLOW-3463] Move Log out of models.py (#4639)
- [AIRFLOW-3458] Move connection tests (#4680)
- [AIRFLOW-3461] Move TaskFail out of models.py (#4630)
- [AIRFLOW-3462] Move TaskReschedule out of models.py (#4618)
- [AIRFLOW-3474] Move SlaMiss out of models.py (#4608)
- [AIRFLOW-3475] Move ImportError out of models.py (#4383)
- [AIRFLOW-3459] Move DagPickle to separate file (#4374)
- [AIRFLOW-3925] Don't pull docker-images on pretest (#4740)
- [AIRFLOW-4154] Correct string formatting in jobs.py (#4972)
- [AIRFLOW-3458] Deprecation path for moving models.Connection
- [AIRFLOW-3458] Move models.Connection into separate file (#4335)
- [AIRFLOW-XXX] Remove old/non-test files that nose ignores (#4930)
Doc-only changes
""""""""""""""""
- [AIRFLOW-3996] Add view source link to included fragments
- [AIRFLOW-3811] automatic generation of API Reference in docs (#4788)
- [AIRFLOW-3810] Remove duplicate autoclass directive (#4656)
- [AIRFLOW-XXX] Mention that statsd must be installed to gather metrics (#5038)
- [AIRFLOW-XXX] Add contents to cli (#4825)
- [AIRFLOW-XXX] fix check docs failure on CI (#4998)
- [AIRFLOW-XXX] Fix syntax docs errors (#4789)
- [AIRFLOW-XXX] Docs rendering improvement (#4684)
- [AIRFLOW-XXX] Automatically link Jira/GH on doc's changelog page (#4587)
- [AIRFLOW-XXX] Mention Oracle in the Extra Packages documentation (#4987)
- [AIRFLOW-XXX] Drop deprecated sudo option; use default docker compose on Travis. (#4732)
- [AIRFLOW-XXX] Update kubernetes.rst docs (#3875)
- [AIRFLOW-XXX] Improvements to formatted content in documentation (#4835)
- [AIRFLOW-XXX] Add Daniel to committer list (#4961)
- [AIRFLOW-XXX] Add Xiaodong Deng to committers list
- [AIRFLOW-XXX] Add history become ASF top level project (#4757)
- [AIRFLOW-XXX] Move out the examples from integration.rst (#4672)
- [AIRFLOW-XXX] Extract reverse proxy info to a separate file (#4657)
- [AIRFLOW-XXX] Reduction of the number of warnings in the documentation (#4585)
- [AIRFLOW-XXX] Fix GCS Operator docstrings (#4054)
- [AIRFLOW-XXX] Fix Docstrings in Hooks, Sensors & Operators (#4137)
- [AIRFLOW-XXX] Split guide for operators to multiple files (#4814)
- [AIRFLOW-XXX] Split connection guide to multiple files (#4824)
- [AIRFLOW-XXX] Remove almost all warnings from building docs (#4588)
- [AIRFLOW-XXX] Add backreference in docs between operator and integration (#4671)
- [AIRFLOW-XXX] Improve linking to classes (#4655)
- [AIRFLOW-XXX] Mock optional modules when building docs (#4586)
- [AIRFLOW-XXX] Update plugin macros documentation (#4971)
- [AIRFLOW-XXX] Add missing docstring for 'autodetect' in GCS to BQ Operator (#4979)
- [AIRFLOW-XXX] Add missing GCP operators to Docs (#4260)
- [AIRFLOW-XXX] Fixing the issue in Documentation (#3756)
- [AIRFLOW-XXX] Add Hint at user defined macros (#4885)
- [AIRFLOW-XXX] Correct schedule_interval in Scheduler docs (#4157)
- [AIRFLOW-XXX] Improve airflow-jira script to make RelManager's life easier (#4857)
- [AIRFLOW-XXX] Add missing class references to docs (#4644)
- [AIRFLOW-XXX] Fix typo (#4564)
- [AIRFLOW-XXX] Add a doc about fab security (#4595)
- [AIRFLOW-XXX] Speed up DagBagTest cases (#3974)
- [AIRFLOW-XXX] Reduction of the number of warnings in the documentation (#4585)
Airflow 1.10.2, 2019-01-19
--------------------------
New features
""""""""""""
- [AIRFLOW-2658] Add GCP specific k8s pod operator (#3532)
- [AIRFLOW-2440] Google Cloud SQL import/export operator (#4251)
- [AIRFLOW-3212] Add AwsGlueCatalogPartitionSensor (#4112)
- [AIRFLOW-2750] Add subcommands to delete and list users
- [AIRFLOW-3480] Add GCP Spanner Database Operators (#4353)
- [AIRFLOW-3560] Add DayOfWeek Sensor (#4363)
- [AIRFLOW-3371] BigQueryHook's Ability to Create View (#4213)
- [AIRFLOW-3332] Add method to allow inserting rows into BQ table (#4179)
- [AIRFLOW-3055] add get_dataset and get_datasets_list to bigquery_hook (#3894)
- [AIRFLOW-2887] Added BigQueryCreateEmptyDatasetOperator and create_emty_dataset to bigquery_hook (#3876)
- [AIRFLOW-2758] Add a sensor for MongoDB
- [AIRFLOW-2640] Add Cassandra table sensor
- [AIRFLOW-3398] Google Cloud Spanner instance database query operator (#4314)
- [AIRFLOW-3310] Google Cloud Spanner deploy / delete operators (#4286)
- [AIRFLOW-3406] Implement an Azure CosmosDB operator (#4265)
- [AIRFLOW-3434] Allows creating intermediate dirs in SFTPOperator (#4270)
- [AIRFLOW-3345] Add Google Cloud Storage (GCS) operators for ACL (#4192)
- [AIRFLOW-3266] Add AWS Athena Hook and Operator (#4111)
- [AIRFLOW-3346] Add hook and operator for GCP transfer service (#4189)
- [AIRFLOW-2983] Add prev_ds_nodash and next_ds_nodash macro (#3821)
- [AIRFLOW-3403] Add AWS Athena Sensor (#4244)
- [AIRFLOW-3323] Support HTTP basic authentication for Airflow Flower (#4166)
- [AIRFLOW-3410] Add feature to allow Host Key Change for SSH Op (#4249)
- [AIRFLOW-3275] Add Google Cloud SQL Query operator (#4170)
- [AIRFLOW-2691] Manage JS dependencies via npm
- [AIRFLOW-2795] Oracle to Oracle Transfer Operator (#3639)
- [AIRFLOW-2596] Add Oracle to Azure Datalake Transfer Operator
- [AIRFLOW-3220] Add Instance Group Manager Operators for GCE (#4167)
- [AIRFLOW-2882] Add import and export for pool cli using JSON
- [AIRFLOW-2965] CLI tool to show the next execution datetime (#3834)
- [AIRFLOW-2874] Enables FAB's theme support (#3719)
- [AIRFLOW-3336] Add new TriggerRule for 0 upstream failures (#4182)
Improvements
""""""""""""
- [AIRFLOW-3680] Consistency update in tests for All GCP-related operators (#4493)
- [AIRFLOW-3675] Use googlapiclient for google apis (#4484)
- [AIRFLOW-3205] Support multipart uploads to GCS (#4084)
- [AIRFLOW-2826] Add GoogleCloudKMSHook (#3677)
- [AIRFLOW-3676] Add required permission to CloudSQL export/import example (#4489)
- [AIRFLOW-3679] Added Google Cloud Base Hook to documentation (#4487)
- [AIRFLOW-3594] Unify different License Header
- [AIRFLOW-3197] Remove invalid parameter KeepJobFlowAliveWhenNoSteps in example DAG (#4404)
- [AIRFLOW-3504] Refine the functionality of "/health" endpoint (#4309)
- [AIRFLOW-3103][AIRFLOW-3147] Update flask-appbuilder (#3937)
- [AIRFLOW-3168] More resillient database use in CI (#4014)
- [AIRFLOW-3076] Remove preloading of MySQL testdata (#3911)
- [AIRFLOW-3035] Allow custom 'job_error_states' in dataproc ops (#3884)
- [AIRFLOW-3246] Make hmsclient optional in airflow.hooks.hive_hooks (#4080)
- [AIRFLOW-3059] Log how many rows are read from Postgres (#3905)
- [AIRFLOW-2463] Make task instance context available for hive queries
- [AIRFLOW-3190] Make flake8 compliant (#4035)
- [AIRFLOW-1998] Implemented DatabricksRunNowOperator for jobs/run-now … (#3813)
- [AIRFLOW-2267] Airflow DAG level access (#3197)
- [AIRFLOW-2359] Add set failed for DagRun and task in tree view (#3255)
- [AIRFLOW-3008] Move Kubernetes example DAGs to contrib
- [AIRFLOW-3402] Support global k8s affinity and toleration configs (#4247)
- [AIRFLOW-3610] Add region param for EMR jobflow creation (#4418)
- [AIRFLOW-3531] Fix test for GCS to GCS Transfer Hook (#4452)
- [AIRFLOW-3531] Add gcs to gcs transfer operator. (#4331)
- [AIRFLOW-3034]: Readme updates : Add Slack & Twitter, remove Gitter
- [AIRFLOW-3028] Update Text & Images in Readme.md
- [AIRFLOW-208] Add badge to show supported Python versions (#3839)
- [AIRFLOW-2238] Update PR tool to push directly to Github
- [AIRFLOW-2238] Flake8 fixes on dev/airflow-pr
- [AIRFLOW-2238] Update PR tool to remove outdated info (#3978)
- [AIRFLOW-3005] Replace 'Airbnb Airflow' with 'Apache Airflow' (#3845)
- [AIRFLOW-3150] Make execution_date templated in TriggerDagRunOperator (#4359)
- [AIRFLOW-1196][AIRFLOW-2399] Add templated field in TriggerDagRunOperator (#4228)
- [AIRFLOW-3340] Placeholder support in connections form (#4185)
- [AIRFLOW-3446] Add Google Cloud BigTable operators (#4354)
- [AIRFLOW-1921] Add support for https and user auth (#2879)
- [AIRFLOW-2770] Read `dags_in_image` config value as a boolean (#4319)
- [AIRFLOW-3022] Add volume mount to KubernetesExecutorConfig (#3855)
- [AIRFLOW-2917] Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed (#3766)
- [AIRFLOW-2712] Pass annotations to KubernetesExecutorConfig
- [AIRFLOW-461] Support autodetected schemas in BigQuery run_load (#3880)
- [AIRFLOW-2997] Support cluster fields in bigquery (#3838)
- [AIRFLOW-2916] Arg `verify` for AwsHook() & S3 sensors/operators (#3764)
- [AIRFLOW-491] Add feature to pass extra api configs to BQ Hook (#3733)
- [AIRFLOW-2889] Fix typos detected by github.com/client9/misspell (#3732)
- [AIRFLOW-850] Add a PythonSensor (#4349)
- [AIRFLOW-2747] Explicit re-schedule of sensors (#3596)
- [AIRFLOW-3392] Add index on dag_id in sla_miss table (#4235)
- [AIRFLOW-3001] Add index 'ti_dag_date' to taskinstance (#3885)
- [AIRFLOW-2861] Add index on log table (#3709)
- [AIRFLOW-3518] Performance fixes for topological_sort of Tasks (#4322)
- [AIRFLOW-3521] Fetch more than 50 items in `airflow-jira compare` script (#4300)
- [AIRFLOW-1919] Add option to query for DAG runs given a DAG ID
- [AIRFLOW-3444] Explicitly set transfer operator description. (#4279)
- [AIRFLOW-3411] Add OpenFaaS hook (#4267)
- [AIRFLOW-2785] Add context manager entry points to mongoHook
- [AIRFLOW-2524] Add SageMaker doc to AWS integration section (#4278)
- [AIRFLOW-3479] Keeps records in Log Table when DAG is deleted (#4287)
- [AIRFLOW-2948] Arg check & better doc - SSHOperator & SFTPOperator (#3793)
- [AIRFLOW-2245] Add remote_host of SSH/SFTP operator as templated field (#3765)
- [AIRFLOW-2670] Update SSH Operator's Hook to respect timeout (#3666)
- [AIRFLOW-3380] Add metrics documentation (#4219)
- [AIRFLOW-3361] Log the task_id in the PendingDeprecationWarning from BaseOperator (#4030)
- [AIRFLOW-3213] Create ADLS to GCS operator (#4134)
- [AIRFLOW-3395] added the REST API endpoints to the doc (#4236)
- [AIRFLOW-3294] Update connections form and integration docs (#4129)
- [AIRFLOW-3236] Create AzureDataLakeStorageListOperator (#4094)
- [AIRFLOW-3062] Add Qubole in integration docs (#3946)
- [AIRFLOW-3306] Disable flask-sqlalchemy modification tracking. (#4146)
- [AIRFLOW-2867] Refactor Code to conform standards (#3714)
- [AIRFLOW-2753] Add dataproc_job_id instance var holding actual DP jobId
- [AIRFLOW-3132] Enable specifying auto_remove option for DockerOperator (#3977)
- [AIRFLOW-2731] Raise psutil restriction to <6.0.0
- [AIRFLOW-3384] Allow higher versions of Sqlalchemy and Jinja2 (#4227)
- [Airflow-2760] Decouple DAG parsing loop from scheduler loop (#3873)
- [AIRFLOW-3004] Add config disabling scheduler cron (#3899)
- [AIRFLOW-3175] Fix docstring format in airflow/jobs.py (#4025)
- [AIRFLOW-3589] Visualize reschedule state in all views (#4408)
- [AIRFLOW-2698] Simplify Kerberos code (#3563)
- [AIRFLOW-2499] Dockerise CI pipeline (#3393)
- [AIRFLOW-3432] Add test for feature "Delete DAG in UI" (#4266)
- [AIRFLOW-3301] Update DockerOperator CI test for PR #3977 (#4138)
- [AIRFLOW-3478] Make sure that the session is closed
- [AIRFLOW-3687] Add missing @apply_defaults decorators (#4498)