PUDL v2024.02.05

What's Changed

New Data

• Extract EIA923 emissions control table and add 2022 final release dat… by @aesharpe in #3100
• Update CEMS partitions to handle year-quarter files by @e-belfer in #3096
• Start EIA-176 pipelines: company data by @davidmudrauskas in #3227
• WIP: Extract raw PHMSA transmission data (A-D, H, I) by @e-belfer in #3235
• Update EIA bulk electricity data archive. by @zaneselvans in #3249
• Extract raw PHMSA distribution and start of transmission data (Table A-D, H, I) by @e-belfer in #2932
• Extract raw tables for PHMSA transmission data Part F & G by @e-belfer in #3242
• Map PHMSA Natural Gas Transmission Part R columns by @e-belfer in #3269
• Map PHMSA Natural Gas Transmission Part S columns by @e-belfer in #3262
• Map PHMSA Natural Gas Transmission Part T columns by @e-belfer in #3267
• Map PHMSA Natural Gas Transmission Part N-O columns by @e-belfer in #3260
• 3243 PHMSA transmission part J by @jdangerx in #3266
• Map PHMSA Natural Gas Transmission Part L columns by @e-belfer in #3254
• Map PHMSA Natural Gas Transmission Part M columns by @cmgosnell in #3270
• Map PHMSA Natural Gas Transmission Part Q columns by @cmgosnell in #3280
• PHMSA: fix to Part Q - fix the column names for the other materials (NOT MILES!) by @cmgosnell in #3291
• PHMSA transmission part P by @jdangerx in #3279
• Clean up and standardize PHMSA raw assets by @e-belfer in #3295
• Add EIA Forms 191 and 757 to sources in PUDL metadata by @e-belfer in #3304

Other Changes

• Sort DBs Fly Datasette; don't distribute Datasette's metadata.yml by @zaneselvans in #3106
• Add support for choosing between multiprocess and inprocess executors via cli flag by @rousik in #2895
• Improve flexibility for publishing options by @rousik in #2964
• Hide diffs in lock files by default. by @jdangerx in #3103
• Clean up PUDL CLI tools; use Click framework by @zaneselvans in #3086
• Create new issue template for adding a new year of data by @aesharpe in #3089
• Reorganize contributing docs + add process description. by @jdangerx in #3044
• add data corrections for "sizable minority" utilities by @cmgosnell in #3078
• Skip slow tests in pre-commit hook by @jdangerx in #3132
• Replace ferc714 @multi_asset with asset factory by @rousik in #3123
• WIP Transition CEMS paritions to year_quarter from year and quarter by @cmgosnell in #3139
• Table diff tools by @jdangerx in #3128
• Refactor calculation of annualized_respondents_ferc714 by @rousik in #3024
• Include sub-annual updates in annual_updates docs by @aesharpe in #3129
• Script to sync a local directory up to a Zenodo DOI by @zaneselvans in #3127
• Attempt to create sandbox data release in nightly builds. by @zaneselvans in #3158
• Alter output path for the nightly builds by @rousik in #3157
• Add EPA CEMS concurrency limit to pudl_etl by @bendnorman in #3160
• Remove obsolete Docker data access instructions. by @zaneselvans in #3156
• Feature branch: Rename core + output assets to match new naming protocols by @e-belfer in #2818
• Knowledge contribution docs by @jdangerx in #3151
• Update EPA CEMS docs for quarterly data, new table name. by @zaneselvans in #3178
• Improve nightly build and deployment logic by @zaneselvans in #3164
• Clean up nightly build/deploy w/o nightly branch update. by @zaneselvans in #3188
• Update FERC rate base tags with RMI guidance by @cmgosnell in #3162
• update the 860m doi by @cmgosnell in #3189
• Update ferc-ferc plant matching with ccai implementation. by @zschira in #3007
• Tell setuptools_scm to ignore non-version tags by @zaneselvans in #3193
• Clarify consistency check and lower threshold from 75% to 74% by @zaneselvans in #3194
• Reduce FERC1 match threshold in test to 85% by @zaneselvans in #3197
• Update nightly branch after successful build by @zaneselvans in #3195
• Use digest instead of tag so VM uses right image. by @jdangerx in #3206
• It's german. It means "the dev, the" by @zaneselvans in #3212
• Switch from dev to main/nightly/stable branch structure. by @zaneselvans in #3216
• Suppress excessive numba logs by @zschira in #3217
• Make CEMS extraction handle new listed year_quarter partitions by @e-belfer in #3187
• Tweak metadata and pyarrow schema methods to work for all tables by @zaneselvans in #3222
• Add --build-only flag to datasette deploy script. by @jdangerx in #3231
• Generalize passing args through to flyctl deploy by @zaneselvans in #3236
• Create directory for local only notebooks by @bendnorman in #3230
• fill in the annual columns with quarterly balances by @cmgosnell in #3234
• FERC to EIA Entity Matching Refactor by @katie-lamb in #3184
• Apply new naming conventions to devtool notebooks by @bendnorman in #3228
• Update gas price upper bound; Enable null-row check for MCOE. by @zaneselvans in #3252
• Tiny fix to make FERC to EIA tests xfail by @katie-lamb in #3257
• PHMSA gas extract step for transmission part k by @cmgosnell in #3258
• Fix dataframe embedder op factory and tune ferc-ferc model by @zschira in #3247
• Nightly build quality of life improvements by @jdangerx in #3287
• Push to prod Zenodo sometimes, fix pypi release flow, update docs by @jdangerx in #3292
• Add issue template for nightly build by @jdangerx in #3286
• Parameterize the FERC1 transform step what transfers quarterly filed data into annual columns by @cmgosnell in #3300
• Add retries, and use fsspec to handle GCS by @jdangerx in #3299
• Use Google Batch for full ETL runs by @jdangerx in #3211
• Hide _out tables and restore metadata in Datasette by @bendnorman in #3226
• Update build ref by @jdangerx in #3321
• Rename some straggler assets by @bendnorman in #3294
• Resource field description cleanup by @aesharpe in #3283
• Fix nightly builds 2024-01-31 by @zaneselvans in #3329
• Trigger CI on merge group; only trigger integration on merge group. by @jdangerx in #3332
• Fix nightly build failure 2024-02-01 by @zaneselvans in #3334
• ensure all the corrections get tags and add the begining of a rate base asset by @cmgosnell in #3214
• Update release notes for PUDL v2024.02.03 by @zaneselvans in #3342
• Fix v2024.02.03 build issues by @zaneselvans in #3344
• Fix (more) v2024.02.03 release issues by @zaneselvans in #3346

New Contributors

• @davidmudrauskas made their first contribution in #3227

Full Changelog: v2023.12.01...v2024.02.05