Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FERC 714: transform of hourly demand table (dbf +xbrl) #3842

Merged
merged 47 commits into from
Sep 26, 2024
Merged

FERC 714: transform of hourly demand table (dbf +xbrl) #3842

Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
7d3b11d
first very wip draft of transofmring the hourly 714 table
cmgosnell Sep 12, 2024
f48054b
early processing of datetimes and initial cleaning of timezone codes
cmgosnell Sep 13, 2024
ea25923
lil function suffix cleanup
cmgosnell Sep 16, 2024
b1d5c0a
group the table-specific transforms into staticmethods of a table tra…
cmgosnell Sep 17, 2024
f5bfc62
yay add the hour into the csv report_date early so i'm not oopsies lo…
cmgosnell Sep 17, 2024
b0d1219
lil extra doc clean
cmgosnell Sep 17, 2024
f27b04b
Merge branch 'main' into transform-714-xbrl
cmgosnell Sep 17, 2024
c1e66af
Map FERC 714 XBRL and CSV IDs (#3849)
e-belfer Sep 17, 2024
05408d5
Preliminary fixes to the 714 data source page
aesharpe Sep 17, 2024
0054b4d
integrate the respondent_id_ferc714 map into transforms
cmgosnell Sep 17, 2024
6936a38
Merge branch 'main' into transform-714-xbrl
cmgosnell Sep 18, 2024
1986b80
Add notes on CSV-XBRL ID linkage to docs
e-belfer Sep 18, 2024
46ba25e
Merge branch 'transform-714-xbrl' into ferc714-data-source
e-belfer Sep 18, 2024
780ebb7
Write preliminary transform class and function for XBRL and CSV core_…
aesharpe Sep 19, 2024
978e664
wip first round of respondent table transforming
cmgosnell Sep 19, 2024
30d34d8
Combine XBRL and CSV tables
aesharpe Sep 19, 2024
bbd6e9a
Add forecast to forecast column names
aesharpe Sep 19, 2024
37a0643
Add migration file for new forecast cols
aesharpe Sep 19, 2024
42716aa
finish eia_code mapping and wrap up transforms
cmgosnell Sep 20, 2024
858b744
udpate docs
cmgosnell Sep 20, 2024
549a5ab
udpate docs again lol spaces
cmgosnell Sep 20, 2024
afc60ac
fix forcast to forecast type and add to run() docstring
aesharpe Sep 20, 2024
75e29ae
convert :meth: to :func:
aesharpe Sep 20, 2024
e4c3b95
lower expected forecast year range
aesharpe Sep 20, 2024
cfc9176
fix docs typo
aesharpe Sep 20, 2024
4a3c2bf
Use split/apply/combine for deduping and update assertion
aesharpe Sep 20, 2024
a459344
responding to pr comments mostly doc updates
cmgosnell Sep 23, 2024
956af05
Add new years to Ferc714CheckSpec
aesharpe Sep 23, 2024
6bb79de
update docs
cmgosnell Sep 23, 2024
5cb6492
Merge pull request #3856 from catalyst-cooperative/transform-714-year…
aesharpe Sep 23, 2024
10692cb
Merge pull request #3857 from catalyst-cooperative/transform-714-xbrl…
cmgosnell Sep 23, 2024
94752b5
first pass of adding respondend id tables
cmgosnell Sep 23, 2024
df95744
Merge branch 'transform-714-xbrl' into ferc714-data-source
aesharpe Sep 23, 2024
48e9ff0
add alembic migration for the glue tables
cmgosnell Sep 23, 2024
33085e6
remove the lil post process step
cmgosnell Sep 23, 2024
66b908d
Light edits
aesharpe Sep 23, 2024
25fb6dd
Merge branch 'ferc714-data-source' of https://github.com/catalyst-coo…
aesharpe Sep 23, 2024
74e8fe0
release notes and metadata updates
cmgosnell Sep 23, 2024
0c811e6
Merge pull request #3850 from catalyst-cooperative/ferc714-data-source
aesharpe Sep 23, 2024
9b938c3
Add table description for annual forecast table and fix indentation e…
aesharpe Sep 24, 2024
91c8859
update docs and metadata, plus stop trying to impute midnight jan 1st…
cmgosnell Sep 24, 2024
b101086
Merge branch 'main' into transform-714-xbrl
cmgosnell Sep 24, 2024
d4d93fc
update the validation test expectations for the analysis downstream s…
cmgosnell Sep 24, 2024
708a413
update the settinggggsss omigosh plus restrict the imputations based …
cmgosnell Sep 25, 2024
706c7df
add module-level design notes
cmgosnell Sep 25, 2024
551325d
add move color to the fast test 12 assertion
cmgosnell Sep 25, 2024
20959c8
remove the lil context thing that is no longer necessary
cmgosnell Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/pudl/transform/ferc714.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -805,7 +805,7 @@ def run(
process the combined datasets.

The main transforms include spot-fixing forecast years with
:meth:`spot_fix_forecast_years_xbrl`and averaging out duplicate forecast values
:func:`spot_fix_forecast_years_xbrl` and averaging out duplicate forecast values
cmgosnell marked this conversation as resolved.
Show resolved Hide resolved
for duplicate primary key rows in the CSV table.

"""
Expand Down
Loading