Code

Record linkage for SEC to EIA#120

Code

Open

katie-lamb wants to merge 173 commits intomaincatalyst-cooperative/mozilla-sec-eia:mainfrom splink-skeletoncatalyst-cooperative/mozilla-sec-eia:splink-skeletonCopy head branch name to clipboard

+14,059-11

Commits

Commits on Aug 13, 2024

• Initial dagster integration

  zschira

  committed
  5767035

  Copy full SHA for 5767035

Commits on Aug 14, 2024

• Update validate integration test to dagster infra

  zschira

  committed
  9d9fbfd

  Copy full SHA for 9d9fbfd

Commits on Aug 20, 2024

• Merge branch 'main' into dagster_integration

  zschira

  committed
  3da9659

  Copy full SHA for 3da9659

Commits on Aug 27, 2024

• Generalize mltools

  zschira

  committed
  ee77e7a

  Copy full SHA for ee77e7a

Commits on Aug 28, 2024

• Reorg repo to move towards generalized modelling repo

  zschira

  committed
  53d3354

  Copy full SHA for 53d3354

• Change library module structure

  zschira

  committed
  014bcb1

  Copy full SHA for 014bcb1

• Create turn experiment_tracking into sub-package

  zschira

  committed
  5404148

  Copy full SHA for 5404148

• Remove unused function

  zschira

  committed
  886614f

  Copy full SHA for 886614f

• Gracefully handle mlflow run on failure

  zschira

  committed
  dec80b8

  Copy full SHA for dec80b8

• Fix variable name

  zschira

  committed
  e725f3d

  Copy full SHA for e725f3d

• Change experiment tracker resource names

  zschira

  committed
  df44ed5

  Copy full SHA for df44ed5

• Add mlflow artifact io-manager

  zschira

  committed
  93da052

  Copy full SHA for 93da052

Commits on Aug 29, 2024

• Simplify pudl_models decorator

  zschira

  committed
  07713e9

  Copy full SHA for 07713e9

• Split extraction logging into two funcs

  zschira

  committed
  5d89ec6

  Copy full SHA for 5d89ec6

• Add mlflow metrics io-manager

  zschira

  committed
  c57818a

  Copy full SHA for c57818a

• Change pudl_model to pudl_pipeline

  zschira

  committed
  625783b

  Copy full SHA for 625783b

Commits on Aug 30, 2024

• Add validation pipeline

  zschira

  committed
  4f50a7b

  Copy full SHA for 4f50a7b

Commits on Sep 2, 2024

• Streamline construction of dagster jobs for running/testing pudl models

  zschira

  committed
  f6ab22c

  Copy full SHA for f6ab22c

• Remove old comment

  zschira

  committed
  f20fb7d

  Copy full SHA for f20fb7d

Commits on Sep 3, 2024

• Add ex21 to dagster jobs

  zschira

  committed
  92e2e00

  Copy full SHA for 92e2e00

• Prep for multiple code locations

  zschira

  committed
  520e6d1

  Copy full SHA for 520e6d1

• Add top-level worksapce file

  zschira

  committed
  e99ee1a

  Copy full SHA for e99ee1a

• Restructure docs

  zschira

  committed
  559c0e6

  Copy full SHA for 559c0e6

• Add train model job

  zschira

  committed
  93d02f3

  Copy full SHA for 93d02f3

• Log mlflow artifacts as parquet until csv is fixed

  zschira

  committed
  5190bf9

  Copy full SHA for 5190bf9

Commits on Sep 4, 2024

• Fix ex21 extraction

  zschira

  committed
  ca9599e

  Copy full SHA for ca9599e

• Add development section to docs

  zschira

  committed
  7e7a503

  Copy full SHA for 7e7a503

• Fix integration tests

  zschira

  committed
  61f48c3

  Copy full SHA for 61f48c3

• Don't run ruff on notebooks

  zschira

  committed
  0fd8ffc

  Copy full SHA for 0fd8ffc

• xfail ex21 integration test

  zschira

  committed
  97d5587

  Copy full SHA for 97d5587

Commits on Sep 5, 2024

• Add parquet upath io-manager

  zschira

  committed
  ace268b

  Copy full SHA for ace268b

• Remove nb-output clear

  zschira

  committed
  fb1feeb

  Copy full SHA for fb1feeb

• Test docker deployment

  zschira

  committed
  294ec72

  Copy full SHA for 294ec72

• Chunk ex 21 extraction

  zschira

  committed
  4de51b3

  Copy full SHA for 4de51b3

Commits on Sep 6, 2024

• Fix asign copy

  zschira

  committed
  214e28f

  Copy full SHA for 214e28f

• Add job for testing ex21 resource usage

  zschira

  committed
  c5736e0

  Copy full SHA for c5736e0

• Merge branch 'test_parquet_logging' into dagster_integration

  zschira

  committed
  4a81e88

  Copy full SHA for 4a81e88

• Remove test docker files

  zschira

  committed
  ec39633

  Copy full SHA for ec39633

• Remove complex asset factory

  zschira

  committed
  101ccf1

  Copy full SHA for 101ccf1

• Parallelize ex21 extraction

  zschira

  committed
  7e0c5a5

  Copy full SHA for 7e0c5a5

• Don't chunk in inference module

  zschira

  committed
  080d790

  Copy full SHA for 080d790

• Handle failures in converting to pdf

  zschira

  committed
  44dfc52

  Copy full SHA for 44dfc52

• Delete cached pdfs early

  zschira

  committed
  6e24157

  Copy full SHA for 6e24157

Commits on Sep 9, 2024

• Add metadata to chunk_filings

  zschira

  committed
  cd06d07

  Copy full SHA for cd06d07

• Catch oom errors while extracting ex21

  zschira

  committed
  e3e8c45

  Copy full SHA for e3e8c45

• Fix ex21 gcs io-manager

  zschira

  committed
  350defb

  Copy full SHA for 350defb

• Fix partitions for basic 10k extraction.

  zschira

  committed
  3c80b72

  Copy full SHA for 3c80b72

• Cache layoutlm locally

  zschira

  committed
  31971b7

  Copy full SHA for 31971b7

• Fix caching model

  zschira

  committed
  634a050

  Copy full SHA for 634a050

• Remove bad call

  zschira

  committed
  69ee4c0

  Copy full SHA for 69ee4c0

Commits on Sep 10, 2024

• Test own_per conversion

  zschira

  committed
  63d6600

  Copy full SHA for 63d6600

• Add pandera types for output tables

  zschira

  committed
  c8490d4

  Copy full SHA for c8490d4

• Add missing entities module

  zschira

  committed
  fa4f57d

  Copy full SHA for fa4f57d

• Don't cache model, load with io manager

  zschira

  committed
  35e917d

  Copy full SHA for 35e917d

• Remove float conversion

  zschira

  committed
  a7b1c7f

  Copy full SHA for a7b1c7f

• Add hypothesis to deps

  zschira

  committed
  f019117

  Copy full SHA for f019117

• Make own_per str

  zschira

  committed
  d7d13d8

  Copy full SHA for d7d13d8

• Remove astype

  zschira

  committed
  70f5293

  Copy full SHA for 70f5293

• Validate ex21 return types

  zschira

  committed
  e406092

  Copy full SHA for e406092

Commits on Sep 11, 2024

• Clean model download temp dir

  zschira

  committed
  f3835d9

  Copy full SHA for f3835d9

• Fix model return type

  zschira

  committed
  3c995cd

  Copy full SHA for 3c995cd

• Catch errors in creating ex 21 dataset

  zschira

  committed
  ef55e4b

  Copy full SHA for ef55e4b

• Fix column name

  zschira

  committed
  b37450a

  Copy full SHA for b37450a

Commits on Sep 12, 2024

• Try to catch empty pdf errors

  zschira

  committed
  06b18ed

  Copy full SHA for 06b18ed

• Print traceback in caught exception

  zschira

  committed
  abfc006

  Copy full SHA for abfc006

• Fix empty pdf check

  zschira

  committed
  ff92a55

  Copy full SHA for ff92a55

• Actually fix empty pdf check?

  zschira

  committed
  8aa8c95

  Copy full SHA for 8aa8c95

Commits on Sep 18, 2024

• Use UPath in GCSArchive

  zschira

  committed
  43600bc

  Copy full SHA for 43600bc

• Make _configure_mlflow a standalone function

  zschira

  committed
  05ad82c

  Copy full SHA for 05ad82c

• Merge branch 'main' into error_handling_improvements

  zschira

  committed
  fddc3b2

  Copy full SHA for fddc3b2

• Try to skip notebooks in ruff check

  zschira

  committed
  99fc7ed

  Copy full SHA for 99fc7ed

Commits on Sep 19, 2024

• Pull integration test fixes from main

  zschira

  committed
  b135500

  Copy full SHA for b135500

• Fix typos in README.rst

  zschira

  authored
  6e868f2

  Copy full SHA for 6e868f2

• Cache downloaded layoutlm in dagster home

  zschira

  committed
  df4fd09

  Copy full SHA for df4fd09

• Merge branch 'error_handling_improvements' of github.com:catalyst-cooperative/mozilla-sec-eia into error_handling_improvements

  zschira

  committed
  74d237d

  Copy full SHA for 74d237d

• Fix broken test

  zschira

  committed
  3642765

  Copy full SHA for 3642765

Commits on Sep 20, 2024

• fix rename filings

  katie-lamb

  committed
  830bd74

  Copy full SHA for 830bd74

• fix paths to cache training data

  katie-lamb

  committed
  2cd1fe6

  Copy full SHA for 2cd1fe6

• update root dir path

  katie-lamb

  committed
  64dc8c5

  Copy full SHA for 64dc8c5

• Fix UPath initialization

  zschira

  committed
  226d91c

  Copy full SHA for 226d91c

• Fix path in test

  zschira

  committed
  3c17d33

  Copy full SHA for 3c17d33

• Create huggingface dataset outside model execution

  zschira

  committed
  df69f42

  Copy full SHA for df69f42

• small fixes to path handling

  katie-lamb

  committed
  2d3345c

  Copy full SHA for 2d3345c

• Merge branch 'error_handling_improvements' into second-pass-ex21-improvements

  katie-lamb

  committed
  46e7b40

  Copy full SHA for 46e7b40

Commits on Sep 23, 2024

• Minor fixes

  zschira

  committed
  6f9d34a

  Copy full SHA for 6f9d34a

• Start migrating model training to notebook

  zschira

  committed
  07d500a

  Copy full SHA for 07d500a

Commits on Sep 24, 2024

• Create dataset as dataframe for logging

  zschira

  committed
  81813a7

  Copy full SHA for 81813a7

• Modify dataset return type

  zschira

  committed
  5174ed7

  Copy full SHA for 5174ed7

• Fix dataset types for model signature

  zschira

  committed
  7a572c0

  Copy full SHA for 7a572c0

Commits on Sep 25, 2024

• Migrate ex 21 model training to a notebook

  zschira

  committed
  5728026

  Copy full SHA for 5728026

Commits on Oct 3, 2024

• Merge initial notebook migration (broken)

  zschira

  committed
  5fbbfff

  Copy full SHA for 5fbbfff

Commits on Oct 4, 2024

• Split dataset loading into separate assets

  zschira

  committed
  37edd50

  Copy full SHA for 37edd50

• Minor notebook fixes

  zschira

  committed
  d6889e3

  Copy full SHA for d6889e3

• Fix import in notebook

  zschira

  committed
  d5e013a

  Copy full SHA for d5e013a

• add device to pipeline

  zschira

  committed
  f9810db

  Copy full SHA for f9810db

• Fix signature inference

  zschira

  committed
  2760881

  Copy full SHA for 2760881

• Fix notebook dagster config

  zschira

  committed
  1dcacfa

  Copy full SHA for 1dcacfa

• Fix config param name

  zschira

  committed
  39bb45b

  Copy full SHA for 39bb45b

Commits on Oct 5, 2024

• Partition training data

  zschira

  committed
  cb83862

  Copy full SHA for cb83862

• Add partitions to notebook asset

  zschira

  committed
  c71593c

  Copy full SHA for c71593c

Commits on Oct 6, 2024

• Update ex21 labels

  zschira

  committed
  4efa515

  Copy full SHA for 4efa515

• Use run name for specifying training runs

  zschira

  committed
  581b2e3

  Copy full SHA for 581b2e3

• Rework how notebook is configured

  zschira

  committed
  c67a1be

  Copy full SHA for c67a1be

• Finetune configuration

  zschira

  committed
  b8a5b24

  Copy full SHA for b8a5b24

Commits on Oct 7, 2024

• separate inference dataset creation from model prediction

  zschira

  committed
  45d5cf8

  Copy full SHA for 45d5cf8

• Remove deprecated inference module

  zschira

  committed
  3e15b1f

  Copy full SHA for 3e15b1f

Commits on Oct 8, 2024

• Add notebook for training ex21 classifier

  zschira

  committed
  60a1260

  Copy full SHA for 60a1260

• Pull in model updates

  zschira

  committed
  4105110

  Copy full SHA for 4105110

• Update classifier model

  zschira

  committed
  4d29037

  Copy full SHA for 4d29037

Commits on Oct 9, 2024

• Fix set on copy pandas issue

  zschira

  committed
  85c44ff

  Copy full SHA for 85c44ff

• Fix model uri's

  zschira

  committed
  52e3580

  Copy full SHA for 52e3580

• Fix indices in extraction model

  zschira

  committed
  b709053

  Copy full SHA for b709053

• Fix typo

  zschira

  committed
  b8dad3c

  Copy full SHA for b8dad3c

Commits on Oct 10, 2024

• Add asset factory for loading models

  zschira

  committed
  e6b29ff

  Copy full SHA for e6b29ff

• Catch layout classification NaN exception

  zschira

  committed
  3d11777

  Copy full SHA for 3d11777

• Use GCS pickle io-manager

  zschira

  committed
  df5fe0d

  Copy full SHA for df5fe0d

Commits on Oct 11, 2024

• Switch gcs pickle io manager to upath based

  zschira

  committed
  d6c41a2

  Copy full SHA for d6c41a2

• Remove duplicate logger

  zschira

  committed
  ddd2263

  Copy full SHA for ddd2263

• Fix config warnings

  zschira

  committed
  93bffcb

  Copy full SHA for 93bffcb

• Test pin sphinx

  zschira

  committed
  d717caa

  Copy full SHA for d717caa

Commits on Oct 13, 2024

• add splink and model to environment

  katie-lamb

  committed
  09cd189

  Copy full SHA for 09cd189

Commits on Oct 14, 2024

• Catch errors while normalizing bounding boxes

  zschira

  committed
  15be127

  Copy full SHA for 15be127

• Fix call to pandera example

  zschira

  committed
  4117d0a

  Copy full SHA for 4117d0a

• Fix handle failures in converting to pdf

  zschira

  committed
  8c8dd60

  Copy full SHA for 8c8dd60

• Actually fix handle failures in converting to pdf

  zschira

  committed
  ff821b5

  Copy full SHA for ff821b5

Commits on Oct 16, 2024

• Add model documentation to sec10k readme

  zschira

  committed
  a8eb359

  Copy full SHA for a8eb359

• Fix ex 21 validation integration test

  zschira

  committed
  dc160ac

  Copy full SHA for dc160ac

• Improve classifier error handling

  zschira

  committed
  10b24a9

  Copy full SHA for 10b24a9

• Fully broaden classifier errors

  zschira

  committed
  ad54979

  Copy full SHA for ad54979

• add more docs on running the notebooks

  zschira

  committed
  672e123

  Copy full SHA for 672e123

Commits on Oct 21, 2024

• add splink notebooks and preprocessing functions

  katie-lamb

  committed
  b95b2fb

  Copy full SHA for b95b2fb

Commits on Oct 22, 2024

• clean up feature creation in paragraph classifier

  katie-lamb

  committed
  2dbdcaa

  Copy full SHA for 2dbdcaa

• fix feature creation function

  katie-lamb

  committed
  cda3225

  Copy full SHA for cda3225

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 509b7a0
  pre-commit-ci[bot]

  committed
  509b7a0

  Copy full SHA for 509b7a0

• small fixes to read in comments in tracking dataframe

  katie-lamb

  committed
  8855e5e

  Copy full SHA for 8855e5e

• Merge branch 'prep_paragraph_classifier' of https://github.com/catalyst-cooperative/mozilla-sec-eia into prep_paragraph_classifier

  katie-lamb

  committed
  f806ad8

  Copy full SHA for f806ad8

Commits on Oct 23, 2024

• updates to model pipeline

  katie-lamb

  committed
  590ba60

  Copy full SHA for 590ba60

• take out logging messages

  katie-lamb

  committed
  3db47d4

  Copy full SHA for 3db47d4

• Merge branch 'prep_paragraph_classifier' into splink-skeleton

  katie-lamb

  committed
  858ecff

  Copy full SHA for 858ecff

• make pudl editable

  katie-lamb

  committed
  61c8abf

  Copy full SHA for 61c8abf

Commits on Nov 27, 2024

• add in record linkage modules

  katie-lamb

  committed
  e5148d8

  Copy full SHA for e5148d8

Commits on Nov 29, 2024

• fix errors with asset creation

  katie-lamb

  committed
  88f17f2

  Copy full SHA for 88f17f2

Commits on Nov 30, 2024

• clean up sec output table creation

  katie-lamb

  committed
  c9b62ba

  Copy full SHA for c9b62ba

• splink notebook change

  katie-lamb

  committed
  01b2d23

  Copy full SHA for 01b2d23

• Merge branch 'main' into splink-skeleton

  katie-lamb

  committed
  5d9419b

  Copy full SHA for 5d9419b

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 30d22c9
  pre-commit-ci[bot]

  committed
  30d22c9

  Copy full SHA for 30d22c9

• fix pre commit

  katie-lamb

  committed
  3fca3c9

  Copy full SHA for 3fca3c9

• Merge branch 'splink-skeleton' of https://github.com/catalyst-cooperative/mozilla-sec-eia into splink-skeleton

  katie-lamb

  committed
  a50dd9d

  Copy full SHA for a50dd9d

Commits on Dec 1, 2024

• update python dependency in test environment

  katie-lamb

  committed
  d53ab25

  Copy full SHA for d53ab25

• update github tox env

  katie-lamb

  committed
  2eb1555

  Copy full SHA for 2eb1555

Commits on Dec 2, 2024

• restructure input table assets

  katie-lamb

  committed
  44eb70b

  Copy full SHA for 44eb70b

• include pseudo code of SEC output table module

  katie-lamb

  committed
  7563df8

  Copy full SHA for 7563df8

Commits on Dec 3, 2024

• Try using conda env to run tox

  jdangerx

  committed
  3c88ff2

  Copy full SHA for 3c88ff2

• Add PUDL dependency and restrict to Py3.12

  jdangerx

  committed
  fb3d772

  Copy full SHA for fb3d772

• Guess you don't need to specify tox env with setup-micromamba

  jdangerx

  committed
  390770f

  Copy full SHA for 390770f

• Install GDAL version via conda since we rely on PUDL now

  jdangerx

  committed
  8e8beee

  Copy full SHA for 8e8beee

• notebook has cells for SEC and EIA hook up

  katie-lamb

  committed
  50e7b7e

  Copy full SHA for 50e7b7e

Commits on Dec 4, 2024

• Merge pull request #123 from catalyst-cooperative/splink-skeleton-tox-fix

  Show description for 224721b
  jdangerx

  authored
  224721b

  Copy full SHA for 224721b

• Fix dagster setup for record linkage inputs

  zschira

  committed
  7dc78e1

  Copy full SHA for 7dc78e1

Commits on Dec 9, 2024

• fix util functions

  katie-lamb

  committed
  daa8f0a

  Copy full SHA for daa8f0a

Commits on Dec 10, 2024

• Handle missing partitions in extracted data

  zschira

  committed
  dbefe34

  Copy full SHA for dbefe34

Commits on Dec 11, 2024

• Fix basic_10k partitions

  zschira

  committed
  97f5d68

  Copy full SHA for 97f5d68

• debug materialization of rl input assets

  katie-lamb

  committed
  b26f1f8

  Copy full SHA for b26f1f8

Commits on Dec 16, 2024

• clean up notebook to work with dagster assets

  katie-lamb

  committed
  acaf3d1

  Copy full SHA for acaf3d1

Commits on Dec 17, 2024

• clean up new structure of sec assets

  katie-lamb

  committed
  f4cceb7

  Copy full SHA for f4cceb7

Commits on Dec 18, 2024

• add in final match between ex 21 subs and eia utilities

  katie-lamb

  committed
  fa9e52e

  Copy full SHA for fa9e52e

• remove sec output table module

  katie-lamb

  committed
  599ae87

  Copy full SHA for 599ae87

• add drop duplicates on sec company id

  katie-lamb

  committed
  c340718

  Copy full SHA for c340718

Commits on Dec 19, 2024

• clean up notbook

  katie-lamb

  committed
  24de7d6

  Copy full SHA for 24de7d6

• add markdown cell note

  katie-lamb

  committed
  26b1a72

  Copy full SHA for 26b1a72

• make asset not multi asset

  katie-lamb

  committed
  70427a0

  Copy full SHA for 70427a0

Commits on Jan 7, 2025

• Fix asset keywords

  zschira

  committed
  136709d

  Copy full SHA for 136709d

Commits on Jan 16, 2025

• splink notebook update

  katie-lamb

  committed
  4da70e0

  Copy full SHA for 4da70e0