Code

Upgrade report data pipelines#30

Code

Merged

max-ostapenko merged 81 commits intomainHTTPArchive/dataform:mainfrom reportsHTTPArchive/dataform:reportsCopy head branch name to clipboardDec 9, 2024

+4,113-201

Commits

Commits on Nov 16, 2024

• demo report

  max-ostapenko

  committed
  48a3b35

  Copy full SHA for 48a3b35

• fix local package

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  f752d9f

  Copy full SHA for f752d9f

• crawl reports tag triggered

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  172e120

  Copy full SHA for 172e120

Commits on Nov 17, 2024

• Merge branch 'reports' of https://github.com/HTTPArchive/dataform into reports

  max-ostapenko

  committed
  39ae950

  Copy full SHA for 39ae950

• timeseries added

  max-ostapenko

  committed
  eb76476

  Copy full SHA for eb76476

• split tables

  max-ostapenko

  committed
  4a2d145

  Copy full SHA for 4a2d145

• lint

  max-ostapenko

  committed
  a8e2137

  Copy full SHA for a8e2137

Commits on Nov 20, 2024

• tech report tables

  max-ostapenko

  committed
  4250677

  Copy full SHA for 4250677

Commits on Nov 21, 2024

• check tech report sql

  max-ostapenko

  committed
  607c6b2

  Copy full SHA for 607c6b2

• Merge branch 'main' into main

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  aba1af4

  Copy full SHA for aba1af4

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  791c0e8

  Copy full SHA for 791c0e8

• missing declaration

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  3fff267

  Copy full SHA for 3fff267

• formatting

  max-ostapenko

  committed
  3be0274

  Copy full SHA for 3be0274

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  4c361a9

  Copy full SHA for 4c361a9

• preOps

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  b804540

  Copy full SHA for b804540

Commits on Nov 23, 2024

• dataset change

  max-ostapenko

  committed
  cc04bb6

  Copy full SHA for cc04bb6

• cwv_tech_report tested

  max-ostapenko

  committed
  9c8e567

  Copy full SHA for 9c8e567

• Merge branch 'main' into main

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  45f1095

  Copy full SHA for 45f1095

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  e38d4f0

  Copy full SHA for e38d4f0

Commits on Nov 25, 2024

• tech_reports moved

  max-ostapenko

  committed
  1185758

  Copy full SHA for 1185758

• exporter function draft

  max-ostapenko

  committed
  9591bf0

  Copy full SHA for 9591bf0

• fix depependencies

  max-ostapenko

  committed
  4acdf05

  Copy full SHA for 4acdf05

• rename

  max-ostapenko

  committed
  9a1f13e

  Copy full SHA for 9a1f13e

• dataset renamed

  max-ostapenko

  committed
  b2cd6b4

  Copy full SHA for b2cd6b4

• storage exp draft

  max-ostapenko

  committed
  02d1db7

  Copy full SHA for 02d1db7

• date column for histograms

  max-ostapenko

  committed
  e090ebc

  Copy full SHA for e090ebc

• dev flag

  max-ostapenko

  committed
  256fd88

  Copy full SHA for 256fd88

• gsc export tested

  max-ostapenko

  committed
  c3f75d2

  Copy full SHA for c3f75d2

Commits on Nov 26, 2024

• pubsub sink prepared

  max-ostapenko

  committed
  6ae6e72

  Copy full SHA for 6ae6e72

• export fn deployed

  max-ostapenko

  committed
  537aa60

  Copy full SHA for 537aa60

• order incompatible with partitions

  max-ostapenko

  committed
  b5b625b

  Copy full SHA for b5b625b

• monitoring

  max-ostapenko

  committed
  486ec2e

  Copy full SHA for 486ec2e

• lint

  max-ostapenko

  committed
  3fbf2bb

  Copy full SHA for 3fbf2bb

• event parsing draft

  max-ostapenko

  committed
  c34c57a

  Copy full SHA for c34c57a

• cleanup before inserts

  max-ostapenko

  committed
  1db9ff6

  Copy full SHA for 1db9ff6

• event parsing

  max-ostapenko

  committed
  f8bc51a

  Copy full SHA for f8bc51a

• partitioned exports

  max-ostapenko

  committed
  08d9fa6

  Copy full SHA for 08d9fa6

• exclude scripts

  max-ostapenko

  committed
  1a2188d

  Copy full SHA for 1a2188d

Commits on Nov 27, 2024

• firestore export draft

  max-ostapenko

  committed
  0e11edf

  Copy full SHA for 0e11edf

• Merge branch 'main' into reports

  max-ostapenko

  authored
  65d310d

  Copy full SHA for 65d310d

Commits on Nov 28, 2024

• optional description

  max-ostapenko

  committed
  d46b68e

  Copy full SHA for d46b68e

Commits on Dec 2, 2024

• single dataset

  max-ostapenko

  committed
  8d316ce

  Copy full SHA for 8d316ce

• move

  max-ostapenko

  committed
  941e157

  Copy full SHA for 941e157

• incremental operations

  max-ostapenko

  committed
  4b34849

  Copy full SHA for 4b34849

• docs update

  max-ostapenko

  committed
  dd38945

  Copy full SHA for dd38945

Commits on Dec 3, 2024

• firestore dict tested

  max-ostapenko

  committed
  d64a316

  Copy full SHA for d64a316

• reports tested

  max-ostapenko

  committed
  73c3100

  Copy full SHA for 73c3100

• full sql export

  max-ostapenko

  committed
  d786036

  Copy full SHA for d786036

• trigger params

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  3d6657d

  Copy full SHA for 3d6657d

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  b4dd900

  Copy full SHA for b4dd900

• hashed doc ids

  max-ostapenko

  committed
  76b3d5f

  Copy full SHA for 76b3d5f

• more resources and timeout

  max-ostapenko

  committed
  0819a7b

  Copy full SHA for 0819a7b

• extend timeout

  max-ostapenko

  committed
  eee1311

  Copy full SHA for eee1311

Commits on Dec 5, 2024

• gzip

  max-ostapenko

  committed
  9b317ca

  Copy full SHA for 9b317ca

• event example

  max-ostapenko

  committed
  33df626

  Copy full SHA for 33df626

Commits on Dec 6, 2024

• esm

  max-ostapenko

  committed
  58b6c3c

  Copy full SHA for 58b6c3c

• more parallelization improvements

  max-ostapenko

  committed
  da4718c

  Copy full SHA for da4718c

• Merge branch 'main' into main

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  8e042a3

  Copy full SHA for 8e042a3

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  1a1caa6

  Copy full SHA for 1a1caa6

Commits on Dec 8, 2024

• tested batch reports

  max-ostapenko

  committed
  7baa4df

  Copy full SHA for 7baa4df

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  687750a

  Copy full SHA for 687750a

• testing fast deletion

  max-ostapenko

  committed
  d61666b

  Copy full SHA for d61666b

• deletion tested

  max-ostapenko

  committed
  ab581df

  Copy full SHA for ab581df

• limit concurrency

  max-ostapenko

  committed
  2f5bed6

  Copy full SHA for 2f5bed6

• retries

  max-ostapenko

  committed
  a78999b

  Copy full SHA for a78999b

• wait to resolve

  max-ostapenko

  committed
  85a5690

  Copy full SHA for 85a5690

Commits on Dec 9, 2024

• tested deployed version

  max-ostapenko

  committed
  2d116dd

  Copy full SHA for 2d116dd

• cleanup for test merge

  max-ostapenko

  committed
  9fd868a

  Copy full SHA for 9fd868a

• cwv-tech-report to prod db

  max-ostapenko

  committed
  e859d29

  Copy full SHA for e859d29

• note to unwrap pubsub payloads

  max-ostapenko

  committed
  0c81fb2

  Copy full SHA for 0c81fb2

• cleanup

  max-ostapenko

  committed
  dbe38a1

  Copy full SHA for dbe38a1

• lint

  max-ostapenko

  committed
  dc5732e

  Copy full SHA for dc5732e

• Merge branch 'main' into reports

  max-ostapenko

  committed
  ae875d9

  Copy full SHA for ae875d9

• Merge branch 'main' into reports

  max-ostapenko

  committed
  a4eba5a

  Copy full SHA for a4eba5a

• revisited template builder

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  963ebfa

  Copy full SHA for 963ebfa

• cleanup

  max-ostapenko

  committed
  91822e1

  Copy full SHA for 91822e1

• tf 6.13

  max-ostapenko

  committed
  e0de181

  Copy full SHA for e0de181

• lint

  max-ostapenko

  committed
  87909ef

  Copy full SHA for 87909ef

• renamed

  max-ostapenko

  committed
  24a9bac

  Copy full SHA for 24a9bac

• aligned timeout with prod

  max-ostapenko

  committed
  ade5867

  Copy full SHA for ade5867

• simplify tags

  max-ostapenko

  committed
  f2b56f0

  Copy full SHA for f2b56f0