Code

[feature]: add embedding parallel#438

Code

Merged

chengmengli06 merged 69 commits intomasteralibaba/EasyRec:masterfrom add_embedding_parallelalibaba/EasyRec:add_embedding_parallelCopy head branch name to clipboardJan 3, 2024

+6,603-705

Commits

Commits on Oct 18, 2023

• add support for keep model

  

  杨熙

  committed
  54c9e9e

  Copy full SHA for 54c9e9e

• add adam_s

  chengmengli06

  committed
  bd5bd02

  Copy full SHA for bd5bd02

Commits on Oct 19, 2023

• add sok debug case2

  chengmengli06

  committed
  ae74f18

  Copy full SHA for ae74f18

Commits on Oct 20, 2023

• add drop_remainder option as required by sok

  chengmengli06

  committed
  bbfe530

  Copy full SHA for bbfe530

• add adam sparse

  chengmengli06

  committed
  5477cb2

  Copy full SHA for 5477cb2

Commits on Oct 21, 2023

• fix parquet input bugs

  chengmengli06

  committed
  23a9fba

  Copy full SHA for 23a9fba

• add all2all impl

  chengmengli06

  committed
  4ce5229

  Copy full SHA for 4ce5229

Commits on Oct 26, 2023

• add saver for sok

  chengmengli06

  committed
  628674a

  Copy full SHA for 628674a

• add missing files

  chengmengli06

  committed
  a1032da

  Copy full SHA for a1032da

• add support for eval

  chengmengli06

  committed
  7f33ddb

  Copy full SHA for 7f33ddb

Commits on Oct 28, 2023

• add load kv embed op

  chengmengli06

  committed
  5d858be

  Copy full SHA for 5d858be

Commits on Oct 29, 2023

• fix bug

  chengmengli06

  committed
  b5a143e

  Copy full SHA for b5a143e

• add export sok embed

  chengmengli06

  committed
  d0910e3

  Copy full SHA for d0910e3

Commits on Oct 30, 2023

• fix sok optimizer wrapper call _finish two times bug

  chengmengli06

  committed
  e844169

  Copy full SHA for e844169

• no need to switch non slot

  chengmengli06

  committed
  b60e3ba

  Copy full SHA for b60e3ba

• add support for lookup only

  chengmengli06

  committed
  4124a16

  Copy full SHA for 4124a16

Commits on Oct 31, 2023

• remove debug code

  chengmengli06

  committed
  40fd1a7

  Copy full SHA for 40fd1a7

• fix dynamic partition bug

  chengmengli06

  committed
  592f7bd

  Copy full SHA for 592f7bd

• add load embed to whl package

  chengmengli06

  committed
  a1909cb

  Copy full SHA for a1909cb

Commits on Nov 1, 2023

• optimize segment_ids performance

  chengmengli06

  committed
  dc3e45b

  Copy full SHA for dc3e45b

• add predictor files

  chengmengli06

  committed
  528fa3b

  Copy full SHA for 528fa3b

• fix predict bug

  chengmengli06

  committed
  34938de

  Copy full SHA for 34938de

• fix bug

  chengmengli06

  committed
  03fb4d9

  Copy full SHA for 03fb4d9

Commits on Nov 2, 2023

• fix eval bug and dense grad bug

  chengmengli06

  committed
  e3613f3

  Copy full SHA for e3613f3

• add support for dense lookup in distributed mode

  chengmengli06

  committed
  3292707

  Copy full SHA for 3292707

• refactor code style

  chengmengli06

  committed
  bf0edb0

  Copy full SHA for bf0edb0

• add support for finetune

  chengmengli06

  committed
  76937dc

  Copy full SHA for 76937dc

Commits on Nov 6, 2023

• add support for ALL_COLUMNS in parquet predictor

  chengmengli06

  committed
  e3b91a5

  Copy full SHA for e3b91a5

Commits on Nov 10, 2023

• fix predict bug

  chengmengli06

  committed
  62ef27d

  Copy full SHA for 62ef27d

• fix numpy version

  chengmengli06

  committed
  7c55ff7

  Copy full SHA for 7c55ff7

• make batch_size work for parquet predictor

  chengmengli06

  committed
  77efe8a

  Copy full SHA for 77efe8a

• fix multiproc bug

  chengmengli06

  committed
  2b34fda

  Copy full SHA for 2b34fda

Commits on Nov 13, 2023

• make read parquet fast

  chengmengli06

  committed
  f606c01

  Copy full SHA for f606c01

Commits on Nov 19, 2023

• fix dataset miss padding data bug

  chengmengli06

  committed
  fab2dad

  Copy full SHA for fab2dad

• add loss weight update

  chengmengli06

  committed
  9f903e7

  Copy full SHA for 9f903e7

Commits on Nov 21, 2023

• update load_embed to support large embedding tables

  chengmengli06

  committed
  cb1a931

  Copy full SHA for cb1a931

Commits on Nov 22, 2023

• support old training path

  chengmengli06

  committed
  3ac7494

  Copy full SHA for 3ac7494

Commits on Nov 27, 2023

• support rebuild queue

  chengmengli06

  committed
  4fd07b9

  Copy full SHA for 4fd07b9

Commits on Nov 28, 2023

• drop old code

  chengmengli06

  committed
  9eab301

  Copy full SHA for 9eab301

Commits on Nov 29, 2023

• add fast auc calculate

  chengmengli06

  committed
  41df7ce

  Copy full SHA for 41df7ce

• add debug info

  chengmengli06

  committed
  51b306c

  Copy full SHA for 51b306c

Commits on Dec 8, 2023

• merge master and remove extra files

  chengmengli06

  committed
  cf1159b

  Copy full SHA for cf1159b

Commits on Dec 9, 2023

• merge master and remove extra files

  chengmengli06

  committed
  23a379f

  Copy full SHA for 23a379f

• add embedding parallel support

  chengmengli06

  committed
  b15eec9

  Copy full SHA for b15eec9

• drop useless files

  chengmengli06

  committed
  9a768fb

  Copy full SHA for 9a768fb

• update embedding parallel test

  chengmengli06

  committed
  4a51066

  Copy full SHA for 4a51066

Commits on Dec 10, 2023

• remove nsys profile code

  chengmengli06

  committed
  e1a3670

  Copy full SHA for e1a3670

• remove debug code

  chengmengli06

  committed
  c349303

  Copy full SHA for c349303

• remove queue multiprocessing extra params

  chengmengli06

  committed
  9ac2f84

  Copy full SHA for 9ac2f84

• remove queue multiprocessing extra params

  chengmengli06

  committed
  7e4b06b

  Copy full SHA for 7e4b06b

Commits on Dec 11, 2023

• fix unit test bug

  chengmengli06

  committed
  aab9cab

  Copy full SHA for aab9cab

Commits on Dec 18, 2023

• fix bug

  chengmengli06

  committed
  e7d94d4

  Copy full SHA for e7d94d4

• fix bug

  chengmengli06

  committed
  ce0eb4f

  Copy full SHA for ce0eb4f

Commits on Dec 19, 2023

• fix ut bug

  chengmengli06

  committed
  8ee2981

  Copy full SHA for 8ee2981

• add support for non gpu case

  chengmengli06

  committed
  c39c744

  Copy full SHA for c39c744

Commits on Dec 22, 2023

• add custom op src and build scripts

  chengmengli06

  committed
  cbb8596

  Copy full SHA for cbb8596

Commits on Dec 27, 2023

• remove useless code

  chengmengli06

  committed
  cdf6ccc

  Copy full SHA for cdf6ccc

Commits on Dec 29, 2023

• add support for scalar data when loading parquet_files

  chengmengli06

  committed
  97b2f08

  Copy full SHA for 97b2f08

• fix merge conflicts

  chengmengli06

  committed
  091476b

  Copy full SHA for 091476b

• fix bug

  chengmengli06

  committed
  c0813a9

  Copy full SHA for c0813a9

Commits on Jan 1, 2024

• add comments on parquet input

  chengmengli06

  committed
  cc3f7d0

  Copy full SHA for cc3f7d0

Commits on Jan 2, 2024

• add test cases on embedding parallel and parquet dataset

  chengmengli06

  committed
  6203f24

  Copy full SHA for 6203f24

Commits on Jan 3, 2024

• limit parquet case to run on python3 and tf2.x

  chengmengli06

  committed
  22d54cb

  Copy full SHA for 22d54cb

• limit parquet case to run on python3 and tf2.x

  chengmengli06

  committed
  b58e246

  Copy full SHA for b58e246

• fix export typo

  chengmengli06

  committed
  1987ce7

  Copy full SHA for 1987ce7

• fix export typo

  chengmengli06

  committed
  c10b951

  Copy full SHA for c10b951

• reset init var settings, fix fast auc bug

  chengmengli06

  committed
  66e1ebd

  Copy full SHA for 66e1ebd

• fix merge conflicts

  chengmengli06

  committed
  bb72631

  Copy full SHA for bb72631

• add script for custom ops build

  chengmengli06

  committed
  8de576e

  Copy full SHA for 8de576e