Code

[FEATURE] Update tokenizers#158

Code

Merged

KenelmQLH merged 21 commits intobigdata-ustc:devbigdata-ustc/EduNLP:devfrom KINGNEWBLUSH:devKINGNEWBLUSH/EduNLP:devCopy head branch name to clipboardMar 14, 2024

+126-14

Commits

Commits on Mar 10, 2024

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for dbe8936
  KINGNEWBLUSH

  committed
  dbe8936

  Copy full SHA for dbe8936

• Update tokenization.py

  KINGNEWBLUSH

  authored
  8fd96b2

  Copy full SHA for 8fd96b2

Commits on Mar 11, 2024

• modified: AUTHORS.md

  Show description for 025fa86
  KINGNEWBLUSH

  committed
  025fa86

  Copy full SHA for 025fa86

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for aea99a2
  KINGNEWBLUSH

  committed
  aea99a2

  Copy full SHA for aea99a2

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for 970c1b9
  KINGNEWBLUSH

  committed
  970c1b9

  Copy full SHA for 970c1b9

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for a289a7a
  KINGNEWBLUSH

  committed
  a289a7a

  Copy full SHA for a289a7a

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for ad7df8b
  KINGNEWBLUSH

  committed
  ad7df8b

  Copy full SHA for ad7df8b

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for 9423b31
  KINGNEWBLUSH

  committed
  9423b31

  Copy full SHA for 9423b31

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for 5792e48
  KINGNEWBLUSH

  committed
  5792e48

  Copy full SHA for 5792e48

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for edc266f
  KINGNEWBLUSH

  committed
  edc266f

  Copy full SHA for edc266f

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  KINGNEWBLUSH

  committed
  c526016

  Copy full SHA for c526016

Commits on Mar 12, 2024

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  KINGNEWBLUSH

  committed
  64c6cda

  Copy full SHA for 64c6cda

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  KINGNEWBLUSH

  committed
  3a53b51

  Copy full SHA for 3a53b51

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  KINGNEWBLUSH

  committed
  569bb9f

  Copy full SHA for 569bb9f

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for 4542258
  KINGNEWBLUSH

  committed
  4542258

  Copy full SHA for 4542258

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  KINGNEWBLUSH

  committed
  721bc0a

  Copy full SHA for 721bc0a

• modified: tests/test_tokenizer/test_tokenizer.py

  KINGNEWBLUSH

  committed
  1476f8a

  Copy full SHA for 1476f8a

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for f02ccce
  KINGNEWBLUSH

  committed
  f02ccce

  Copy full SHA for f02ccce

• modified: tests/test_tokenizer/test_tokenizer.py

  KINGNEWBLUSH

  committed
  05172b4

  Copy full SHA for 05172b4

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  Show description for 767778f
  KINGNEWBLUSH

  committed
  767778f

  Copy full SHA for 767778f

• modified: EduNLP/SIF/tokenization/text/tokenization.py

  KINGNEWBLUSH

  committed
  e86a5e6

  Copy full SHA for e86a5e6