bug: Trying to run T5 tutorial and getting free(): invalid pointer error.

[gilljon]

Description

I'm attempting to run the T5 tutorial on an A6000.

I am getting the following error:

~$ python test_kernl.py
~/.local/lib/python3.9/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
  warn(f"Failed to load image Python extension: {e}")
~/.local/lib/python3.9/site-packages/transformers/models/t5/tokenization_t5_fast.py:156: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on t5-small automatically truncating your input to 512 when padding/encoding.
- If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.
  warnings.warn(
0.10650781099997175
free(): invalid pointer
Aborted (core dumped)

The package versions are listed here:

Package                           Version
--------------------------------- -----------------------
absl-py                           1.4.0
agate                             1.6.0
agate-dbf                         0.2.0
agate-excel                       0.2.3
agate-sql                         0.5.2
aiohttp                           3.8.3
aiosignal                         1.3.1
anyio                             3.6.2
apturl                            0.5.2
argon2-cffi                       21.3.0
argon2-cffi-bindings              21.2.0
asttokens                         2.2.1
astunparse                        1.6.3
async-timeout                     4.0.2
attrs                             22.2.0
Automat                           0.8.0
awscli                            1.18.69
Babel                             2.11.0
backcall                          0.2.0
beautifulsoup4                    4.11.1
black                             23.1.0
bleach                            5.0.1
blinker                           1.4
blis                              0.7.9
boto3                             1.24.66
botocore                          1.27.96
Brlapi                            0.7.0
cachetools                        5.2.1
catalogue                         2.0.8
certifi                           2019.11.28
cffi                              1.15.1
chardet                           3.0.4
charset-normalizer                2.1.1
click                             8.1.3
click-completion                  0.5.2
click-didyoumean                  0.3.0
click-help-colors                 0.9.1
cloud-init                        22.4.2
cloudpickle                       2.1.0
cmake                             3.25.2
colorama                          0.4.3
comm                              0.1.2
command-not-found                 0.3
confection                        0.0.4
configobj                         5.0.6
constantly                        15.1.0
cryptography                      2.8
csvkit                            1.0.2
cupshelpers                       1.0
cycler                            0.11.0
cymem                             2.0.7
Cython                            0.29.32
datasets                          2.4.0
dbfread                           2.0.7
dbus-python                       1.2.16
debugpy                           1.6.5
decorator                         5.1.1
defer                             1.0.6
defusedxml                        0.7.1
dill                              0.3.5.1
distro                            1.4.0
distro-info                       0.23ubuntu1
docker-pycreds                    0.4.0
docutils                          0.16
entrypoints                       0.3
et-xmlfile                        1.0.1
exceptiongroup                    1.1.0
executing                         1.2.0
fastjsonschema                    2.16.2
filelock                          3.9.0
flake8                            6.0.0
flatbuffers                       1.12
fonttools                         4.38.0
frozenlist                        1.3.3
fsspec                            2022.11.0
future                            0.18.2
gast                              0.4.0
gdown                             4.5.1
gitdb                             4.0.10
GitPython                         3.1.30
google-auth                       2.16.0
google-auth-oauthlib              0.4.6
google-pasta                      0.2.0
gql                               3.0.0a6
gradient                          2.0.6
gradient-utils                    0.5.0
graphql-core                      3.1.7
greenlet                          2.0.1
grpcio                            1.51.1
h5py                              3.7.0
halo                              0.0.31
httplib2                          0.14.0
huggingface-hub                   0.11.1
hyperlink                         19.0.0
idna                              2.8
imageio                           2.24.0
importlib-metadata                6.0.0
incremental                       16.10.1
iniconfig                         2.0.0
ipykernel                         6.15.2
ipython                           8.5.0
ipython-genutils                  0.2.0
ipywidgets                        8.0.2
isodate                           0.6.0
isort                             5.12.0
jax                               0.4.1
jaxlib                            0.4.1+cuda11.cudnn82
jdcal                             1.0
jedi                              0.18.2
Jinja2                            3.1.2
jmespath                          0.9.4
joblib                            1.2.0
json5                             0.9.11
jsonify                           0.5
jsonpatch                         1.22
jsonpointer                       2.0
jsonschema                        4.17.3
jupyter_client                    7.4.8
jupyter-contrib-core              0.4.2
jupyter-contrib-nbextensions      0.7.0
jupyter_core                      5.1.3
jupyter-highlight-selected-word   0.2.0
jupyter-nbextensions-configurator 0.6.1
jupyter-server                    1.23.5
jupyter-server-mathjax            0.2.6
jupyterlab                        3.4.6
jupyterlab-git                    0.41.0
jupyterlab-pygments               0.2.2
jupyterlab_server                 2.18.0
jupyterlab-widgets                3.0.5
keras                             2.9.0
Keras-Preprocessing               1.1.2
kernl                             0.2.2
keyring                           18.0.1
kiwisolver                        1.4.4
langcodes                         3.3.0
language-selector                 0.1
launchpadlib                      1.10.13
lazr.restfulclient                0.14.2
lazr.uri                          1.0.3
leather                           0.3.3
libclang                          15.0.6.1
log-symbols                       0.0.14
louis                             3.12.0
lxml                              4.5.0
macaroonbakery                    1.3.1
Markdown                          3.4.1
MarkupSafe                        2.1.1
marshmallow                       2.21.0
matplotlib                        3.5.3
matplotlib-inline                 0.1.6
mccabe                            0.7.0
mistune                           2.0.4
more-itertools                    4.2.0
mpmath                            1.3.0
multidict                         6.0.4
multiprocess                      0.70.13
murmurhash                        1.0.9
mypy-extensions                   1.0.0
nbclassic                         0.4.8
nbclient                          0.7.2
nbconvert                         7.2.7
nbdime                            3.1.1
nbformat                          5.7.3
nest-asyncio                      1.5.6
netifaces                         0.10.4
networkx                          3.0
nltk                              3.7
notebook                          6.5.2
notebook_shim                     0.2.2
numpy                             1.23.2
oauthlib                          3.1.0
olefile                           0.46
opencv-python                     4.6.0.66
openpyxl                          3.0.3
opt-einsum                        3.3.0
packaging                         23.0
pandas                            1.4.4
pandocfilters                     1.5.0
parsedatetime                     2.4
parso                             0.8.3
pathspec                          0.11.0
pathtools                         0.1.2
pathy                             0.10.1
pexpect                           4.6.0
pickleshare                       0.7.5
Pillow                            9.2.0
pip                               22.3.1
platformdirs                      2.6.2
pluggy                            1.0.0
preshed                           3.0.8
progressbar2                      4.2.0
prometheus-client                 0.9.0
promise                           2.3
prompt-toolkit                    3.0.36
protobuf                          3.19.6
psutil                            5.9.4
ptyprocess                        0.7.0
pure-eval                         0.2.2
pyarrow                           10.0.1
pyasn1                            0.4.2
pyasn1-modules                    0.2.1
pycairo                           1.16.2
pycodestyle                       2.10.0
pycparser                         2.21
pycups                            1.9.73
pydantic                          1.9.2
pyflakes                          3.0.1
Pygments                          2.14.0
PyGObject                         3.36.0
PyHamcrest                        1.9.0
PyJWT                             1.7.1
pymacaroons                       0.13.0
pymongo                           3.13.0
PyNaCl                            1.3.0
pyOpenSSL                         19.0.0
pyparsing                         3.0.9
pyRFC3339                         1.1
pyrsistent                        0.15.5
pyserial                          3.4
PySocks                           1.7.1
pytest                            7.2.2
python-apt                        2.0.0+ubuntu0.20.4.8
python-dateutil                   2.8.2
python-debian                     0.1.36ubuntu1
python-distutils-extra            2.39
python-slugify                    4.0.0
python-utils                      3.4.5
pytimeparse                       1.1.5
pytorch-triton                    2.0.0+0d7e753227
pytz                              2022.7
PyWavelets                        1.4.1
pyxdg                             0.26
PyYAML                            5.4.1
pyzmq                             25.0.0
regex                             2022.10.31
reportlab                         3.5.34
requests                          2.28.2
requests-oauthlib                 1.3.1
requests-toolbelt                 0.10.1
requests-unixsocket               0.2.0
responses                         0.18.0
roman                             2.0.0
rsa                               4.0
s3transfer                        0.6.0
scikit-image                      0.19.3
scikit-learn                      1.1.2
scipy                             1.9.1
screen-resolution-extra           0.0.0
seaborn                           0.12.0
SecretStorage                     2.3.1
Send2Trash                        1.8.0
sentence-transformers             2.2.2
sentencepiece                     0.1.97
sentry-sdk                        1.13.0
service-identity                  18.1.0
setproctitle                      1.3.2
setuptools                        45.2.0
shellingham                       1.5.0.post1
shortuuid                         1.0.11
simplejson                        3.16.0
six                               1.14.0
smart-open                        6.3.0
smmap                             5.0.0
sniffio                           1.3.0
sos                               4.4
soupsieve                         2.3.2.post1
spacy                             3.4.1
spacy-legacy                      3.0.11
spacy-loggers                     1.0.4
spinners                          0.0.24
SQLAlchemy                        1.4.40
srsly                             2.4.5
ssh-import-id                     5.10
stack-data                        0.6.2
sympy                             1.11.1
systemd-python                    234
tabulate                          0.8.10
tensorboard                       2.9.1
tensorboard-data-server           0.6.1
tensorboard-plugin-wit            1.8.1
tensorflow                        2.9.2
tensorflow-estimator              2.9.0
tensorflow-io-gcs-filesystem      0.29.0
termcolor                         2.2.0
terminado                         0.17.1
terminaltables                    3.1.10
thinc                             8.1.6
threadpoolctl                     3.1.0
tifffile                          2022.10.10
tinycss2                          1.2.1
tokenize-rt                       5.0.0
tokenizers                        0.12.1
tomli                             2.0.1
torch                             2.0.0.dev20230128+cu117
torchaudio                        0.12.1+cu116
torchvision                       0.13.1+cu116
tornado                           6.2
tqdm                              4.64.1
traitlets                         5.8.1
transformers                      4.21.3
triton                            2.0.0.dev20221202
Twisted                           18.9.0
typer                             0.4.2
typing_extensions                 4.4.0
ubuntu-advantage-tools            27.12
ubuntu-drivers-common             0.0.0
ufw                               0.36
unattended-upgrades               0.1
Unidecode                         1.1.1
urllib3                           1.26.14
wadllib                           1.3.3
wandb                             0.13.4
wasabi                            0.10.1
wcwidth                           0.2.5
webencodings                      0.5.1
websocket-client                  0.57.0
Werkzeug                          2.2.2
wheel                             0.35.1
widgetsnbextension                4.0.5
wrapt                             1.14.1
xgboost                           1.6.2
xkit                              0.0.0
xlrd                              1.1.0
xxhash                            3.2.0
yarl                              1.8.2
zipp                              1.0.0
zope.interface                    4.7.1

Steps to reproduce

1. Install kernl using:

pip install 'git+https://github.com/ELS-RD/kernl' --extra-index-url https://download.pytorch.org/whl/nightly/cu117
# or for local dev, after git clone ...
pip install -e . --extra-index-url https://download.pytorch.org/whl/nightly/cu117

2. Try to execute T5 Optimization.

Expected Behavior

Code should execute without error. Speed-up is a plus.

Actual Behavior

Errors.

Your environment

CPU Info:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              1
Core(s) per socket:              8
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           106
Model name:                      Intel(R) Xeon(R) Gold 5315Y CPU @ 3.20GHz
Stepping:                        6
CPU MHz:                         3200.270
BogoMIPS:                        6400.13
Hypervisor vendor:               Xen
Virtualization type:             full
L1d cache:                       384 KiB
L1i cache:                       256 KiB
L2 cache:                        10 MiB
L3 cache:                        96 MiB
NUMA node0 CPU(s):               0-7

OS: Linux psfbhdmgj 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Self-service

• I would be willing to help fix this bug myself.

Code of Conduct

• I agree to follow this project's Code of Conduct

[jonathlela]

Thanks for reporting this @gilljon. Could you give the content of test_kernl.py ?
We have some issues with t5 and bf16 but I don't think it's related.