The project is forked from "Neural Circuit Policies" (ncps) 📖 Docs. The analysis made in this repository were conducted study purpose only.
Our project is an experiment with Liquid Neural Networks. We extend the functionality of the LTCCell by introducing a Self-Learning Activation Function, i.e. swish function, to fight the vanishing gradient problem well known to all Deep Learning architectures.
We use the Kaggle dataset to test the performance of introducing swish function as the main activation function. Besides, we benchmark the achieved predictions with Liquid Time Constant Neural Networks against well-known statistical models, like ARIMA. The detailed research and conducted results can be found in examination/examination.pdf. (currently available only in german language)
├── LICENSE
├── README.md <- The top-level README for developers using this project.
├── data
│ ├── csv <- Time-Series data in csv format (one csv - one station).
│ └── parquet <- Time-Series data in parquet format (one file for all stations).
│
├── ncps <- Forked code with minor changes (see Disclaimer)
│
├── examination <- Paper where the final project is fully described. (Currently in german only)
│
│
├── notebooks <- Jupyter notebooks. Training, Evaluation and PCA Analysis notebooks can be found here.
│
├── project <- Python utility code used for data preparation and training.
│
├── analysis <- Folder containing all stuff related to the data analysis made in R.
│ ├── R <- R Scripts used for data cleaning and feature engineering.
│ └── plots <- Generated graphics and figures to be used in data analysis.
│
├── requirements.txt <- The requirements file for reproducing the analysis environment, e.g.
│ generated with `pip freeze > requirements.txt`
│
│── run.py <- `main` python file for training the LNN models.
│── config.py <- Configuration file containing data related variables as well as LNN hyperparameters.
└── log <- Logging information automatically generated by pytorch_lightning.
Please use poetry.lock or requirements.txt for creating the python virtual environment for the project:
poetry install -E cpu --with cpu
// or
python -m pip install -r ./requirements_cpu.txtpoetry install -E cuda --with cuda
// or
python -m pip install -r ./requirements_gpu.txtTo train the model by using the commad shell:
poetry run python -m run.py
// or
python -m run.pyThe output (checkpoints) containing the models, will be placed into pl_checkpoints/ directory. Feel free use other location by changing the variable in config.py.
You also may use jupyter notebooks provided in notebooks/ for training and evaluation purposes.
If you would like to only experiment with already trained models, please find those in models.tar. You can use them by executing following python code:
# best_lnn_slaf_model
from ncps.torch import LTC
from ncps.wirings import AutoNCP
from project.model import SequenceLearner
wiring = AutoNCP(8, 1) # 8 units, 1 motor neuron
ltc_model = LTC(
in_features,
wiring,
batch_first=True,
use_swish_activation=True
)
trainer = SequenceLearner.load_from_checkpoint(path)
model = trainer.model
model.to(device)
model.eval()
# your input
X = ...
preds = model(X).detach().numpy()# best_lnn_model
from ncps.torch import LTC
from ncps.wirings import AutoNCP
from project.model import SequenceLearner
wiring = AutoNCP(32, 1) # 8 units, 1 motor neuron
ltc_model = LTC(
in_features,
wiring,
batch_first=True,
use_swish_activation=False
)
trainer = SequenceLearner.load_from_checkpoint(path)
model = trainer.model
model.to(device)
model.eval()
# your input
X = ...
preds = model(X).detach().numpy()Vitali Krilov & Vladislav Stasenko (both FHSWF students) authored this GitHub repository.
We belive that experimenting with LNN is a highly interesting topic, therefore further community help and ideas is highly welcome!