How to fine tune this model

[pepi99]

Hello,
I am wondering - is it possible to fine-tune this model for some domain/specific words? I have training data, but I am not sure how exactly to fine tune what is already trained.

My idea is to use transfer learning on the trained Dense layer and preserve the Bert main layer.
Is there an intuitive way to do that?

Cheers,
Petar

[pepi99]

I thought about this, and this approach might lead to catastrophic forgetting. What if I just add some new examples into the SemEval task 4 dataset and retrain the model ?