> Can I actually add "features" to the model?
Yes. You can use a number of libraries to add, mix, merge, etc. layers [1]
> Not with weights. I need it to learn case histories to extend its "feature-set".
Again, yes. You can add attention heads, other features, heck you can even add classification if you want [2]. Because you are working with an open architecture! What you think of weights are not binary blobs. That is a common missconception.
[1] - https://github.com/arcee-ai/mergekit
[2] - https://github.com/center-for-humans-and-machines/transforme...
