this post was submitted on 20 Dec 2024
12 points (92.9% liked)
LocalLLaMA
2328 readers
7 users here now
Community to discuss about LLaMA, the large language model created by Meta AI.
This is intended to be a replacement for r/LocalLLaMA on Reddit.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Here's a guide: https://medium.com/@yuxiaojian/fine-tuning-llama3-1-and-deploy-to-ollama-f500a6579090
Not sure what the impact of a few 100 examples will be, though, even if you freeze most parameters.
Can SFT be used on partial generations? What I mean by a "steer" is a correction to only a portion, and not even the end, of model output.
For example, a "bad" partial output might be:
and the "steer" might be:
but the full response will eventually be:
The corrections don't include the full output.
I do not know what SFT means. So I can't comment on that, I'm afraid.
Models only predict the distribution of the next token. So "partial response" vs "full response" is a consequence repeated inference untill the stop token is reached. It's mostly unrelated to the model parameters.
For training, it makes no difference.
The article you linked to uses SFT (supervised fine tuning, a specific training technique) as its alignment strategy. There are other ways to fine-tune a model.
I guess I'm wondering if you can train on these partial responses without needing the full rest of the output, without the stop token, or if you need full examples as the article hints to.
I was unaware of that acronym, thank you. It does make me wonder: is there unsupervised training of LLMs?
Yes, you can train without the stop token. The stop token is just that: another token. I do not expect the model to unlearn the usage of the stop token from training on a few 100 new examples.
Unsupervised training happens during the pre-training phase when you dump all kinds of quality documents and it learns the relationship between tokens
Could you perhaps share a reference for this? I'm eager to learn as I don't quite understand.
I've always trained LLM supervised: predict token N+1 based on tokens 1 to N.
This pre-training was done by Meta. It's what Llama-3.1-405B is (in contrast to Llama-3.1-405B-Instruct). https://huggingface.co/meta-llama/Llama-3.1-405B
Oh I see the origin of my confusion. The terminology "supervised learning" got repurposed.
It's all supervised learning if the model is learning the relationship between input and expected output (using supervised learning as described in (1)). The methodology of "pre-training" is the same as that of "supervised fine tuning".
There's no unsupervised learning happening, as described in (2)
No, it's unsupervised. In pre-training, the text data isn't structured at all. It's books, documents, online sources, all put together.
Supervised learning uses data with "ground truth" labels.
Have you worked with (variational) auto-encoders? I think they're a great example of what I would call unsupervised learning.
What are "ground truth" labels?
Ground truth labels are just prescriptive labels that we recognize as being true. The main thing that distinguishes unsupervised from supervised is that in unsupervised learning, what is "good" is learned from the unstructured data itself. In supervised learning, what is "good" is learned from some external input, like "good" human-provided examples.
Would you call token (N+1), given tokens (1 to N) as a ground truth?
No, in that case there's no labelling required. That would be unsupervised learning.
https://en.wikipedia.org/wiki/Unsupervised_learning
So supervised vs unsupervised, according to you, is a property of the dataset?
Sorry, I really don't care to continue talking about the difference between supervised and unsupervised learning. It's a pattern used to describe how you are doing ML. It's not a property of a dataset (you wouldn't call Dataset A "unsupervised"). Read the Wikipedia articles for more details.
It's alright :)