Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
appvoid 
posted an update 1 day ago
Post
106
If you lack ideas for a cool model, here's one.

Train a model from scratch on wikipedia with one twist: the tokenizer changes the actual token ids used on every sample fed. If somehow still learns English, you have made an astonishing discovery.

You would have answered the question: Can a model learn human languages from structure alone?
appvoid

Moreover, the implications are that finetuning models already pretrained on structure could outperform ones starting from scratch.
In this post