Analyzing Similarity Metrics for Data Selection for Language Model Pretraining

This research reveals a critical architectural mismatch where retrieval-trained embeddings fail to optimize language model pretraining. Learn why corpus-spec...

Level: advanced

By Dylan Sam, Ayan Chakrabarti, Afshin Rostamizadeh, Srikumar Ramalingam, Gui Citovsky, Sanjiv Kumar

Category: research