A diverse English text corpus from 22 sources, including papers, code, books and web text.
Its licence doesn't allow Rubynox to redistribute it, so we never deliver it. Get it directly from EleutherAI; we list it so you can see all your options in one place.
The original release is no longer distributed because it included copyrighted books (Books3).
More in AI training text
AI training text · Common Crawl Foundation
An open archive of the web, crawled every month since 2008. The raw material behind most large language models.
AI training text · Wikimedia Foundation
Every article in every language edition, as published in the regular Wikimedia database dumps.
AI training text · Hugging Face
Cleaned and de-duplicated English web text from 96 Common Crawl snapshots, built for LLM pre-training.
AI training text · Google and Allen Institute for AI
Cleaned Common Crawl text created for Google's T5 model.
AI training text · Technology Innovation Institute (TII)
Filtered, de-duplicated web text used to train the Falcon models.
AI training text · Together AI
Open web text with quality signals, built to reproduce LLaMA-style training data.