Introduction

Small simple synthetic natural language datasets suitable for end-to-end training of tiny LLMs serve as an important resource for LLM interpretability researchers. Some well know examples include roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m (TinyFabulist). This post solves key problems that degrade the quality of these datasets, while also offering an efficient accessible pipeline that can be run locally on an NVIDIA 5060 Ti (16GB) graphics card. The core problems this post solves, include:

• True Small Vocabulary. The aforementioned datasets attempt to produce a corpus with a small vocabulary, but arguably fall a bit short of that goal. E.g., TinyStories has 49,187 unique words, SimpleStories has 40,567, and TinyFabulist has 41,502.
• Guaranteed minimal word frequencies. In the aforementioned datasets, 15 to 24 percent of the unique words occur less than 2 times, while between around 43 to 54 percent occur less than 8 times. This means that most of the unique words are likely not learnable, and mostly contribute to noise and vocabulary bloat.
• Error free text. The aforementioned datasets, include lots of errors, such as misspelled and mangled words.
• Reliable Name Disambiguation and Stratification. The aforementioned datasets, have various name management issues, ranging from collision with existing words (e.g., May vs may), name bloat, and no control over gender balance, or bias (e.g., certain names may be more likely to co-occur with potentially negative moral concepts).

Methods

This section summarizes the methods. The source code is available here, and the proof of concept dataset Small World 345.6k.

We use using unsloth/gemma-4-26B-A4B-it-qat-GGUF  for generation, through a  local llama-cpp server, which gets about 250 tokens/second on a NVIDIA 5060 Ti (16GB). The server config is available here.

Small Capped vocabulary

One difficulty we encounter is that the model will tend to keep introducing new relatively rare words, causing the vocabulary to grow large over time, yet most words never end up occurring with enough frequency for their meaning to be learnable.

We address this problem through a feature where, after a certain number of examples have been generated, we begin redoing generations up to a number of times when they come back with new words. Since a new form of an existing word has different implications regarding sufficient statistics, we separately issue retries on new forms of an existing word at a different later point depending on the project configuration. In simpler-world-345.6k, we begin capping new words after 47,232 examples, and new forms after 115,200 examples. The vocabulary at that stage had grown to 8,972 words, and by the end became 8,973 in total (one stubborn word resisted 10+ retries at some point). From 47,232 onward, between rejection from other errors (see following subsections) and from rejecting new words, we had to redo somewhere around 0 to 8 completions per batch of 144, which only marginally reduced overall throughput.

Sufficient Word Statistics

Word distributions in text follow Zipf's law, which in simple terms, means that more common words have a vastly higher frequency than somewhat more rare words. When generating a relatively small text dataset, by default many or most of the words that show up will have a very low frequency. Words that occur only a few times in the whole corpus effectively become noise, and if encountered during interpretability experiments, might become cause for confusion.

We find that in order for each word, even in a relatively small capped vocabulary, to naturally occur a sufficient number of times, the number of examples would need to be extremely large. Thus, with some frequency, we randomly sample a word to integrate into a prompt, with inverse-squared frequency weighting.

We compare against a few popular datasets meant to have a simple vocabulary, including TinyStories, SimpleStories, and TinyFables. The word statistics are calculated from the train splits after converted to lowercase. With this, TinyStories has 49,187 unique words, SimpleStories has 40,567, TinyFables has 41,502, and Small World 345.6k has 8,873.

The plot below shows the percentage of the corpus words which have a frequency greater than or equal to N, for N up to 1000.

For the other three datasets, between around 15 to 24 percent of the unique words occure less than 2 times, while between around 43 to 54 percent occure less than 8 times. Through capping and boosting, 100 percent of the unique words in Small World 345.6k occure at lea

Introduction

Small simple synthetic natural language datasets suitable for end-to-end training of tiny LLMs serve as an important resource for LLM interpretability researchers. Some well know examples include roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m (TinyFabulist). This post solves key problems that degrade the quality of these datasets, while also offering an efficient accessible pipeline that can be run locally on an NVIDIA 5060 Ti (16GB) graphics card. The core problems this post solves, include:

• True Small Vocabulary. The aforementioned datasets attempt to produce a corpus with a small vocabulary, but arguably fall a bit short of that goal. E.g., TinyStories has 49,187 unique words, SimpleStories has 40,567, and TinyFabulist has 41,502.
• Guaranteed minimal word frequencies. In the aforementioned datasets, 15 to 24 percent of the unique words occur less than 2 times, while between around 43 to 54 percent occur less than 8 times. This means that most of the unique words are likely not learnable, and mostly contribute to noise and vocabulary bloat.
• Error free text. The aforementioned datasets, include lots of errors, such as misspelled and mangled words.
• Reliable Name Disambiguation and Stratification. The aforementioned datasets, have various name management issues, ranging from collision with existing words (e.g., May vs may), name bloat, and no control over gender balance, or bias (e.g., certain names may be more likely to co-occur with potentially negative moral concepts).

Methods

This section summarizes the methods. The source code is available here, and the proof of concept dataset Small World 345.6k.

We use using unsloth/gemma-4-26B-A4B-it-qat-GGUF  for generation, through a  local llama-cpp server, which gets about 250 tokens/second on a NVIDIA 5060 Ti (16GB). The server config is available here.

Small Capped vocabulary

One difficulty we encounter is that the model will tend to keep introducing new relatively rare words, causing the vocabulary to grow large over time, yet most words never end up occurring with enough frequency for their meaning to be learnable.

We address this problem through a feature where, after a certain number of examples have been generated, we begin redoing generations up to a number of times when they come back with new words. Since a new form of an existing word has different implications regarding sufficient statistics, we separately issue retries on new forms of an existing word at a different later point depending on the project configuration. In simpler-world-345.6k, we begin capping new words after 47,232 examples, and new forms after 115,200 examples. The vocabulary at that stage had grown to 8,972 words, and by the end became 8,973 in total (one stubborn word resisted 10+ retries at some point). From 47,232 onward, between rejection from other errors (see following subsections) and from rejecting new words, we had to redo somewhere around 0 to 8 completions per batch of 144, which only marginally reduced overall throughput.

Sufficient Word Statistics

Word distributions in text follow Zipf's law, which in simple terms, means that more common words have a vastly higher frequency than somewhat more rare words. When generating a relatively small text dataset, by default many or most of the words that show up will have a very low frequency. Words that occur only a few times in the whole corpus effectively become noise, and if encountered during interpretability experiments, might become cause for confusion.

We find that in order for each word, even in a relatively small capped vocabulary, to naturally occur a sufficient number of times, the number of examples would need to be extremely large. Thus, with some frequency, we randomly sample a word to integrate into a prompt, with inverse-squared frequency weighting.

We compare against a few popular datasets meant to have a simple vocabulary, including TinyStories, SimpleStories, and TinyFables. The word statistics are calculated from the train splits after converted to lowercase. With this, TinyStories has 49,187 unique words, SimpleStories has 40,567, TinyFables has 41,502, and Small World 345.6k has 8,873.

The plot below shows the percentage of the corpus words which have a frequency greater than or equal to N, for N up to 1000.

For the other three datasets, between around 15 to 24 percent of the unique words occure less than 2 times, while between around 43 to 54 percent occure less than 8 times. Through capping and boosting, 100 percent of the unique words in Small World 345.6k occure at lea