Two small models, almost a billion prepared tokens, a few hours on a rented GPU, and a much better understanding of what happens before a model starts generating text.
We are living in a strange time in AI. You can open an editor, call a model through an API, give an agent some instructions, and have a working application before lunch. I do that too. It is exciting, and I don't think we should pretend otherwise.
But I kept coming back to a question: what do I actually understand about the model on the other side of that API?
I know the idea of a Transformer. I know attention, embeddings, tokenization, gradient descent. I have worked with machine learning models before. Still, understanding the pieces separately is different from putting them together, starting with random weights, and watching the system learn something from text.
So I built Tenslam MiniLM, a small language-model project in PyTorch. Not because I thought I could compete with large labs. I wanted to go through the whole process myself, from preparing the text to loading the final weights and seeing what came out.
And it turns out pretraining from scratch is surprisingly fun in an era when almost everyone is talking about post-training.
Starting with something small
The first model had about 29.9 million parameters. Eight Transformer blocks, a 512-token context window, and an 8,192-token vocabulary. I trained it on TinyStories, a dataset of simple, short stories.
The choice was deliberate. I didn't want my first experiment to become a giant infrastructure project before I could see whether the model learned anything at all.
At the beginning, its next-token predictions were basically noise. There were numbers changing on the screen, but numbers alone are not very satisfying. Then, as training progressed, the generated sentences started looking like sentences. The model began producing simple stories with recognizable characters and events.
After 100 million training tokens, the recorded validation loss was 1.6312 on the run's evaluated TinyStories slice, with perplexity around 5.11.
That was the first moment I really enjoyed. The system wasn't retrieving a memorized answer from a search engine or calling another model. Those randomly initialized parameters had been adjusted by the training process until the model could continue a certain kind of text.
Of course, it could mostly write simple stories. That was what I had trained it on. But for this experiment, seeing that transition mattered more than making it sound intelligent.
Then I made the problem bigger
The second model had 60.1 million parameters. This time I used a 16,384-token byte-level BPE tokenizer and a mixture of six sources: TinyStories, FineWeb-Edu, Wikipedia, Gutenberg, and two Cosmopedia subsets.
After preparation, I had 946 million training tokens.
That number looks neat when written in a sentence. Getting there was much less neat.
You have to collect the data, clean it, remove duplicates, separate validation documents, make sure the tokenizer isn't trained on validation text, tokenize everything, and write it into a format the training loop can read efficiently. I also wanted pinned dataset revisions, hashes, and a record of what was kept and rejected, so I could understand what actually went into the model.
There is something particularly irritating about data preparation. None of the individual steps sounds that difficult. But each can quietly change the result. A duplicate document slips through. A validation example overlaps with training. A dataset takes far longer to stream than expected. An upload appears to finish but the file is incomplete.
The Stage 2 preparation alone took about 57 minutes on RunPod. Before that, I had already dealt with slow data streaming on my laptop and broken multipart uploads. At one point, repeated commands also led to overlapping preparation jobs. I eventually moved to a single locked script that could resume safely instead of relying on me to remember which command I had run.
Building the Transformer was one kind of problem. Making sure the data and training process were trustworthy was another.
I found the second one more time-consuming than I expected.
Watching a GPU work
For the longer run, I rented an NVIDIA L4 on RunPod. The model used a 512-token context and processed 65,536 tokens per optimizer step, using gradient accumulation. The observed throughput was roughly 40,700 tokens per second.
There is a very specific kind of satisfaction in seeing a training log move:
step 12070/15259 | loss 2.8657 | lr 1.20e-04 | gnorm 0.34 | 40,983 tok/s
I kept checking the steps on my phone. The original target was about one billion training tokens, or 15,259 steps. Based on the speed, I was estimating when it would finish.
It stopped earlier.
The pipeline had a 5.5-hour training limit, and that limit arrived at step 12,076. It had processed 791,412,736 training tokens. Not the full billion I had targeted, but still a completed, clean run under the configured time budget.
There was no dramatic crash. The script exported the model, packaged the important files, and reported that it was done.
The interesting part came afterward.
A finished model, and a missing checkpoint
I was using my phone to download the results. The final archive was about 223 MB in decimal units, which some apps displayed as 213 MiB. For a moment, even that looked suspicious. It wasn't; it was just two different ways to report the same number of bytes.
I downloaded the archive and its checksum file, then terminated the pod. What I didn't manage to download was the much larger last.pt training checkpoint, around 703 MB in the remote listing.
Fortunately, the archive contained the exported model weights, tokenizer, and configuration. I could recover the model and run it locally on my CPU. But the lost checkpoint meant I couldn't resume the original training run exactly, with its optimizer and random-number-generator state intact.
That difference is easy to overlook until it matters. A model you can use for inference is not necessarily a checkpoint you can resume training from.
Next time, I'll treat checkpoint backup as part of the training plan, not something to remember after training ends.
The number that looked too good
Then I made another discovery, and this one was more important than the checkpoint.
The logs showed an aggregate validation loss around 1.567 and a perplexity of 4.79. At first glance, that looked great. The model had been trained on a mixed corpus and apparently reached a very low validation loss.
But something didn't fit. The separately reported losses for Wikipedia, FineWeb-Edu, Gutenberg, and the other sources were much higher.
When I checked the evaluation code, the reason became clear. The aggregate evaluator read the first 819,200 target tokens from a concatenated validation stream. TinyStories was the first source, and its validation split alone contained more than that. So the supposed aggregate result had actually evaluated TinyStories only.
The number itself wasn't invented. It measured something real. It just didn't measure what the label seemed to imply.
That was probably the most useful mistake in this project.
We corrected the evaluation code for future runs so it evaluates sources separately and combines them with proper weighting. But the old archived result remains what it was: a partial TinyStories measurement, not a full mixed-corpus benchmark. The original validation bins weren't in my recovered archive, so I cannot honestly give a measured full-mix perplexity for that run.
A training dashboard can look impressive while asking the wrong question. You have to understand where the number came from.
So, can it actually write?
Yes, in a limited way.
The 60M model can continue simple stories with locally coherent sentences. Give it a prompt about a girl finding a map, and it can turn that into a small narrative. That was enjoyable to see.
Ask it to continue something factual and the limits become obvious. In one test involving Einstein, it confidently invented a publication detail. Other prompts produced repetition, loops, or statements that simply weren't correct.
This isn't an instruction-tuned assistant. I didn't train it to follow a conversation format or prefer helpful answers. It is a base next-token prediction model, and a very small one. It learned patterns from the text it saw, not an obligation to tell the truth.
I don't think there is much value in pretending the outputs are better than they are. The point was to understand the process, and the failures explain the process too.
What I actually learned
A model is not just a network architecture. It is an entire chain:
text → cleaning → tokenization → training examples → forward pass → loss → gradients → optimizer → evaluation → weights → generation
And every arrow hides decisions.
A vocabulary that is too large wastes a lot of a small model's capacity on embeddings. A benchmark on one GPU doesn't predict performance on another. A clean-looking validation score might be sampling the wrong data. A completed export may not contain everything required to resume training.
None of these discoveries is a new research breakthrough. Engineers have solved and documented many of these problems already. But there is a difference between reading about them and discovering why they matter in your own code.
I have published the Tenslam MiniLM source code. The model weights remain private for now while I review the redistribution terms of the mixed datasets. The code is there to inspect, reproduce, challenge, and improve.
My next experiment will probably move toward post-training a small existing base model. I want to understand supervised fine-tuning, LoRA, and preference learning with the same level of care.
But I'm glad I started at the other end.
In a world where using intelligence through an API is becoming almost effortless, spending time understanding how a tiny piece of that intelligence is trained still feels worth doing.
Not everything has to become a product. Sometimes I just want to know how something works.