Introducing PKaI Nano 2: A Bigger Nano and Our First Instruct Model
A bigger Nano at 792M parameters with 2048-token context and a broader training mix, released as a base model and our first instruction-tuned chat model. It also serves as the base for the Nano 2 line.
The Next Nano
Today we are releasing PKaI Nano 2, the fourth model in the Nano series and the first one we are shipping in two versions: PKaI Nano 2, a base model, and PKaI Nano 2 Instruct, an instruction-tuned chat model.
Nano 2 is a bigger model than Nano 1.2, with 792M parameters across 30 layers, up from 311M across 24, and double the context length at 2048 tokens. Like every Nano before it, both versions were trained from scratch on our own stack.
Just as important, Nano 2 is the base of the Nano 2 line. Further models in this line will build on this checkpoint, so the architecture, data mix, and evaluation setup here are the foundation for what comes next.
What Improved
Across the seven accuracy benchmarks in our suite, Nano 2 averages 53.18, up from 51.69 for Nano 1.2, 48.83 for Nano 1.1, and 43.98 for Nano 1.
Five of the seven accuracy tasks improved over Nano 1.2, and WikiText perplexity fell from 24.13 to 20.65, a 14.4 percent reduction. The largest gain is HellaSwag, up 4.16 points from 39.71 to 43.87, and SciQ now reaches 85.40.
Two metrics moved the other way: WinoGrande dipped from 52.96 to 52.49, and LAMBADA OpenAI from 44.07 to 42.46. We noticed both, and they are part of what we are working to tune in the Nano 2 line.
| Benchmark | PKaI Nano 2 | PKaI Nano 1.2 | PKaI Nano 1.1 | PKaI Nano 1 |
|---|---|---|---|---|
| HellaSwag | 43.87 | 39.71 | 36.52 | 31.02 |
| SciQ | 85.40 | 81.50 | 80.00 | 73.00 |
| PIQA | 67.52 | 65.94 | 64.96 | 59.74 |
| WinoGrande | 52.49 | 52.96 | 52.72 | 52.57 |
| ARC-Easy | 51.77 | 49.92 | 45.45 | 43.35 |
| ARC-Challenge | 28.75 | 27.73 | 28.24 | 24.83 |
| LAMBADA OpenAI | 42.46 | 44.07 | 33.82 | 23.33 |
| WikiText Perplexity | 20.65 | 24.13 | 35.47 | 57.37 |
What Changed
Behind the numbers is a broader, higher-quality training mix than any Nano before it, with more web text, reference-style writing, math and code, scientific papers, and public-domain books.
Everything still goes through the same in-house cleaning, decontamination, and deduplication process before training. What changed is what we let in, not the standards we hold it to.
The Instruct Version
PKaI Nano 2 Instruct starts from the Nano 2 base weights and is fine-tuned on a synthetic multi-turn conversation mix.
Shipping a chat model is a big step for the Nano series. For the first time, a Nano model can follow instructions in a conversation, using our pkai-chat-v1 chat format with the template bundled in tokenizer.json.
This is our first instruct model, and we are proud of where it landed. We are not done improving, though: what we learn from it carries into future instruct versions and Nano 2 models, and those will be even better.
Both checkpoints are on Hugging Face under the Apache 2.0 license: PKaI-Nano-2 for the base model and PKaI-Nano-2-Instruct for the chat model.
Side by Side
To show what the instruct version actually does, we ran the same prompts through an experimental instruct build of Nano 1.2 that we trained but never released, because it was not good enough to ship. Same chat format, same greedy decoding, responses lightly trimmed:
What These Models Are
PKaI Nano 2 is a base language model: it continues text rather than following instructions. PKaI Nano 2 Instruct follows instructions in a chat format, but it is still a small model and has not been safety-tuned.
Like any model this size, both can be inconsistent and factually unreliable. We release them as research milestones and for anyone to use and build on, as a public record of how far the stack has come.
What Comes Next
Nano 2 is our largest release yet, and it is meant to be a foundation, not a finish line. The Nano 2 line will grow from this base with further models, including more capable instruct versions.
They apply everything the Nano series taught us about data quality, scale, and instruction tuning, and they will be smarter and more capable in ways that actually matter. More on those soon.
Credits and Notes
PKaI Nano 2 and PKaI Nano 2 Instruct were trained from scratch by PowderKeg Intelligence.
The models use the Mistral-7B-v0.1 tokenizer from Mistral AI, released under the Apache 2.0 license.
LAMBADA OpenAI refers to the OpenAI-preprocessed LAMBADA test variant used in our evaluation setup.
Benchmarks are reported from our own evaluation setup and should be read as comparative signals between our releases, not as absolute claims.
Model Files
PKaI Nano 2 is released for free under the Apache 2.0 license, with model files available on Hugging Face.