Skip to content
Back to AI Research
Release

Introducing PKaI Nano 2: A Bigger Nano and Our First Instruct Model

November 18, 20265 min read

A bigger Nano at 792M parameters with 2048-token context and a broader training mix, released as a base model and our first instruction-tuned chat model. It also serves as the base for the Nano 2 line.

Nano
Model class
792M
Parameters
53.18
Seven-task average
6/8
Suite improvements

The Next Nano

Today we are releasing PKaI Nano 2, the fourth model in the Nano series and the first one we are shipping in two versions: PKaI Nano 2, a base model, and PKaI Nano 2 Instruct, an instruction-tuned chat model.

Nano 2 is a bigger model than Nano 1.2, with 792M parameters across 30 layers, up from 311M across 24, and double the context length at 2048 tokens. Like every Nano before it, both versions were trained from scratch on our own stack.

Just as important, Nano 2 is the base of the Nano 2 line. Further models in this line will build on this checkpoint, so the architecture, data mix, and evaluation setup here are the foundation for what comes next.

What Improved

Across the seven accuracy benchmarks in our suite, Nano 2 averages 53.18, up from 51.69 for Nano 1.2, 48.83 for Nano 1.1, and 43.98 for Nano 1.

Five of the seven accuracy tasks improved over Nano 1.2, and WikiText perplexity fell from 24.13 to 20.65, a 14.4 percent reduction. The largest gain is HellaSwag, up 4.16 points from 39.71 to 43.87, and SciQ now reaches 85.40.

Two metrics moved the other way: WinoGrande dipped from 52.96 to 52.49, and LAMBADA OpenAI from 44.07 to 42.46. We noticed both, and they are part of what we are working to tune in the Nano 2 line.

+1.49 pts
Seven-task average · 53.18 vs 51.69
+4.16 pts
HellaSwag · 43.87 vs 39.71
+3.90 pts
SciQ · 85.40 vs 81.50
Accuracy comparison
Higher is better. WikiText perplexity is listed separately below.
Seven-task accuracy average
Nano 2
53.18
Nano 1.2
51.69
Nano 1.1
48.83
Nano 1
43.98
HellaSwag
Nano 2
43.87
Nano 1.2
39.71
Nano 1.1
36.52
Nano 1
31.02
SciQ
Nano 2
85.40
Nano 1.2
81.50
Nano 1.1
80.00
Nano 1
73.00
PIQA
Nano 2
67.52
Nano 1.2
65.94
Nano 1.1
64.96
Nano 1
59.74
WinoGrande
Nano 2
52.49
Nano 1.2
52.96
Nano 1.1
52.72
Nano 1
52.57
ARC-Easy
Nano 2
51.77
Nano 1.2
49.92
Nano 1.1
45.45
Nano 1
43.35
ARC-Challenge
Nano 2
28.75
Nano 1.2
27.73
Nano 1.1
28.24
Nano 1
24.83
LAMBADA OpenAI
Nano 2
42.46
Nano 1.2
44.07
Nano 1.1
33.82
Nano 1
23.33
BenchmarkPKaI Nano 2PKaI Nano 1.2PKaI Nano 1.1PKaI Nano 1
HellaSwag43.8739.7136.5231.02
SciQ85.4081.5080.0073.00
PIQA67.5265.9464.9659.74
WinoGrande52.4952.9652.7252.57
ARC-Easy51.7749.9245.4543.35
ARC-Challenge28.7527.7328.2424.83
LAMBADA OpenAI42.4644.0733.8223.33
WikiText Perplexity20.6524.1335.4757.37
Lower is better for WikiText perplexity.

What Changed

Behind the numbers is a broader, higher-quality training mix than any Nano before it, with more web text, reference-style writing, math and code, scientific papers, and public-domain books.

Everything still goes through the same in-house cleaning, decontamination, and deduplication process before training. What changed is what we let in, not the standards we hold it to.

The Instruct Version

PKaI Nano 2 Instruct starts from the Nano 2 base weights and is fine-tuned on a synthetic multi-turn conversation mix.

Shipping a chat model is a big step for the Nano series. For the first time, a Nano model can follow instructions in a conversation, using our pkai-chat-v1 chat format with the template bundled in tokenizer.json.

This is our first instruct model, and we are proud of where it landed. We are not done improving, though: what we learn from it carries into future instruct versions and Nano 2 models, and those will be even better.

Both checkpoints are on Hugging Face under the Apache 2.0 license: PKaI-Nano-2 for the base model and PKaI-Nano-2-Instruct for the chat model.

Side by Side

To show what the instruct version actually does, we ran the same prompts through an experimental instruct build of Nano 1.2 that we trained but never released, because it was not good enough to ship. Same chat format, same greedy decoding, responses lightly trimmed:

PKaI Nano 2 Instruct
Nano 1.2 experimental instruct
Prompt
What is the capital of Australia? Many people guess Sydney, so explain why.
The capital of Australia is Canberra, a city located in the Australian Capital Territory. ...
The capital of Australia is Sydney, which is a city in the Australian Capital Territory. ... (it then repeats itself in circles)
Prompt
Write a Python function second_largest(nums) that returns the second largest number in a list of integers. Include one example call.
def second_largest(nums): return sorted(nums, reverse=True)[:2] Example call: nums = [4, 2, 9, 6, 5, 1] print(second_largest(nums))
def second_largest(nums): return second_largest(nums) ... (the same recursive call, repeated until the token limit)
Prompt
List exactly three ways to reduce food waste at home, as a bulleted list.
Here are three ways to reduce food waste at home: 1. Eliminate Non-Meat Foods: ... 2. Eliminate Dairy Products: ... 3. Eliminate Processed Foods: ... (exactly three items)
1. Use a reusable shopping bag to carry your groceries. 2. Use a compostable bag to recycle your food waste. 3. Use a reusable water bottle to save water. 4. ... (runs to 11 items, repeating the same lines)
Prompt
Will it rain in Miami, Florida tomorrow? If you are not sure, say so.
As an AI, I don't have real-time access to current weather conditions or the ability to provide current forecasts. ... If you're unsure about the weather in Miami, Florida, please check a reliable weather forecast or the latest news from your local weather station.
Yes, it will rain in Miami, Florida tomorrow. The weather forecast predicts a high chance of rain for today.
Same prompts, same chat format and decoding. Model output in regular type; trims and editorial notes in muted italics.

What These Models Are

PKaI Nano 2 is a base language model: it continues text rather than following instructions. PKaI Nano 2 Instruct follows instructions in a chat format, but it is still a small model and has not been safety-tuned.

Like any model this size, both can be inconsistent and factually unreliable. We release them as research milestones and for anyone to use and build on, as a public record of how far the stack has come.

What Comes Next

Nano 2 is our largest release yet, and it is meant to be a foundation, not a finish line. The Nano 2 line will grow from this base with further models, including more capable instruct versions.

They apply everything the Nano series taught us about data quality, scale, and instruction tuning, and they will be smarter and more capable in ways that actually matter. More on those soon.

Credits and Notes

PKaI Nano 2 and PKaI Nano 2 Instruct were trained from scratch by PowderKeg Intelligence.

The models use the Mistral-7B-v0.1 tokenizer from Mistral AI, released under the Apache 2.0 license.

LAMBADA OpenAI refers to the OpenAI-preprocessed LAMBADA test variant used in our evaluation setup.

Benchmarks are reported from our own evaluation setup and should be read as comparative signals between our releases, not as absolute claims.

Model Files

PKaI Nano 2 is released for free under the Apache 2.0 license, with model files available on Hugging Face.