Introducing PKaI Nano 1: Our First Language Model

A First Foundation

Today we're excited to introduce PKaI Nano 1, our first language model trained from scratch.

PKaI Nano 1 is a 120M-parameter base model built as a first step toward our own model-training stack: our own tooling, our own training pipeline, our own evaluation workflow, and a growing understanding of what it takes to turn raw data and compute into a usable language model.

This is not a giant frontier model, and we're not pretending it is. It is something more important for us: a working foundation. A small model we trained ourselves, measured carefully, improved through follow-up training, and can now share as the first public checkpoint in the PKaI model family.

Why Nano?

We wanted our first release to be small enough to study closely, iterate on quickly, and make practical decisions around training quality rather than just scale.

At 120M parameters, Nano 1 sits in a size class where every choice matters: dataset quality, optimizer behavior, schedule, context length, evaluation setup, and even small continuation runs can visibly change the final result. That made it the right first model for us.

Benchmark Highlights

Across six benchmark accuracy tasks, PKaI Nano 1 reaches GPT-2 Small-level aggregate performance in our evaluation setup: 47.42 for PKaI Nano 1 versus 47.11 for GPT-2 Small.

That is a small margin, but an exciting one for a first from-scratch model. Nano 1 also outperforms GPT-2 Small on ARC-Easy, ARC-Challenge, and WinoGrande in this evaluation mix.

GPT-2 Small remains stronger on several other metrics, including HellaSwag, PIQA, SciQ, and WikiText perplexity, so we view this as parity on aggregate accuracy, not a claim of overall superiority.

Built With Our Own Tools

One of the most important parts of this release is that PKaI Nano 1 was trained using our own tooling.

That includes the training workflow, checkpointing, evaluation support, export process, and the practical infrastructure around running and comparing experiments. This model helped us stress-test the system, find rough edges, improve the training process, and better understand what we need for larger runs.

In other words, Nano 1 is both a model and a proof point for the pipeline behind it. It is the first result from a stack we intend to keep improving.

What This Model Is

PKaI Nano 1 is a base language model, not an instruction-tuned assistant. It is designed to continue text, not follow chat-style instructions reliably.

Like other small base models, it can be inconsistent, factually unreliable, and sensitive to prompt formatting. We're releasing it as a research and development milestone: a compact model trained from scratch, evaluated transparently, and useful as a baseline for what comes next.

What Comes Next

Nano 1 gave us something valuable: signal.

It showed us where the training worked, where the model is still weak, and how much room there is to improve with better scale, longer context, stronger data mixtures, and more refined training runs.

This is our first model, not our final one. We're already working on what comes next.

Credits and Notes

PKaI Nano 1 was trained from scratch by PowderKeg Intelligence.

The model uses the Mistral-7B-v0.1 tokenizer from Mistral AI, released under the Apache 2.0 license.

GPT-2 Small is by OpenAI. Benchmark comparisons are provided for context only; PKaI Nano 1 is not affiliated with or endorsed by OpenAI.

Benchmarks are reported from our evaluation setup and should be interpreted as comparative signals rather than absolute claims.