karpathy/nanochat

It's also a great base for experimentation. If you have an idea for an architecture improvement you can try it for $36 on the 20 layer nanocode setting, then for another $200 see how it holds up on the "full scale" nanocode Kaparthy's notes on improving nanochat [1] are one of my favorite blog-like things to read. Really neat to see which features have how much influence, and how the scaling laws evolve as you improve the architecture There's also modded-nanogpt which turns the same kind of experimentation into a training speedrun (and maybe loses some rigor on the way) [2] 1 https://github.com/karpathy/nanochat/blob/master/dev/LOG.md 2 https://github.com/kellerjordan/modded-nanogpt

Karpathy's contribution to teaching around deep learning is just immense. He's got a mountain of fantastic material from short articles like this, longer writing like https://karpathy.github.io/2015/05/21/rnn-effectiveness/ (on recurrent neural networks) and all of the stuff on YouTube. Plus his GitHub. The recently released nanochat https://github.com/karpathy/nanochat is fantastic. Having minimal, understandable and complete examples like that is invaluable for anyone who really wants to understand this stuff.

You're missing the point. Karpathy has other projects, e.g. : https://github.com/karpathy/nanochat You can train a model with GPT-2 level of capability for $20-$100. But, guess what, that's exactly what thousands of AI researchers have been doing for the past 5+ years. They've been training smallish models. And while these smallish models might be good for classification and whatnot, people strongly prefer big-ass frontier models for code generation.

The Olmo team is AFAIK the only SOTA-ish model that has fully open source code and data. Their report is fantastic: https://www.datocms-assets.com/64837/1763662397-1763646865-olmo_3_technical_report-1.pdf It should give you an idea of how hard it is to do a SOTA model from scratch! If you relax the SOTA aspect, Karpathy's nanochat has you covered: https://github.com/karpathy/nanochat

…dy to go. If you have weaker hardware (say an 8GB nVidia GPU, which is what I have) you're going to be limited to fine tuning small models or torturing yourself working the GPU for days per iteration trying to run things like https://github.com/karpathy/nanochat , which is hardly an educational experience. Renting cloud GPUs is expensive, and at this age the most I could muster up for experimentation is probably $100 or so, which only gets me 25 or so hours on a B200 which just isn't enough. So why would I bother myself with this if I'm already at a disadvantage because of not having access to the right hardware and when surely there are better ways to spend my time? I concluded the only way…

As it says on the GitHub page, [1] this is like a kindergartner in some ways. I just started with the same question as on that page, “why is the sky blue?” and followed up with why it’s orange during sunrise or sunset. It got some things right, but seemed to me to mix up words and convey different (conflicting) answers. So I finally told it the following. > Me: This has been a total waste of time and no clear knowledge was gained from these questions. You seem to be confused, have been mixing things up and making things up. > Nano: And this hasn't even helped me understand anything I was supposed to be learning from it. I've been making mistakes, and I'm not getting anywhere with these "a…