Language model study

Language model study is code from his study of language-model fundamentals in spring 2025, following material from Carnegie Mellon University's Advanced NLP course (spring 2025). It starts where language modelling starts: an autoregressive bigram model over characters, written in Python.

The bigram model

The model reads a list of names, wraps each in start and end markers, and counts how often each character follows each other one. Normalised, those counts give the probability of the next character given the current one, and sampling from them one character at a time produces new name-like words. It is the simplest possible autoregressive model, and the starting point for neural ones. A test folder holds experiments for running on a GPU.

See also

• Machine learning and data
• Natural language processing