Everything called the creativity of a language model comes down to dividing every score by a number T before the softmax. A small T sharpens the distribution into a spike, the top token wins nearly every time and the prose turns confident but monotone; a large T flattens it, unlikely tokens get their chance and the prose grows varied but prone to derailing. Before that, each token emits a query and a key, and how well they match becomes an attention weight through softmax, enough for a pronoun at the end of a sentence to reach straight back to a noun at its start. The model looks nothing up: it predicts the next token, appends it, and predicts again.
Everything called the creativity of a language model comes down to dividing every score by a number T before the softmax. A small T sharpens the distribution into a spike, the top token wins nearly every time and the prose turns confident but monotone; a large T flattens it, unlikely tokens get their chance and the prose grows varied but prone to derailing. Before that, each token emits a query and a key, and how well they match becomes an attention weight through softmax, enough for a pronoun at the end of a sentence to reach straight back to a noun at its start. The model looks nothing up: it predicts the next token, appends it, and predicts again.
Related topics in Mind & Machine