Welcome to LM Zoo¶
The Language Model Zoo (LM Zoo) is an open-source repository of
state-of-the-art language models, designed to support black-box access to model
predictions and representations. It provides the command line tool
a standard interface for interacting with language models.
You can use
lm-zoo to extract word-level surprisal data for natural
$ wget https://cpllab.github.io/lm-zoo/metamorphosis.txt -O metamorphosis.txt $ head -n3 metamorphosis.txt One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin. He lay on his armour-like back, and if he lifted his head a little he could see his brown belly, slightly domed and divided by arches into stiff sections. The bedding was hardly able to cover it and seemed ready to slide off any moment. $ lm-zoo get-surprisals ngram metamorphosis.txt sentence_id token_id token surprisal 1 1 one 7.76847 1 2 morning 9.40638 1 3 , 1.05009 1 4 when 7.08489 1 5 gregor 18.8963 1 6 <unk> 4.27466 ... $ lm-zoo get-surprisals GRNN metamorphosis.txt sentence_id token_id token surprisal 1 1 One 0.000000 1 2 morning 11.585580 1 3 , 2.359496 1 4 when 6.319129 1 5 Gregor 18.608147 1 6 <unk> 3.706054 ...
For more features and installation instructions, see our Quickstart guide.
LM Zoo is maintained by the MIT Computational Psycholinguistics Laboratory.