syntaxgym commands

The following commands are available in the syntaxgym binary.

Most of these commands take references to a “model.” This can be a reference to any valid LM Zoo model.

syntaxgym

syntaxgym [OPTIONS] COMMAND [ARGS]...

Options

-v, --verbose

compute-surprisals

Compute per-region surprisals for a language model on the given test suite

syntaxgym compute-surprisals [OPTIONS] MODEL SUITE_FILE

Options

--checkpoint <checkpoint>

--tabular_results, --json_results

If –tabular_results, outputs a TSV structured like the output of syntaxgym.suite.Suite.as_dataframe().

Arguments

MODEL

Required argument

SUITE_FILE

Required argument

evaluate

Evaluate prediction results on the given test suite

syntaxgym evaluate [OPTIONS] SUITE_FILE

Arguments

SUITE_FILE

Required argument

run

Run the model and test suite through the full pipeline

syntaxgym run [OPTIONS] MODEL SUITE_FILE

Options

--checkpoint <checkpoint>

Arguments

MODEL

Required argument

SUITE_FILE

Required argument