Python API reference

syntaxgym.compute_surprisals(model: lm_zoo.models.Model, suite)

Compute per-region surprisals for a language model on the given suite.

Parameters

  • model – An LM Zoo Model.

  • suite_file – A path or open file stream to a suite JSON file, or an already loaded suite dict

Returns

An evaluated test suite dict — a copy of the data from suite_file, now including per-region surprisal data

syntaxgym.evaluate(suite, return_df=True)

Evaluate prediction results on the given suite. The suite must contain surprisal estimates for all regions.

class syntaxgym.suite.Suite(condition_names, region_names, items, predictions, meta)

A test suite represents a targeted syntactic evaluation experiment.

For more information, see The SyntaxGym architecture.

Variables

  • condition_names – A list of condition name strings

  • region_names – An ordered list of region name strings

  • items – An array of item dicts, represented just as in a suite JSON representation. See Test suite JSON representation for more information.

  • predictions – A list of Prediction objects.

  • meta – A dict of metadata about this suite, represented just as in a suite JSON representation. See Test suite JSON representation for more information.

as_dataframe(metric: Optional[str] = None) → pandas.core.frame.DataFrame

Convert self to a data frame describing per-region surprisals. Only usable / sensible for Suite instances which have been evaluated with surprisals.

Returns

  • item_number

  • condition_name

  • region_number

  • content

  • metric_value: per-region metric, specified in Suite meta or

    overridden with metric argument

  • oovs: comma-separated list of OOV items

Return type

A long Pandas DataFrame, one row per region, with columns

evaluate_predictions() → Dict[int, Dict[syntaxgym.prediction.Prediction, bool]]

Compute prediction results for each item.

Returns

a nested dict mapping ``(item_number => prediction =>

prediction_result)``

Return type

results

class syntaxgym.prediction.Prediction(idx: int, formula: Union[str, syntaxgym.prediction.BinaryOp], metric: str)

Predictions state expected relations between language model surprisal measures in different regions and conditions of a test suite. For more information, see The SyntaxGym architecture.

__call__(item)

Evaluate the prediction on the given item dict representation. For more information on item representations, see Test suite JSON representation.

__init__(idx: int, formula: Union[str, syntaxgym.prediction.BinaryOp], metric: str)
Parameters

  • idx – A unique prediction ID. This is only relevant for serialization.

  • formula – A string representation of the prediction formula, or an already parsed formula. For more information, see The SyntaxGym architecture.

  • metric – Metric for aggregating surprisals within regions.

as_dict()

Serialize as a prediction dictionary representation (see Test suite JSON representation).

classmethod from_dict(pred_dict, idx: int, metric: str)

Parse from a prediction dictionary representation (see Test suite JSON representation).

property referenced_regions

Get a set of the regions referenced by this formula. Each item is a tuple of the form (condition_name, region_number).