すべてのプロジェクト

typed-decision-bench

kyr0/typed-decision-bench

Large typed decision model evaluation benchmark and novel calibration standard (`calibration.json`) for any typed decision inference system, covering 275 distinct use-cases and 27,598 individual decisions for Jev-like open typed decision models.

Python
スター
0
フォーク
0

掲載理由

typed-decision-benchはSystem One型付き決定APIを実装するシステム向けの評価・調整スイートです。READMEは大規模な保留データを記載していますが、結果は再現していません。

関連プロジェクト

  • typed-decision-bench

    Bench of typed decision models: Jev vs OpenJev vs Laya, small local LLMs and cheap hosted LLMs on the same zero-shot classification tasks, in English and French.

    共通トピック: benchmark, calibration

  • snake-arena-jev-vs-llms

    How many decisions can a model make in a minute, and what do they cost? Jev, a System One decision model, raced against six LLMs on the same Snake boards.

    共通トピック: benchmark, calibration

  • jev-sim

    Jev-compatible /v1/systemone server reading typed decisions from LLM logits, benchmarked against TypeSafe's Jev on the same items via JevBench

    共通トピック: benchmark, calibration