ooai_llm.benchmarks.livecodebench_pro

Exploratory LiveCodeBench Pro leaderboard client.

Purpose:

Provide typed, read-only access to the public LiveCodeBench Pro leaderboard surfaces currently used by the project website.

Design:

• Keep this module explicitly best-effort: the backend endpoints are discoverable from the public frontend but are not documented as a stable API contract.

• Use only the Python standard library for HTTP so the package does not gain a required network-client dependency.

• Normalize the useful result shapes while allowing extra fields because the upstream payload may evolve.

Examples

>>> from ooai_llm.benchmarks.livecodebench_pro import LiveCodeBenchProModel
>>> row = LiveCodeBenchProModel(name="gpt-5", provider="openai", rating=2176)
>>> row.label
'gpt-5'

Attributes

DEFAULT_LIVECODEBENCH_PRO_BASE_URL
LiveCodeBenchProDifficulty
LiveCodeBenchProSort
JsonTransport

Exceptions

LiveCodeBenchProError

Raised when LiveCodeBench Pro data cannot be fetched or parsed.

Classes

LiveCodeBenchProEndpoint	Document one known LiveCodeBench Pro endpoint.
LiveCodeBenchProRatingEvent	One rating event from the LiveCodeBench Pro leaderboard.
LiveCodeBenchProModel	One model row from the LiveCodeBench Pro leaderboard.
LiveCodeBenchProDifficultyRow	One per-difficulty row with validation and pass rates.
LiveCodeBenchProDifficultyResult	Per-difficulty leaderboard payload.
LiveCodeBenchProProblemResult	One problem verdict for a model submission set.
LiveCodeBenchProContestResult	One contest grouping in a model submission set.
LiveCodeBenchProSubmissionsResult	Contest/problem verdicts for one model and difficulty.
LiveCodeBenchProSubmissionDetail	Individual submission detail including generated code when exposed.
LiveCodeBenchProSnapshot	High-level snapshot of leaderboard rows and difficulty views.
LiveCodeBenchProClient	Small client for the current LiveCodeBench Pro leaderboard backend.

Functions

livecodebench_pro_endpoints(...)	Return the currently known LiveCodeBench Pro endpoint surfaces.
list_livecodebench_pro_models(...)	Return LiveCodeBench Pro model rows using a default client.
get_livecodebench_pro_difficulty(...)	Return a LiveCodeBench Pro per-difficulty result using a default client.
get_livecodebench_pro_submissions(...)	Return LiveCodeBench Pro contest/problem verdicts using a default client.
get_livecodebench_pro_submission(...)	Return one LiveCodeBench Pro submission detail using a default client.
get_livecodebench_pro_snapshot(→ LiveCodeBenchProSnapshot)	Return a high-level LiveCodeBench Pro snapshot using a default client.

Module Contents

ooai_llm.benchmarks.livecodebench_pro.DEFAULT_LIVECODEBENCH_PRO_BASE_URL = 'https://webhook.cp-bench.orzzh.com'

ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProDifficulty

ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProSort

ooai_llm.benchmarks.livecodebench_pro.JsonTransport

exception ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProError

Bases: RuntimeError

Raised when LiveCodeBench Pro data cannot be fetched or parsed.

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProEndpoint(/, **data: Any)

Bases: pydantic.BaseModel

Document one known LiveCodeBench Pro endpoint.

Parameters:

• name – Stable local label for the endpoint.

• method – HTTP method.

• path – Backend path.

• description – What the endpoint currently returns.

• query – Required or useful query parameters.

• stability – Stability note for callers.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

method: str = 'GET'

path: str

description: str

query: list[str] = None

stability: str = 'undocumented'

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProRatingEvent(/, **data: Any)

Bases: pydantic.BaseModel

One rating event from the LiveCodeBench Pro leaderboard.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

rating: int | None = None

time: int | None = None

contest_id: int | None = None

name: str | None = None

calc_rating: int | None = None

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProModel(/, **data: Any)

Bases: pydantic.BaseModel

One model row from the LiveCodeBench Pro leaderboard.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

provider: str

status: str | None = None

model_license: str | None = None

organization: str | None = None

rating: int | None = None

display_name: str | None = None

rating_events: list[LiveCodeBenchProRatingEvent] = None

property label: str

Return a display label for the row.

property is_active: bool

Whether the leaderboard marks the row active.

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProDifficultyRow(/, **data: Any)

Bases: LiveCodeBenchProModel

One per-difficulty row with validation and pass rates.

validrate: float | None = None

passrate: float | None = None

property passrate_percent: float | None

Return passrate as a percentage.

property validrate_percent: float | None

Return validrate as a percentage.

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProDifficultyResult(/, **data: Any)

Bases: pydantic.BaseModel

Per-difficulty leaderboard payload.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

difficulty: LiveCodeBenchProDifficulty

llms: list[LiveCodeBenchProDifficultyRow] = None

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProProblemResult(/, **data: Any)

Bases: pydantic.BaseModel

One problem verdict for a model submission set.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

problem_index: str | None = None

problem_link: str | None = None

problem_name: str | None = None

problem_extra: str | None = None

verdict: str | None = None

status: str | None = None

submission_id: str | None = None

property accepted: bool

Whether the verdict indicates an accepted solution.

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProContestResult(/, **data: Any)

Bases: pydantic.BaseModel

One contest grouping in a model submission set.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

contest_title: str | None = None

contest_link: str | None = None

contest_start_time: int | None = None

problems: list[LiveCodeBenchProProblemResult] = None

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProSubmissionsResult(/, **data: Any)

Bases: pydantic.BaseModel

Contest/problem verdicts for one model and difficulty.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

difficulty: LiveCodeBenchProDifficulty

contests: list[LiveCodeBenchProContestResult] = None

property problem_count: int

Return the number of listed problems.

property accepted_count: int

Return the number of accepted listed problems.

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProSubmissionDetail(/, **data: Any)

Bases: pydantic.BaseModel

Individual submission detail including generated code when exposed.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

problem_id: str | None = None

platform: str | None = None

model_name: str | None = None

model_provider: str | None = None

verdict: str | None = None

status: str | None = None

code: str | None = None

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProSnapshot(/, **data: Any)

Bases: pydantic.BaseModel

High-level snapshot of leaderboard rows and difficulty views.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

models: list[LiveCodeBenchProModel] = None

difficulties: dict[LiveCodeBenchProDifficulty, LiveCodeBenchProDifficultyResult] = None

endpoints: list[LiveCodeBenchProEndpoint] = None

notes: list[str] = None

property active_count: int

Return the number of active model rows.

class ooai_llm.benchmarks.livecodebench_pro.LiveCodeBenchProClient(*, base_url: str = DEFAULT_LIVECODEBENCH_PRO_BASE_URL, timeout: float = 30, transport: JsonTransport | None = None)

Small client for the current LiveCodeBench Pro leaderboard backend.

Parameters:

• base_url – Backend URL. Defaults to the public website backend observed in the frontend bundle.

• timeout – HTTP timeout in seconds.

• transport – Optional testing hook. It receives the fully built URL and returns a decoded JSON-compatible object.

base_url = ''

timeout = 30

list_models(*, status: str | None = None, providers: collections.abc.Sequence[str] | None = None, organizations: collections.abc.Sequence[str] | None = None, query: str | None = None, sort_by: LiveCodeBenchProSort = 'rating', descending: bool = True, limit: int | None = None) → list[LiveCodeBenchProModel]

Return leaderboard model rows with local filters applied.

get_difficulty(difficulty: LiveCodeBenchProDifficulty, *, providers: collections.abc.Sequence[str] | None = None, organizations: collections.abc.Sequence[str] | None = None, query: str | None = None, sort_by: LiveCodeBenchProSort = 'rating', descending: bool = True, limit: int | None = None) → LiveCodeBenchProDifficultyResult

Return a per-difficulty leaderboard view.

get_submissions(*, model_name: str, model_provider: str, difficulty: LiveCodeBenchProDifficulty) → LiveCodeBenchProSubmissionsResult

Return contest/problem verdicts for one model and difficulty.

get_submission(submission_id: str) → LiveCodeBenchProSubmissionDetail

Return an individual submission detail.

snapshot(*, include_difficulties: bool = True, active_only: bool = False, limit: int | None = None) → LiveCodeBenchProSnapshot

Return a high-level snapshot of currently exposed data.

ooai_llm.benchmarks.livecodebench_pro.livecodebench_pro_endpoints() → list[LiveCodeBenchProEndpoint]

Return the currently known LiveCodeBench Pro endpoint surfaces.

ooai_llm.benchmarks.livecodebench_pro.list_livecodebench_pro_models(*, client: LiveCodeBenchProClient | None = None, **kwargs: Any) → list[LiveCodeBenchProModel]

Return LiveCodeBench Pro model rows using a default client.

ooai_llm.benchmarks.livecodebench_pro.get_livecodebench_pro_difficulty(difficulty: LiveCodeBenchProDifficulty, *, client: LiveCodeBenchProClient | None = None, **kwargs: Any) → LiveCodeBenchProDifficultyResult

Return a LiveCodeBench Pro per-difficulty result using a default client.

ooai_llm.benchmarks.livecodebench_pro.get_livecodebench_pro_submissions(*, model_name: str, model_provider: str, difficulty: LiveCodeBenchProDifficulty, client: LiveCodeBenchProClient | None = None) → LiveCodeBenchProSubmissionsResult

Return LiveCodeBench Pro contest/problem verdicts using a default client.

ooai_llm.benchmarks.livecodebench_pro.get_livecodebench_pro_submission(submission_id: str, *, client: LiveCodeBenchProClient | None = None) → LiveCodeBenchProSubmissionDetail

Return one LiveCodeBench Pro submission detail using a default client.

ooai_llm.benchmarks.livecodebench_pro.get_livecodebench_pro_snapshot(*, client: LiveCodeBenchProClient | None = None, **kwargs: Any) → LiveCodeBenchProSnapshot

Return a high-level LiveCodeBench Pro snapshot using a default client.