fabricatio_judge.capabilities.advanced_judge

The Capabilities module for advanced judging.

Classes

EvidentlyJudge

A class that judges the evidence and makes a final decision.

VoteLLMConfig

A class that uses a language model to vote on the evidence.

VoteJudge

A class that votes on the evidence and makes a final decision.

Module Contents

class fabricatio_judge.capabilities.advanced_judge.EvidentlyJudge(/, **data: Any)

Bases: fabricatio_core.capabilities.propose.Propose, abc.ABC

A class that judges the evidence and makes a final decision.

async evidently_judge(prompt: str, **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[fabricatio_judge.models.judgement.JudgeMent]]) → fabricatio_judge.models.judgement.JudgeMent | None
async evidently_judge(prompt: List[str], **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[fabricatio_judge.models.judgement.JudgeMent]]) → List[fabricatio_judge.models.judgement.JudgeMent] | List[fabricatio_judge.models.judgement.JudgeMent | None] | None

Judge the evidence and make a final decision.

class fabricatio_judge.capabilities.advanced_judge.VoteLLMConfig(/, **data: Any)

Bases: fabricatio_core.models.generic.ScopedConfig

A class that uses a language model to vote on the evidence.

vote_llm: Dict[float, fabricatio_core.models.kwargs_types.ValidateKwargs[fabricatio_judge.models.judgement.JudgeMent]]

Config that specify the models used to make a decision.

vote_pass_threshold: float | None = None

The threshold for passing the vote.

class fabricatio_judge.capabilities.advanced_judge.VoteJudge(/, **data: Any)

Bases: EvidentlyJudge, VoteLLMConfig, abc.ABC

A class that votes on the evidence and makes a final decision.

async vote_judge(prompt: str, vote_pass_threshold: float | None = None, **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[fabricatio_judge.models.judgement.JudgeMent]]) → bool | None
async vote_judge(prompt: List[str], vote_pass_threshold: float | None = None, **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[fabricatio_judge.models.judgement.JudgeMent]]) → List[bool | None] | None

Vote on the evidence and make a final decision.

Parameters:

  • prompt (str | List[str]) – The input prompt(s) to be judged.

  • vote_pass_threshold (Optional[float], optional) – The threshold for passing the vote. Defaults to None.

  • **kwargs (Unpack[ValidateKwargs[JudgeMent]]) – Additional keyword arguments for configuring the judgment process.

Returns:

A list of boolean results indicating

whether each vote passed, or None if no prompt is provided.

Return type:

Optional[List[bool] | List[bool | None]]

static resolve_pass(weights: List[float], judgments: List[fabricatio_judge.models.judgement.JudgeMent], vote_pass_threshold: float) → bool

Determine if the vote passes based on the provided weights and judgments.

Parameters:

  • weights (List[float]) – The weights assigned to each judge.

  • judgments (List[JudgeMent]) – The list of judgments from the judges.

  • vote_pass_threshold (float) – The threshold for a successful vote.

Returns:

True if the weighted sum of judgments meets or exceeds the threshold,

False otherwise.

Return type:

bool