fabricatio_capabilities.capabilities.rating

A module that provides functionality to rate tasks based on a rating manual and score range.

Classes

Rating

A class that provides functionality to rate tasks based on a rating manual and score range.

Module Contents

class fabricatio_capabilities.capabilities.rating.Rating(/, **data: Any)

Bases: fabricatio_core.capabilities.propose.Propose, abc.ABC

A class that provides functionality to rate tasks based on a rating manual and score range.

References

Lu X, Li J, Takeuchi K, et al. AHP-powered LLM reasoning for multi-criteria evaluation of open-ended responses[A/OL]. arXiv, 2024. DOI: 10.48550/arXiv.2410.01246.

async rate_fine_grind(to_rate: str | List[str], rating_manual: Dict[str, str], score_range: Tuple[float, float], **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[Dict[str, float]]]) → Dict[str, float] | List[Dict[str, float]] | List[Dict[str, float] | None] | None

Rate a given string based on a rating manual and score range.

Parameters:

  • to_rate (str) – The string to be rated.

  • rating_manual (Dict[str, str]) – A dictionary containing the rating criteria.

  • score_range (Tuple[float, float]) – A tuple representing the valid score range.

  • **kwargs (Unpack[ValidateKwargs]) – Additional keyword arguments for the LLM usage.

Returns:

A dictionary with the ratings for each dimension.

Return type:

Dict[str, float]

async rate(to_rate: str, topic: str, criteria: Set[str], manual: Dict[str, str] | None = None, score_range: Tuple[float, float] = (0.0, 1.0), **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[Dict[str, float]]]) → Dict[str, float]
async rate(to_rate: List[str], topic: str, criteria: Set[str], manual: Dict[str, str] | None = None, score_range: Tuple[float, float] = (0.0, 1.0), **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[Dict[str, float]]]) → List[Dict[str, float]]

Rate a given string or a sequence of strings based on a topic, criteria, and score range.

Parameters:

  • to_rate (Union[str, List[str]]) – The string or sequence of strings to be rated.

  • topic (str) – The topic related to the task.

  • criteria (Set[str]) – A set of criteria for rating.

  • manual (Optional[Dict[str, str]]) – A dictionary containing the rating criteria. If not provided, then this method will draft the criteria automatically.

  • score_range (Tuple[float, float], optional) – A tuple representing the valid score range. Defaults to (0.0, 1.0).

  • **kwargs (Unpack[ValidateKwargs]) – Additional keyword arguments for the LLM usage.

Returns:

A dictionary with the ratings for each criterion if a single string is provided, or a list of dictionaries with the ratings for each criterion if a sequence of strings is provided.

Return type:

Union[Dict[str, float], List[Dict[str, float]]]

async draft_rating_manual(topic: str, criteria: Set[str] | None = None, **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[Dict[str, str]]]) → Dict[str, str] | None

Drafts a rating manual based on a topic and dimensions.

Parameters:

  • topic (str) – The topic for the rating manual.

  • criteria (Optional[Set[str]], optional) – A set of criteria for the rating manual. If not specified, then this method will draft the criteria automatically.

  • **kwargs (Unpack[ValidateKwargs]) – Additional keyword arguments for the LLM usage.

Returns:

A dictionary representing the drafted rating manual.

Return type:

Dict[str, str]

async draft_rating_criteria(topic: str, criteria_count: pydantic.NonNegativeInt = 0, **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[Set[str]]]) → Set[str] | None

Drafts rating dimensions based on a topic.

Parameters:

  • topic (str) – The topic for the rating dimensions.

  • criteria_count (NonNegativeInt, optional) – The number of dimensions to draft, 0 means no limit. Defaults to 0.

  • **kwargs (Unpack[ValidateKwargs]) – Additional keyword arguments for the LLM usage.

Returns:

A set of rating dimensions.

Return type:

Set[str]

async draft_rating_criteria_from_examples(topic: str, examples: List[str], m: pydantic.NonNegativeInt = 0, reasons_count: pydantic.PositiveInt = 2, criteria_count: pydantic.PositiveInt = 5, **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs]) → Set[str] | None

Asynchronously drafts a set of rating criteria based on provided examples.

This function generates rating criteria by analyzing examples and extracting reasons for comparison, then further condensing these reasons into a specified number of criteria.

Parameters:

  • topic (str) – The subject topic for the rating criteria.

  • examples (List[str]) – A list of example texts to analyze.

  • m (NonNegativeInt, optional) – The number of examples to sample from the provided list. Defaults to 0 (no sampling).

  • reasons_count (PositiveInt, optional) – The number of reasons to extract from each pair of examples. Defaults to 2.

  • criteria_count (PositiveInt, optional) – The final number of rating criteria to draft. Defaults to 5.

  • **kwargs (Unpack[ValidateKwargs]) – Additional keyword arguments for validation.

Returns:

A set of drafted rating criteria.

Return type:

Set[str]

Warning

Since this function uses pairwise comparisons, it may not be suitable for large lists of examples. For that reason, consider using a smaller list of examples or setting m to a non-zero value smaller than the length of the examples.

async drafting_rating_weights_klee(topic: str, criteria: Set[str], **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[float]]) → Dict[str, float]

Drafts rating weights for a given topic and criteria using the Klee method.

Parameters:

  • topic (str) – The topic for the rating weights.

  • criteria (Set[str]) – A set of criteria for the rating weights.

  • **kwargs (Unpack[ValidateKwargs]) – Additional keyword arguments for the LLM usage.

Returns:

A dictionary representing the drafted rating weights for each criterion.

Return type:

Dict[str, float]

async composite_score(topic: str, to_rate: List[str], criteria: Set[str] | None = None, weights: Dict[str, float] | None = None, manual: Dict[str, str] | None = None, approx: bool = False, **kwargs: Unpack[fabricatio_core.models.kwargs_types.ValidateKwargs[Dict[str, float]]]) → List[float]

Calculates the composite scores for a list of items based on a given topic and criteria.

Parameters:

  • topic (str) – The topic for the rating.

  • to_rate (List[str]) – A list of strings to be rated.

  • criteria (Optional[Set[str]]) – A set of criteria for the rating. Defaults to None.

  • weights (Optional[Dict[str, float]]) – A dictionary of rating weights for each criterion. Defaults to None.

  • manual (Optional[Dict[str, str]]) – A dictionary of manual ratings for each item. Defaults to None.

  • approx (bool) – Whether to use approximate rating criteria. Defaults to False.

  • **kwargs (Unpack[ValidateKwargs]) – Additional keyword arguments for the LLM usage.

Returns:

A list of composite scores for the items.

Return type:

List[float]

async best(candidates: List[str], k: int = 1, **kwargs: Unpack[fabricatio_capabilities.models.kwargs_types.CompositeScoreKwargs]) → List[str]
async best(candidates: List[T], k: int = 1, **kwargs: Unpack[fabricatio_capabilities.models.kwargs_types.CompositeScoreKwargs]) → List[T]

Choose the best candidates from the list of candidates based on the composite score.

Parameters:

  • k (int) – The number of best candidates to choose.

  • candidates (List[str]) – A list of candidates to choose from.

  • **kwargs (CompositeScoreKwargs) – Additional keyword arguments for the composite score calculation.

Returns:

The best candidates.

Return type:

List[str]