Skip to content

[sampling parameters] new parameters #2220

@k33g

Description

@k33g

Some models providers allow these sampling parameters:
- top_k (int)
- repetition_penalty (float)

Is it possible to add these parameters?

unless it is possible to use:

provider_opts:
    top_k: 20
    repetition_penalty: 1.05

?

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)kind/enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions