# Thinking Budget - FastDeploy: Large Language Model Deployment

Skip to content

# Thinking Budget Logits Processor

## Overview

`ThinkingBudgetLogitsProcessor` limits the number of tokens generated inside the `<think> ... </think>` segment. When the budget is reached, it terminates thinking by forcing `</think>`. If `think_stop_sentence` is configured, it forces the custom sentence first and then `</think>`.

## When to Use

- Models that emit `<think>`/`</think>` tokens for reasoning.
- You need a hard cap on thinking length without changing sampling logic.

## How It Works

1. Request-side precompute (DataProcessor): when a request includes `thinking_budget`, the prompt token ids are scanned to determine whether thinking has started, whether it already ended, and how many tokens are already inside the thinking section.
2. Per-step update: during decoding, the processor tracks `last_token_id` and `tokens_after_start`.
3. Budget enforcement: once the budget is reached, it forces `</think>` directly. If `think_stop_sentence` is configured, it forces that sentence first and then `</think>`.

## Requirements

- The model must provide valid token ids for `think_start_id` and `think_end_id` (via `ModelConfig`).
- If either of these ids is invalid, the processor is disabled and `thinking_budget` will not take effect.

## Request Parameters

- `thinking_budget` (int, required to enable): maximum number of decode-time tokens after `<think>` before forced termination.
- `think_stop_sentence` (string, optional): a literal custom sentence that will be tokenized on the request side and enforced near the budget boundary.

## Operator-Level vs LogitsProcessor

FastDeploy has two ways to limit thinking length:

- Operator-level limit (`enable_thinking=true` + `reasoning_max_tokens`):
- Implemented in built-in post-processing kernels.
- Lower overhead and better throughput under high concurrency.
- Best for simple "cap the thinking length" use cases.
- `ThinkingBudgetLogitsProcessor` (`logits_processors_args.thinking_budget`):
- Implemented in per-step Python logits processing.
- Supports flexible controls, such as `think_stop_sentence` (custom inserted sentence before ending thinking).
- Higher runtime overhead under high concurrency compared with operator-level limit.

In short:

- If you only need a hard cap on thinking length, prefer `reasoning_max_tokens`.
- If you need custom behavior (for example, inserting a custom sentence before `</think>`), use `ThinkingBudgetLogitsProcessor`.

## Practical guidance

`reasoning_max_tokens` and `thinking_budget` are not mutually exclusive in current implementation. If both are configured for the same request, both constraints can take effect, and whichever triggers first will end the thinking phase.

- To use operator-level-only behavior: this is request-level config only. Set `enable_thinking=true` and `reasoning_max_tokens` in request, and do not set `thinking_budget`.
- To use logits-processor-only behavior (especially with `think_stop_sentence`): this requires service-level + request-level config. Start service with `--logits-processors ThinkingBudgetLogitsProcessor`, and set `thinking_budget` (and optional `think_stop_sentence`) in `logits_processors_args`; leave `reasoning_max_tokens` unset.
- `thinking_budget` itself does not require `enable_thinking=true`.
- If an ERNIE chat template already appends `<think>` in the prompt, `thinking_budget` should still take effect; it does not require the model to emit another `<think>` during decoding.
- Avoid enabling both for strict custom sentence insertion requirements, because operator-level termination may cut the custom sentence path earlier.

## Online Usage

### 1. Start service

    python -m fastdeploy.entrypoints.openai.api_server \
      --model Qwen/Qwen3-0.6B \
      --port 8180 \
      --metrics-port 8181 \
      --engine-worker-queue-port 8182 \
      --max-model-len 32768 \
      --max-num-seqs 32 \
      --logits-processors ThinkingBudgetLogitsProcessor

### 2. Send request

    curl -X POST "http://0.0.0.0:8180/v1/chat/completions" \
      -H "Content-Type: application/json" \
      -d '{
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_completion_tokens": 30,
        "logits_processors_args": {
          "thinking_budget": 20,
          "think_stop_sentence": "Thinking limit reached, now replying."
        }
      }'

If you do not need thinking control for a request, simply omit `thinking_budget`.

### 3. Operator-level thinking cap only (no logits processor)

    curl -X POST "http://0.0.0.0:8180/v1/chat/completions" \
      -H "Content-Type: application/json" \
      -d '{
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_completion_tokens": 512,
        "enable_thinking": true,
        "reasoning_max_tokens": 200
      }'

## Offline Usage

    from fastdeploy import LLM, SamplingParams

    llm = LLM(
        model="Qwen/Qwen3-0.6B",
        engine_worker_queue_port=8282,
        cache_queue_port=8383,
        logits_processors=["ThinkingBudgetLogitsProcessor"],
    )

    sampling_params = SamplingParams(
        max_tokens=512,
        logits_processors_args={"thinking_budget": 20, "think_stop_sentence": "Thinking limit reached, now replying."},
    )

    outputs = llm.chat([{"role": "user", "content": "Hello, who are u?"}], sampling_params)
    print(outputs[0].outputs.text)

## Performance Note

This processor runs `update_state` and `apply` on every decode step. If you only need a hard thinking-length cap and care most about throughput, consider the operator-level reasoning-length controls instead of per-step logits processing.