# Trace - FastDeploy: Large Language Model Deployment

Skip to content

# FastDeploy Tracing with OpenTelemetry

FastDeploy exports request tracing data through the OpenTelemetry Collector. Tracing can be enabled when starting the server using the `--trace-enable` flag, and the OpenTelemetry Collector endpoint can be configured via `--otlp-traces-endpoint`.

## Setup Guide

### 1. Install Dependencies

    # Manual installation
    pip install opentelemetry-sdk \
                opentelemetry-api \
                opentelemetry-exporter-otlp \
                opentelemetry-exporter-otlp-proto-grpc

### 2. Start OpenTelemetry Collector and Jaeger

    docker compose -f examples/observability/tracing/tracing_compose.yaml up -d

### 3. Start FastDeploy Server with Tracing Enabled

#### Configure FastDeploy Environment Variables

    # Enable tracing
    "TRACES_ENABLE": "true",

    # Service name
    "FD_SERVICE_NAME": "FastDeploy",

    # Instance name
    "FD_HOST_NAME": "trace_test",

    # Exporter type
    "TRACES_EXPORTER": "otlp",

    # OTLP endpoint:
    #   gRPC: 4317
    #   HTTP: 4318
    "EXPORTER_OTLP_ENDPOINT": "http://localhost:4317",

    # Optional headers
    "EXPORTER_OTLP_HEADERS": "Authentication=Txxxxx",

    # Export protocol
    "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL": "grpc",

#### Start FastDeploy

Start the FastDeploy server with the above configuration and ensure that tracing is enabled.

### 4. Send Requests and View Traces

- Open the Jaeger UI in your browser (port `16686`) to visualize request traces.
- The OpenTelemetry Collector will also export the trace data to a local file:

    /tmp/otel_trace.json

## Adding Tracing to Your Own Code

FastDeploy already inserts tracing points at most critical execution stages. Developers can use the APIs provided in `trace.py` to add more fine-grained tracing.

### 4.1 Initialize Tracing

Each process involved in tracing must call:

    process_tracing_init()

Each thread that participates in a traced request must call:

    trace_set_thread_info("thread_label", tp_rank, dp_rank)

- `thread_label`: identifier used for visual distinction of threads.
- `tp_rank` / `dp_rank`: optional values to label tensor parallelism or data parallelism ranks.

### 4.2 Mark Request Start and Finish

    trace_req_start(rid, bootstrap_room, ts, role)
    trace_req_finish(rid, ts, attrs)

- Creates both a Bootstrap Room Span and a Root Span.
- Supports inheritance from spans created by the FastAPI Instrumentor (context copying).
- `attrs` can be used to attach additional attributes to the request span.

### 4.3 Add Tracing for Slices

#### Standard Slice

    trace_slice_start("slice_name", rid)
    trace_slice_end("slice_name", rid)

#### Mark Thread Completion

The last slice in a thread can mark the thread span as finished:

    trace_slice_end("slice_name", rid, thread_finish_flag=True)

### 4.4 Trace Context Propagation Across Threads

#### Sender Side (ZMQ)

    trace_context = trace_get_proc_propagate_context(rid)
    req.trace_context = trace_context

#### Receiver Side (ZMQ)

    trace_set_proc_propagate_context(rid, req.trace_context)

### 4.5 Add Events and Attributes

#### Events (recorded on the current slice)

    trace_event("event_name", rid, ts, attrs)

#### Attributes (attached to the current slice)

    trace_slice_add_attr(rid, attrs)

## Extending the Tracing Framework

### 5.1 Trace Context Hierarchy

- Two levels of Trace Context:

- `TraceReqContext` – request-level context

- `TraceThreadContext` – thread-level context

- Three-level Span hierarchy:

- `req_root_span`

- `thread_span`
- `slice_span`

### 5.2 Available Span Name Enum (`TraceSpanName`)

    FASTDEPLOY
    PREPROCESS
    SCHEDULE
    PREFILL
    DECODE
    POSTPROCESS

- These enums can be used when creating slices to ensure consistent naming.

### 5.3 Important Notes

1. Each thread span must be closed when the final slice of that thread finishes.
2. Spans created by FastAPI Instrumentor are automatically inherited by the internal tracing context.