# MultiNode Deployment - FastDeploy: Large Language Model Deployment

Skip to content

简体中文

# Multi-Node Deployment

## Overview

Multi-node deployment addresses scenarios where a single machine's GPU memory is insufficient to support deployment of large models by enabling tensor parallelism across multiple machines.

## Environment Preparation

### Network Requirements

1. All nodes must be within the same local network
2. Ensure bidirectional connectivity between all nodes (test using `ping` and `nc -zv`)

#### Software Requirements

1. Install the same version of FastDeploy on all nodes
2. [Recommended] Install and configure MPI (OpenMPI or MPICH)

## Tensor Parallel Deployment

### Recommended Launch Method

We recommend using mpirun for one-command startup without manually starting each node.

### Usage Instructions

1. Execute the same command on all machines
2. The IP order in the `ips` parameter determines the node startup sequence
3. The first IP will be designated as the master node

4. Ensure all nodes can resolve each other's hostnames

5. Online inference startup example:

   `shell python -m fastdeploy.entrypoints.openai.api_server \ --model baidu/ERNIE-4.5-300B-A47B-Paddle \ --port 8180 \ --metrics-port 8181 \ --engine-worker-queue-port 8182 \ --max-model-len 32768 \ --max-num-seqs 32 \ --tensor-parallel-size 16 \ --graph-optimization-config '{"use_cudagraph":false}' \ --no-enable-prefix-caching \ --disable-custom-all-reduce \ --ips 192.168.1.101,192.168.1.102`

> :bulb: Multi-node tensor parallel deployment currently does not support CUDA Graphs, Prefix Caching, or Custom AllReduce, and these features must be explicitly disabled in the deployment command.

- Offline startup example:

  ```python from fastdeploy.engine.sampling_params import SamplingParams from fastdeploy.entrypoints.llm import LLM

  model_name_or_path = "baidu/ERNIE-4.5-300B-A47B-Paddle"

  sampling_params = SamplingParams(temperature=0.1, max_tokens=30) llm = LLM(model=model_name_or_path, tensor_parallel_size=16, ips="192.168.1.101,192.168.1.102") if llm._check_master(): output = llm.generate(prompts="Who are you?", use_tqdm=True, sampling_params=sampling_params) print(output) ```

- Notes:

- Only the master node can receive completion requests
- Always send requests to the master node (the first IP in the ips list)
- The master node will distribute workloads across all nodes

### Parameter Description

#### `ips` Parameter

- Type: `string`
- Format: Comma-separated IPv4 addresses
- Description: Specifies the IP addresses of all nodes in the deployment group
- Required: Only for multi-node deployments
- Example: `"192.168.1.101,192.168.1.102,192.168.1.103"`

#### `tensor_parallel_size` Parameter

- Type: `integer`
- Description: Total number of GPUs across all nodes
- Required: Yes
- Example: For 2 nodes with 8 GPUs each, set to 16