# Supported Models - FastDeploy: Large Language Model Deployment

Skip to content

简体中文

# Supported Models

FastDeploy currently supports the following models, which can be downloaded automatically during FastDeploy deployment.Specify the `model` parameter as the model name in the table below to automatically download model weights (all supports resumable downloads). The following three download sources are supported:

- AIStudio
- ModelScope
- HuggingFace

When using automatic download, the default download source is AIStudio. Users can modify the default download source by setting the `FD_MODEL_SOURCE` environment variable, which can be set to “AISTUDIO”, ‘MODELSCOPE’ or “HUGGINGFACE”. The default download path is `~/` (i.e., the user's home directory). Users can modify the default download path by setting the `FD_MODEL_CACHE` environment variable, e.g.:

    export FD_MODEL_SOURCE=AISTUDIO # "AISTUDIO", "MODELSCOPE" or "HUGGINGFACE"
    export FD_MODEL_CACHE=/ssd1/download_models

> Example launch Command using baidu/ERNIE-4.5-21B-A3B-PT:

    python -m fastdeploy.entrypoints.openai.api_server \
           --model baidu/ERNIE-4.5-0.3B-PT \
           --port 8180 \
           --metrics-port 8181 \
           --engine-worker-queue-port 8182 \
           --max-model-len 32768 \
           --max-num-seqs 32

## Large Language Models

These models accept text input.

Models | DataType | Example HF Model
--- | --- | ---

⭐ERNIE

BF16\WINT4\WINT8\W4A8C8\WINT2\FP8

baidu/ERNIE-4.5-VL-424B-A47B-Paddle;
baidu/ERNIE-4.5-300B-A47B-Paddle
quick start best practice;
baidu/ERNIE-4.5-300B-A47B-2Bits-Paddle;
baidu/ERNIE-4.5-300B-A47B-W4A8C8-TP4-Paddle;
baidu/ERNIE-4.5-300B-A47B-FP8-Paddle;
baidu/ERNIE-4.5-300B-A47B-Base-Paddle;
baidu/ERNIE-4.5-21B-A3B-Paddle;
baidu/ERNIE-4.5-21B-A3B-Base-Paddle;
baidu/ERNIE-4.5-21B-A3B-Thinking;
baidu/ERNIE-4.5-VL-28B-A3B-Thinking;
baidu/ERNIE-4.5-0.3B-Paddle
quick start best practice;
baidu/ERNIE-4.5-0.3B-Base-Paddle, etc.

⭐QWEN3-MOE | BF16/WINT4/WINT8/FP8 | Qwen/Qwen3-235B-A22B; Qwen/Qwen3-30B-A3B, etc.
⭐QWEN3 | BF16/WINT8/FP8 | Qwen/qwen3-32B; Qwen/qwen3-14B; Qwen/qwen3-8B; Qwen/qwen3-4B; Qwen/qwen3-1.7B; Qwen/qwen3-0.6B, etc.
⭐QWEN2.5 | BF16/WINT8/FP8 | Qwen/qwen2.5-72B; Qwen/qwen2.5-32B; Qwen/qwen2.5-14B; Qwen/qwen2.5-7B; Qwen/qwen2.5-3B; Qwen/qwen2.5-1.5B; Qwen/qwen2.5-0.5B, etc.
⭐QWEN2 | BF16/WINT8/FP8 | Qwen/Qwen/qwen2-72B; Qwen/Qwen/qwen2-7B; Qwen/qwen2-1.5B; Qwen/qwen2-0.5B; Qwen/QwQ-32, etc.
⭐DEEPSEEK | BF16/WINT4 | unsloth/DeepSeek-V3.1-BF16; unsloth/DeepSeek-V3-0324-BF16; unsloth/DeepSeek-R1-BF16, etc.
⭐GPT-OSS | BF16/WINT8 | unsloth/gpt-oss-20b-BF16, etc.
⭐GLM-4.5/4.6 | BF16/wfp8afp8 | zai-org/GLM-4.5-Air; zai-org/GLM-4.6 最佳实践 etc.

## Multimodal Language Models

These models accept multi-modal inputs (e.g., images and text).

Models | DataType | Example HF Model
--- | --- | ---
ERNIE-VL | BF16/WINT4/WINT8 | baidu/ERNIE-4.5-VL-424B-A47B-Paddle quick start best practice ; baidu/ERNIE-4.5-VL-28B-A3B-Paddle quick start best practice ; baidu/ERNIE-4.5-VL-28B-A3B-Thinking quick start best practice;
PaddleOCR-VL | BF16/WINT4/WINT8 | PaddlePaddle/PaddleOCR-VL best practice ;
QWEN-VL | BF16/WINT4/FP8 | Qwen/Qwen2.5-VL-72B-Instruct; Qwen/Qwen2.5-VL-32B-Instruct; Qwen/Qwen2.5-VL-7B-Instruct; Qwen/Qwen2.5-VL-3B-Instruct

More models are being supported. You can submit requests for new model support via Github Issues.