291 Commits

Author	SHA1	Message	Date
Ayakouji	a8f8791668	[Optimization] Qwen2.5-VL support multi-batch prefill (#5269) ... * update * fix Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix dict access --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-05 18:22:39 +08:00
周周周	c83dc58105	[Feature] support Two batch overlap, mainly used in Prefill (#5078)	2025-12-05 14:58:50 +08:00
fmiao2372	ebe613ccc8	[Intel HPU] fix bug about RP 5138 (#5380)	2025-12-05 11:33:29 +08:00
zccjjj	e927c65742	[XPU] [Optimization] [EP] EP communication optimization. (#5145)	2025-12-05 10:03:45 +08:00
Yonghua Li	f4119d51b4	[PD Disaggregation] support DP via v1 router and decouple DP and EP (#5197) ... * [fix] support DP via v1 router and decouple DP and EP * [fix] fix scripts * [fix] reset model path * [fix] dp use get_output_ep, fix router port type, update scripts * [merge] merge with latest code * [chore] remove some debug log * [fix] fix code style check * [fix] fix test_multi_api_server for log_dir name * [chore] reduce logs * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-12-04 15:38:43 +08:00
Longzhi Wang	5cd17fd662	[Models] Add forward_meta to moe models' forward function (#5138) ... * [Models] Add forward_meta to moe models' forward function * fix missing param * fix * fix * fix forward_meta * fix test and remove chunked MoE releated in config * fix test * fix * fix	2025-12-04 13:26:58 +08:00
fmiao2372	209006e6a6	[Intel HPU] fix memory fragmentation issue due to warmup process and fix moe all_reduce issue (#5357)	2025-12-04 11:29:41 +08:00
lizexu123	946025480e	[Bug fix] fix pooling models (#5358) ... * fix * fix * fix test * fix gpu_model_runner --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-12-04 11:06:30 +08:00
qwes5s5	a52aea073c	fix logprobs (#5335)	2025-12-04 10:38:51 +08:00
ddchenhao66	4e8096bd0d	[XPU] xpu support mm prefix cache (#5356) ... Co-authored-by: ddchenhao66 <dhaochen163.com>	2025-12-03 19:07:34 +08:00
qw86972190	6048ea37bd	[XPU]add enable_logprob (#5279) ... * [XPU]Update document * [XPU]Update documentation * [XPU]add enable_logprob * Fix code style issues * “doc” * “docs” * “doc” * Fix code style via pre-commit --------- Co-authored-by: root <root@gajl-bbc-onlinec-com-1498354.gajl.baidu.com>	2025-12-02 15:32:28 +08:00
lizexu123	c563eca791	[Feature] support reward model (#5301) ... * Your commit message here * add test * update develop * support reward * support enable_chunk_prefill * support bingfa * support convert is reward * update test * delete print * fix enable_thinking * add document * fix place * fix test * fix * support enable_prefix_caching * add no-enable_prefix-caching test * fix * support enable_prefix_caching * delete print * fix document * fix * fix test * fix document and delete chinese * udpate * enable_thinking * fix test	2025-12-02 14:55:31 +08:00
K11OntheBoat	2e1680838f	[PD Disaggregation] Support PD deployment of DeepSeekv3. (#5251) ... * Support deepseekv3 cache transfer for PD deploy * clean some log info --------- Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-12-02 14:11:50 +08:00
qwes5s5	117980dd4e	[LogProbs]Enable prompt logprobs output and modify data transmission method for the online interface. (#5089) ... * add prompt logprobs * Merge prompt_logprobs_tensors and prompt_logprobs * fix param check * trigger ci * fix unitest * fix logprobs bug	2025-12-02 13:49:51 +08:00
Longzhi Wang	add524d80c	[Feature] support chunked moe (#4575) ... * [Feature] support chunked moe * update * update * fix and add test * update * fix conflict and modity test * fix fused_moe * fix fused_moe * fix docstring * fix * fix typo * fix test * fix * fix * fix test * fix test	2025-12-01 15:17:18 +08:00
cmcamdy	9f4977eb74	[xpu] support mtp for xpu(mix) (#5274) ... * [XPU] support kernel for mtp(base) * [XPU] support kernel for mtp(base) * format * format * format * fix gather next token * fix step && add test * fix * mv pre/post process * add adjust batch / gather next token for mtp * fix code style * fix mtp kenrel name * fix mtp kernel test * mv xpu pre/post process * mv xpu pre/post process * [xpu] support mtp * fix code style	2025-12-01 11:03:14 +08:00
fmiao2372	2c7683d551	[Intel HPU] change MoE weights and scales from list to tensor and add… (#5289) ... * [Intel HPU] change MoE weights and scales from list to tensor and add q/k rms norm * update doc * move HPU_CHUNK_SIZE into envs	2025-11-28 19:17:05 +08:00
周周周	73886204d4	[Others] clean code (#5235)	2025-11-28 15:40:49 +08:00
cmcamdy	5a67a6d960	[XPU] support kernel for mtp(base) (#4748) ... * [XPU] support kernel for mtp(base) * [XPU] support kernel for mtp(base) * format * format * format * fix gather next token * fix step && add test * fix * mv pre/post process * add adjust batch / gather next token for mtp * fix code style * fix mtp kenrel name * fix mtp kernel test * mv xpu pre/post process * mv xpu pre/post process	2025-11-27 15:05:44 +08:00
GoldPancake	cfc5b0ccf9	[BugFix] fix mtp logprob bugs in chunk prefill (#5244) ... * fix mtp logprob bugs in chunk prefill * fix * fix	2025-11-27 11:31:29 +08:00
chen	00d0ef5134	check (#5237)	2025-11-26 17:07:26 +08:00
Daci	f25ee3a26f	[Feature] enable guided decoding ENABLE_V1_KVCACHE_SCHEDULER = 1 (#5140) ... * enable guided decoding ENABLE_V1_KVCACHE_SCHEDULER = 1 * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-26 10:22:35 +08:00
kevin	8e4e3ff510	[Feature] support eplb in api_server (#4782) ... * support eplb in api_server * update code * add eplb test case * update eplb * support tp+dp eplb * update test cese * update code * update code * fix bug * update copilot review * update test case name	2025-11-24 20:22:29 +08:00
xiaozude	d5bd64336a	[Metax] support ENABLE_V1_KVCACHE_SCHEDULER (#5163)	2025-11-24 19:19:49 +08:00
xiaoxiaohehe001	95f3c8c641	[Fix] Fix eplb bug and support fp8 load weight (#5178) ... * fix eplb part2 * fix eplb part2 * fix eplb part2	2025-11-24 15:31:37 +08:00
kevin	c068a4f642	[Feature] dyc8 support prefixcache (#5125) ... * dyc8 support prefixcache * fix cache_trans test case * update code	2025-11-21 19:46:26 +08:00
ddchenhao66	e70e2279ce	[PD Disaggregation][XPU] Add XPU support for PD disaggregation (#5113) ... * [XPU] xpu support PD disaggregation * [XPU] fix the issue of cache KV transfer process startup failure on non-zero XPU cards * [XPU] xpu support PD disaggregation in v1 scheduler --------- Co-authored-by: ddchenhao66 <dhaochen163.com>	2025-11-21 14:09:01 +08:00
Yonghua Li	43097a512a	[BugFix] [PD Disaggregation] fix v1 scheduler prefill node profile run & ipc transfer protocol (#5132) ... * [fix] fix v1 scheduler profile run for append attention in prefill node * [fix] skip send_signal if kv signal not inited for gpu and xpu * [fix] extend fix to flash_attn & mla_attn * [fix] fix v1 pd run in ipc transfer protocol * [ci] add test for v1 pd profile run using ipc transfer protocol * [style] fix code style check * [style] fix code style again * [fix] fix profile run * [update] remove --num-gpu-blocks-override in example script * [chore] rename forward_meta is_profiling to is_dummy_or_profile_run	2025-11-20 21:39:22 +08:00
Daci	eab8384da6	[Feature] ThreadPoolExecutor async fill_token_bitmask (#5083) ... * ThreadPoolExecutor async fill_token_bitmask * ThreadPoolExecutor async fill_token_bitmask logging * fix test_guided_decoding * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add fill_bitmask_parallel_batch_size ENV * FD_FILL_BITMASK_BATCH fastdeploy.envs --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-19 10:04:16 +08:00
K11OntheBoat	4a7739ec0b	Fix dummy run when use PD Disaggregation with EP inference. (#5112) ... Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-11-18 21:09:30 +08:00
fmiao2372	74f33efdbf	[Intel HPU] fix bugs caused by other commits (#5074) ... * [Intel HPU] fix bugs caused by other commits * update code by copilot	2025-11-17 15:28:55 +08:00
fmiao2372	e43a5fc055	[Intel HPU] enable level 1 prefix caching and fix some bugs (#4971) ... * [Intel HPU] enable prefix caching and dense tp moe ep and fix some bugs * update code by copilot * remove dense tp and moe ep code	2025-11-14 19:42:50 +08:00
Daci	5fc12eddfe	[Optimization] xgrammar async compile, multi thread, speed up (#4835) ... * xgrammar async compile, multi thread, speed up * fix test_sampler.py & pre-commit err * add redis version check && fix request.llm_engine_recv_req_timestamp * xgrammar prefill & decode & v0 * fix test_gpu_prompt_logprobs.py * add test_guided_decoding.py * Update fastdeploy/scheduler/splitwise_scheduler.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/model_executor/guided_decoding/xgrammar_backend.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/model_executor/guided_decoding/xgrammar_backend.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix torch xgrammar unittest env --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-14 18:05:26 +08:00
周周周	51b1f13547	[Executor]move batch_id_per_token (#4853)	2025-11-14 15:38:48 +08:00
Juncai	36822fa49c	[PD Disaggregation] remove splitwise deployment on single node and refine the code (#4891) ... * remove splitwise deployment on single node and refine the code * up * up * up * add test * up	2025-11-14 09:56:53 +08:00
周周周	6c4ebc5fee	[worker_process.py]modify some var name (#4749)	2025-11-13 14:21:27 +08:00
qwes5s5	a2d06118e1	[Logprobs]Support prompt_logprobs and max_logprobs (#4897) ... * add prompt logprobs * trigger ci * fix unitest * Update fastdeploy/config.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/entrypoints/llm.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/engine/sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/engine/test_sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/engine/test_sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix max_logprobs --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-12 19:29:48 +08:00
ltd0924	5bf48de999	[KVCache] support unified cache backend (#4903) ... * [Feature] support unified cache backend * fix * fix * fix * fix * Update metax_model_runner.py * fix * update * Update test_moba_attention_backend.py --------- Co-authored-by: ltd0924 <luotingdan@baidu.com>	2025-11-12 14:54:52 +08:00
周周周	da6b4c10e5	[ATTENTION] make buffer alloc as a function (#4945)	2025-11-11 19:17:08 +08:00
chen	896ef565cc	[Others] Add Tests for GPU Model Runner and Logprobs Output (#4913)	2025-11-11 18:37:33 +08:00
K11OntheBoat	76be598129	replace paddle.max by numpy to avoid useless error log (#4893) ... Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-11-11 16:28:05 +08:00
SunLei	3098aee05f	[Perf] Support tensor transmission between work and engine with zero-copy to improve efficiency (#4839) ... * feat(zmq): support tensor transmission with zero-copy for improved efficiency * perf: zmq.send disable copy * zmq recv data for debug * convert logprobs tensor to cpu	2025-11-11 15:43:11 +08:00
Yuanle Liu	3dc0ffa46d	[TSP] Support qwen3 moe tsp + cudagraph (#4871) ... * support qwen3_moe tsp mode * fix * fix * update * update * update * fix * support external_rmsnorm * update * fix	2025-11-10 23:37:51 +08:00
chenjian	78895e2c7d	[Bug Fix] fix bug for PD EP (#4823) ... * fix bug for PD EP * fix * optimize perf for engine worker queue * fix bug * fix internode ll two stage * fix for ci * fix bug	2025-11-10 15:33:29 +08:00
chen	80aedb82ce	[BugFix] max_lgprobes=-1 maps to ori_vocab_size (#4884) ... * -1 ori_vobal_size * check * check * check * revert config.py	2025-11-07 22:15:40 +08:00
Neil Zhu	6de1ce3b25	[Metax] support ERNIE-4.5-VL-28B (#4820)	2025-11-07 04:55:49 -08:00
Juncai	08ca0f6aea	[Feature] [PD] add simple router and refine splitwise deployment (#4709) ... * add simple router and refine splitwise deployment * fix	2025-11-06 14:56:02 +08:00
周周周	69fa741763	remove seq_lens_this_time (#4821)	2025-11-06 11:06:28 +08:00
K11OntheBoat	62dfad4a5f	[PD Disaggregation] Support Qwen3-MoE use PD + EP inference. (#4691) ... support Qwen-MoE PD/EP	2025-11-06 10:32:15 +08:00
周周周	876e4a8935	remove input_ids from ForwardMeta (#4793)	2025-11-05 11:55:51 +08:00