79 Commits

Author	SHA1	Message	Date
qwes5s5	b2a2e11551	[Feature] Support stopping the inference for the corresponding request in the online service after a disconnection request. (#5320) ... * request disconnect * request disconnect * fix bug * fix bug--amend --------- Co-authored-by: root <root@yq01-sys-rpm26xc1knu.yq01.baidu.com>	2026-01-16 11:46:13 +08:00
ddchenhao66	9373f373dc	[XPU] fix multi-batch bug in VL model (#6015) ... * [XPU] fix multi-batch bug in VL model * Add command to kill additional port processes --------- Co-authored-by: ddchenhao66 <dhaochen163.com> Co-authored-by: Jiaxin Sui <95567040+plusNew001@users.noreply.github.com>	2026-01-14 19:44:58 +08:00
chenjian	74d0f1c01f	[Optim] Robust sync status when preempted happens (#5796) ... * [Bug fix] Sync status for caching output cache * fix * fix * fix bug * fix * fix * support xpu * fix * fix * fix * fix * fix * fix ci * fix ci * fix xpu --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2026-01-14 12:07:33 +08:00
kevin	cb9f952f32	[BugFix] fix metrics cache tokens (#6001) ... * fix metrics cache tokens * update code	2026-01-12 22:50:56 -08:00
zhupengyang	9db48ecb34	[XPU] fix dp4 (#5946)	2026-01-09 20:36:53 +08:00
kevin	2d2b156252	[BugFix] fix dyc8 cache bug (#5958) ... * fix dyc8 cache bug * update code	2026-01-08 19:25:47 -08:00
Daci	d8c6ba61f3	[BugFix] resource_manager_v1 lock PD (#5616) ... * bugfix resource_manager_v1 lock PD * with lock add_prefilled_request --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2026-01-08 10:02:54 +08:00
chenjian	925e7edd3c	[Bug fix] Limit multi-modal request to 1 (#5901)	2026-01-07 20:25:07 +08:00
chenjian	c883a2d3ec	[Optimization] Reduce preemption occurrence when blocks not enough (#5696) ... * [Optimize] Reduce preemption occurrence when blocks not enough for decoding * fix * fix * fix spell * optimize performance * fix	2026-01-07 20:01:16 +08:00
kevin	eabd01cd21	[BugFix] fix eb5 prefix bug (#5879) ... * fix eb5 prefix bug * update ci test * update code * update code * update code * update code * update code * update code * update code	2026-01-06 23:50:39 -08:00
fmiao2372	1ee285c2d6	[Intel HPU] enable chunked prefill (#5903) ... * [Intel HPU] enable chunked prefill * fix bug by copilot comments	2026-01-06 21:01:50 +08:00
jc	e911ac2ce7	[BugFix] Refine the preparation of cpu and storage cache (#5777) ... * Refine the preparation of cpu and storage cache * fix error * fix error * up * fix * up docs * fix unittest * remove debug info	2026-01-05 10:13:30 +08:00
kevin	52dc9a7b85	[BugFix] skip mm revert (#5848) ... * skip mm revert * update code * update test	2026-01-04 14:25:45 +08:00
kevin	74e162697f	eb5 mm skip prefix cache (#5838)	2025-12-30 05:30:48 -08:00
CSWYF3634076	9286403570	[Models] Add Qwen3-VL Model Support (#5763) ... * support v1 loader * remove useless code * remove useless * [Model] support Qwen3VL images success * [Model] support Qwen3VL rope_3d * [Model] support Qwen3VL remove log * [Model] support Qwen3VL RL * [Model] support Qwen3VL tp * [Model] support Qwen3VL video * [Model] support Qwen3VL fix ernievl * [Model] support Qwen3VL fix get_image_boundaries.cc array out of bounds * [Model] support Qwen3VL fix multi card * [Model] support Qwen3VL file close * [Model] support Qwen3VL fix ce * [Model] support Qwen3VL fix unittest * [Model] support Qwen3VL add unittest --------- Co-authored-by: Ayakouji <yuhongh@qq.com>	2025-12-29 17:39:33 +08:00
kevin	894f4e312b	[FDConfig] disable chunked_mm_input in ernie5 (#5774) ... * disable chunked_mm_input in ernie5 * update code * update code * update test case * update testcase * upate case	2025-12-26 15:31:27 +08:00
RichardWooSJTU	01c18f328f	rename need_block_num_signal (#5623)	2025-12-26 11:02:29 +08:00
Juncai	412867fd99	[Feature] Support KV Cache Storage (#5571) ... * Support Mooncake Store * up * up * add op * fix conflict * fix error * up for comments * avoid thread lock * up * fix unittest * fix unittest * remove debug info * consider tp_size > 1 * add default rdma_nics * add utils * up * fix error --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-12-25 16:30:35 +08:00
ming1753	85db9d5e56	[Others] reschedule preempt task support optional func (#5649) ... * [Others] reschedule preempt task support optional func * fix bug * fix bug	2025-12-23 20:45:52 +08:00
ming1753	81384ef29e	[BugFix] fix download feature bug (#5669)	2025-12-22 13:46:39 +08:00
kevin	807e404369	[BugFix] fix eb5 mm prefix cache bug (#5638) ... * fix eb5 mm prefix cache bug * update code --------- Co-authored-by: Jiaxin Sui <95567040+plusNew001@users.noreply.github.com>	2025-12-19 14:57:37 +08:00
yzwu	ac013803f3	[Iluvatar] Support V1_KVCACHE_SCHEDULER and paddleocr-vl rope mode (#5555)	2025-12-18 02:14:25 -08:00
Jiang-Jia-Jun	8b6395478a	Revert "[BugFix] reschedule_preempt_task append waiting & PREEMPTED blocksize…" (#5575) ... This reverts commit dbedb0797b.	2025-12-16 11:12:57 +08:00
Daci	dbedb0797b	[BugFix] reschedule_preempt_task append waiting & PREEMPTED blocksize (#5506) ... * bugfix reschedule_preempt_task append waiting & PREEMPTED blocksize * bugfix reschedule_preempt_task append waiting & PREEMPTED blocksize * 注释 * [bugfix] PREEMPTED task blocksize * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-12-12 17:43:29 +08:00
kevin	954a145d57	[Optimization] support mm prefill batch (#5313) ... * support mm prefill batch * update code * update code * update code * update code * fix encoder cache bug * update code * update code * fix bug * fix paddle ocr bug * fix xpu bug * update code	2025-12-11 22:21:14 +08:00
Juncai	80efe98f8d	[PD Disaggregation] Add timestamp for analyzing splitwise deployment (#5317) ... * Add timestamp for analyzing splitwise deployment * up * up * up * up * up * up * fix format * fix	2025-12-08 10:08:44 +08:00
kevin	c9d7f9e7c3	[BugFix] fix async download bug (#5349) ... * fix async download bug * update log * Revert "update log" This reverts commit 5816e602f4. * update code * fix mtp bug	2025-12-05 18:59:12 +08:00
Ayakouji	a8f8791668	[Optimization] Qwen2.5-VL support multi-batch prefill (#5269) ... * update * fix Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix dict access --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-05 18:22:39 +08:00
chenjian	3878a99b69	[Fearture] Support cache kv cache for output tokens (#4535) ... * [Fearture] Support cache kv cache for output tokens * fix bug * fix ci bug * improve coverage * enable output caching by default * fix ci --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-12-04 20:53:08 +08:00
kevin	2d69d91ab8	add aksk check (#5273)	2025-11-28 14:28:24 +08:00
Daci	7dc06cac6e	[BugFix] race condition [is_fetching] causing multiple fetch requests (#5238) ... * RouterArgs port str -> int * fix race condition [is_fetching] causing multiple fetch requests * bugfix: Delete duplicate input_ids tensor creation	2025-11-28 13:41:36 +08:00
Yonghua Li	cead6b26fa	[Metrics] Update time_to_first_token to include tokenization & queue time, and remove redundant metrics (#4993) ... * [update] update time_to_first_tokens to include queue time, and remove first_token_latency and infer_latency * [doc] update docs * [ci] fix test * [chore] delete redundant code --------- Co-authored-by: Jiaxin Sui <95567040+plusNew001@users.noreply.github.com>	2025-11-26 14:42:17 +08:00
Juncai	af03da5127	[BugFix] fix release block ids (#5184) ... * fix release block ids * up	2025-11-24 16:48:09 +08:00
Juncai	f9b0545a7f	[PD Disaggregation] [Refine] Refine splitwise deployment (#5151) ... * Refine splitwise deployment * up	2025-11-21 15:30:24 +08:00
kevin	7454480e07	[Feature] support bos download retry (#5137) ... * support bos download retry * update code * update code	2025-11-21 10:18:32 +08:00
Yonghua Li	43097a512a	[BugFix] [PD Disaggregation] fix v1 scheduler prefill node profile run & ipc transfer protocol (#5132) ... * [fix] fix v1 scheduler profile run for append attention in prefill node * [fix] skip send_signal if kv signal not inited for gpu and xpu * [fix] extend fix to flash_attn & mla_attn * [fix] fix v1 pd run in ipc transfer protocol * [ci] add test for v1 pd profile run using ipc transfer protocol * [style] fix code style check * [style] fix code style again * [fix] fix profile run * [update] remove --num-gpu-blocks-override in example script * [chore] rename forward_meta is_profiling to is_dummy_or_profile_run	2025-11-20 21:39:22 +08:00
Juncai	01c30f6b87	Fix schedule error in splitwise deployment (#5149)	2025-11-20 21:18:10 +08:00
yangjianfengo1	af715db763	[Scheduler] Support chunk prefill for video input (#5107) ... * add video chunk prefill * add vit_merge=True for test_tokenizer_client.py --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-11-20 16:29:13 +08:00
kevin	109d48e456	[Feature] support async download features (#5003) ... * support async download features * add test case * update code	2025-11-19 22:23:36 +08:00
Yonghua Li	6c5ab727c1	[BugFix] fix num_requests_running after clear_data (#4927) ... * [BugFix] fix num_requests_running after clear_data * [fix] fix tasks_list and stop flags not cleared when _free_blocks failed	2025-11-13 13:50:21 +08:00
kevin	cc34487810	[Feature] support mm disable_chunked (#4803) ... * support mm disable_chunked * update code * update code * update code	2025-11-06 21:32:25 +08:00
kevin	64e875b460	[Scheduler] update v1 prefill batch (#4611) ... * update v1 prefill batch * update code * update code	2025-10-31 14:03:01 +08:00
李泳桦	a012e3608b	[Feature] support logits processors (#4515) ... * [feat] provide an interface for logits processors and a builtin LogitBiasLogitsProcessor * [chore] fix code style * [fix] add unit test & fix existing bugs * [feat] add engine/worker arg --logits-processors * [fix] redefine user args as logits_processors_args and fix some bugs * [fix] fix test_sampler * Update fastdeploy/model_executor/logits_processor/builtin.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update fastdeploy/model_executor/logits_processor/__init__.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/model_executor/test_logits_processor.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [fix] fix typo * Update fastdeploy/engine/sampling_params.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [fix] fix bracelet * [chore] redefine logits processor interface: pass the entire share_inputs into LP, do not copy share_inputs and logits * [doc] add docs * [fix] fix logit bias processor not applied when decoding is too fast & add docs and tests * [fix] fix redundant code * [feat] skip apply() if no bias is specified --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-10-29 00:08:53 +08:00
ming1753	7681375a19	[BugFix] PaddleOCR-VL fix FD_DEBUG type and support v1 loader (#4605) ... * [Bug Fix] PaddleOCRVL fix FD_DEBUG type and support HF model * fix bug * fix bug * fix bug	2025-10-28 09:47:47 +08:00
kevin	8aab4e367f	[Feature] mm support prefix cache (#4134) ... * support mm prefix caching * update code * fix mm_hashes * support encoder cache * add encoder cache * update code * update encoder cache * fix features bug * fix worker bug * support processor cache, need to optimize yet * refactor multimodal data cache * update code * update code * update v1 scheduler * update code * update code * update codestyle * support turn off processor cache and encoder cache * update pre-commit * fix code * solve review * update code * update code * update test case * set processor cache in GiB * update test case * support mm prefix caching for qwen model * fix code style check * update pre-commit * fix unit test * fix unit test * add ci test case * fix rescheduled bug * change text_after_process to prompt_tokens * fix unit test * fix chat template * change model path * [EP] fix adapter bugs (#4572) * Update expert_service.py * Update common_engine.py * Update expert_service.py * fix v1 hang bug (#4573) * fix import image_ops error on some platforms (#4559) * [CLI]Update parameters in bench latecy cli tool and fix collect-env cli tool (#4558) * add collect-env * del files * [Graph Optimization] Add dy_runnable and introduce cudagraph_switch_threshold for cudagraph mode switching (#4578) * add new branch for sot * reorder * fix batch bug * [XPU]Moe uses a new operator (#4585) * [XPU]Moe uses a new operator * [XPU]Moe uses a new operator * update response * [Feature] Support Paddle-OCR (#4396) * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com> * [DataProcessor] add reasoning_tokens into usage info (#4520) * add reasoning_tokens into usage info initial commit * add unit tests * modify unit test * modify and add unit tests * fix unit test * move steam usage to processor * modify processor * modify test_logprobs * modify test_logprobs.py * modify stream reasoning tokens accumulation * fix unit test * perf: Optimize task queue communication from engine to worker (#4531) * perf: Optimize task queue communication from engine to worker * perf: get_tasks to numpy * perf: get_tasks remove to_numpy * fix: request & replace ENV * remove test_e2w_perf.py * fix code style --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * Clean up ports after processing results (#4587) * [CI] Add /re-run command in PR comments to restart failed CI workflows (#4593) * [Others] api server exits when worker process is dead (#3271) * [fix] fix terminal hangs when worker process is dead * [chore] change sleep time of monitor * [chore] remove redundant comments * update docs --------- Co-authored-by: ApplEOFDiscord <wwy640130@163.com> Co-authored-by: ApplEOFDiscord <31272106+ApplEOFDiscord@users.noreply.github.com> Co-authored-by: ltd0924 <32387785+ltd0924@users.noreply.github.com> Co-authored-by: yinwei <yinwei_hust@163.com> Co-authored-by: JYChen <zoooo0820@qq.com> Co-authored-by: qwes5s5 <45442318+qwes5s5@users.noreply.github.com> Co-authored-by: Ryan <zihaohuang@aliyun.com> Co-authored-by: yyssys <atyangshuang@foxmail.com> Co-authored-by: ming1753 <61511741+ming1753@users.noreply.github.com> Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com> Co-authored-by: kxz2002 <115912648+kxz2002@users.noreply.github.com> Co-authored-by: SunLei <sunlei5788@gmail.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: Zhang Yulong <35552275+ZhangYulongg@users.noreply.github.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: 李泳桦 <39643373+liyonghua0910@users.noreply.github.com>	2025-10-27 17:39:51 +08:00
ming1753	e4e3cede7f	[Feature] Support Paddle-OCR (#4396) ... * init * update code * fix code style & disable thinking * adapt for common_engine.update_mm_requests_chunk_size * use 3d rope * use flash_attn_unpadded * opt siglip * update to be compatible with the latest codebase * fix typo * optim OCR performance * fix bug * fix bug * fix bug * fix bug * normlize name * modify xpu rope * revert logger * fix bug * fix bug * fix bug * support default_v1 * optim performance * fix bug --------- Co-authored-by: root <root@szzj-acg-tge1-fdda9.szzj.baidu.com> Co-authored-by: zhangyue66 <zhangyue66@baidu.com>	2025-10-24 23:34:30 +08:00
chenjian	670aaa3f83	[Bug fix] Fix pd for x1 thinking (#4433)	2025-10-16 12:03:45 +08:00
李泳桦	ffe7af8a97	[fix] fix requests & block metrics (#4404) ... * [fix] fix requests & block metrics * [chore] rename variables	2025-10-15 11:49:24 +08:00
AIbin	48fd5d757d	Support MLA_CACHE & Fix V1_Schedule Bug (#4318) ... Support MLA_CACHE & Fix V1_Schedule Bug	2025-10-09 12:11:25 +08:00
RichardWooSJTU	3740e33fea	【Feature】ResourceManagerV1 support need block num notifying (#4220) ... * support need block num notifying * adapt t2i * fix unexpected change	2025-09-29 11:11:51 +08:00