47 Commits

Author	SHA1	Message	Date
bukejiyu	14d46181b8	[Loader] add multi-thread model loading (#6877) ... * multi-thread-loader * fix ut	2026-04-09 23:40:15 -07:00
bukejiyu	c62f6b4ea5	[Others] Fix PD reorder for MTP (#6792) ... * fix pd reorder in mtp * add ut * update * fix mtp	2026-03-23 21:10:22 +08:00
sunxin	33e01f22a8	[Feature][Sampling] Extend top-k_top-p sampling to all backends and unify greedy decoding with top_k=1 (#6894) ... * update sampling * fix * fix * fix mtp * fix test	2026-03-19 01:43:10 -07:00
gongweibao	fb6c56dfd5	[BugFix][DataProcessor] Force top_k=1 for greedy decoding when temperature=0 (#6748) ... * [BugFix] Force top_k=1 for greedy decoding when temperature=0 When temperature is set to 0 (greedy decoding), only setting temperature to a small epsilon is insufficient — the sampling kernel may still pick non-top-1 tokens. Explicitly set top_k=1 in all processors to guarantee argmax behavior. Additionally, add argmax fast-path in top_k_top_p_sampling() under FD_DETERMINISTIC_MODE to handle non-rejection sampling backends that ignore top_k parameter. * Extract greedy decoding from FD_DETERMINISTIC_MODE guard top_k=1 → argmax is a correctness optimization, not deterministic-specific. Remove the FD_DETERMINISTIC_MODE guard so all-greedy fast-path and mixed-batch override work unconditionally. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update test_torch_model.py --------- Co-authored-by: gongweibao <gognweibao@baidu.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2026-03-18 17:36:43 +08:00
fxyfxy777	250ce40b40	[Feature] use phi permute/unpermute & rm swiglu (#6361) ... * tp文字输出正常 * B eb5 mini文字输出正常 * eb5mini ep B卡 文字输出正常 * default use phi moe op * stash * tp H卡正常 * ep ok * rm debug * rm debug tool * rm del ffn_out * rm swiglu * add envs to swiglu * merge dev * fix ci baseline Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix ci baseline 2 --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-12 02:01:57 -07:00
YuBaoku	56ceeda80c	[CI] Adjust model-specific diff threshold and include iluvatar XPU paths in coverage (#6663)	2026-03-05 10:02:54 +08:00
sunxin	53aaac69da	[Optimization] Enable BF16 gate computation for GLM and Qwen (#6457) ... * gate bf16 * add gate-fp32 * fix * update baseline * update * update * fix	2026-02-26 21:08:46 -08:00
chen	29a313a402	[Optimization] Support FA2/FA3/FA4 with attn_mask_q (#6354) ... * support FA4 sm100 * flash attn backend support mask * flash attn backend run flashmask correct * add test for flash_attn_backend and flash_attn_func * check * add test for fa4 * requirements.txt add fa4 whl * check test on sm100 * fix CI conflict * add enable_torch_proxy for flash_mask * lazy import fa4 * check * fix tests import * check test_load_mpt import	2026-02-05 14:39:00 +08:00
CSWYF3634076	08c411518f	[Loader] support dummy load weight (#6169) ... * [Loader] support dummy load weight * [Loader] support dummy load weight v2 * [Loader] support dummy load weight unittest * [Loader] support dummy load weight unittest v2 * [Loader] support dummy load weight v3 docs and fp8	2026-01-26 13:58:53 +08:00
fxyfxy777	4c92035f2d	[Feature] Unify fp8 block_wise quant ops (#5991) ... * quant stash * blockwise_quant * precommit * rm tensor.cut * tp ok * add swiglu * rm outdate code * fix activate ut * change baseline * fix baseline error	2026-01-15 05:50:37 -08:00
bukejiyu	f0189292df	[CI] fix test_model_cache (#4982) ... * ci * update	2025-11-12 20:26:49 +08:00
YuBaoku	8a96944a0a	[CI] Update PORT range to avoid conflict with system ports (#4953)	2025-11-12 11:17:49 +08:00
bukejiyu	b09ebb2813	refactor pt loading (#4532)	2025-11-11 21:30:39 +08:00
YuBaoku	a139f8f3cb	[CI] Optimize port cleanup logic (#4860)	2025-11-06 19:13:48 +08:00
bukejiyu	41bfa1090d	[CI]delete test_common_model (#4794) ... * fix * update * update	2025-11-04 13:57:55 +08:00
bukejiyu	69c2f3cda1	[CI]test common model (#4697) ... * ut * update	2025-11-03 16:48:36 +08:00
CSWYF3634076	acd331780c	[V1 loader] Qwen25 VL support v1 loader and torch style safetensors load (#4388) ... * [BugFix] qwen2.5vl enable_thinking=true and image_patch_id bug fix * [Docs]offine infer add apply_chat_template add_generation_prompt parameter * [Model]qwen2.5VL support --use-cudagraph * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v2 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v3 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v4 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v5 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v6 * [Model]qwen2.5VL support --use-cudagraph buffer and qwenvl test v7 * qwen25vl v1 loader * qwen25vl v1 loader v2 * qwen25vl v1 loader v3 * qwen25vl v1 loader fix tp2 weight PySafeSlice * qwen25vl v1 loader no test * qwen25vl v1 loader add unit test * qwen25vl v1 loader add unit test v2 * qwen25vl v1 loader add torch unit test v3 * qwen25vl v1 loader add torch unit test v4 * qwen25vl v1 loader add torch unit test v5 * qwen25vl v1 loader add torch unit test v6	2025-10-27 10:54:15 +08:00
Zhang Yulong	83b720804b	Clean up ports after processing results (#4587)	2025-10-27 10:13:24 +08:00
Sunny-bot1	4ffe41a747	WINT4/WINT8 dense gemm default use Machete (#4451)	2025-10-23 17:57:59 +08:00
RAM	775edcc09a	[Executor] Default use CUDAGraph (#3594) ... * add start intercept * Adjustment GraphOptConfig * pre-commit * default use cudagraph * set default value * default use cuda graph * pre-commit * fix test case bug * disable rl * fix moba attention * only support gpu * Temporarily disable PD Disaggregation * set max_num_seqs of test case as 1 * set max_num_seqs and temperature * fix max_num_batched_tokens bug * close cuda graph * success run wint2 * profile run with max_num_batched_tokens * 1.add c++ memchecker 2.success run wint2 * updatee a800 yaml * update docs * 1. delete check 2. fix plas attn test case * default use use_unique_memory_pool * add try-except for warmup * ban mtp, mm, rl * fix test case mock * fix ci bug * fix form_model_get_output_topp0 bug * fix ci bug * refine deepseek ci * refine code * Disable PD * fix sot yaml	2025-10-21 14:25:45 +08:00
bukejiyu	62d1c48363	[v1 loader]code style (#4204) ... * code style * update	2025-09-23 19:36:00 +08:00
yangjianfengo1	4325b737e7	【FIX】Change the name of sparse attn from moba to plas (#4006) (#4076) ... * 【FIX】Change the name of sparse attn from moba to plas (#4006) * 更新文档 * 【docs】 update readme (#4000) * 更新文档 * update readme * update docs * 【FIX】Change the name of sparse attn from moba to plas (#3845) * 更新文档 * 更新文档 * 更新文档 * 更新文档 * 修改moba为plas * code style * update ci * code style * update ci * code style --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> * fix max_num_seqs * fix test load attn --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2025-09-23 10:26:40 +08:00
YuBaoku	2745f37017	[CI] enhance clean port and add waiting time (#4152)	2025-09-17 20:31:49 +08:00
Zhang Yulong	cd09913552	Update test_w4a8_model.py (#4125)	2025-09-16 20:43:10 +08:00
chenjian	67e6d8c691	[Feature] Set prefix caching as default (#3814) ... * Set prefix caching as default * Set prefix caching as default * Set prefix caching as default * skip dynamic load scene * fix kill bug * fix kill bug * fix kill bug * fix * fix * fix ci	2025-09-16 20:34:27 +08:00
co63oc	17a27170bc	fix typos (#4093)	2025-09-15 18:33:30 +08:00
bukejiyu	29ed617f0f	[v1 loader]qwen Offline fp8 (#4036) ... * support offline fp8 * update ut * update ut * update ut * fix * update * update	2025-09-15 13:44:11 +08:00
SuperNova	d60f7c4661	fix import tests.utils error in tests/model_loader/test_load_mtp.py (#4027) ... Co-authored-by: yongqiangma <xing.wo@163.com>	2025-09-11 16:47:16 +08:00
bukejiyu	2650f58740	[docs] Update environment variables documentation (#3957)	2025-09-10 21:17:06 -07:00
Sunny-bot1	3b1da6e4dd	support v1 loader for machete (#3999)	2025-09-10 10:21:33 +08:00
YuanRisheng	b3fac5bde1	[V1 Loader] Ernie kv cache quant support v1 loader (#3899) ... * support c8 for ernie * add unittest * support vl * fix c8	2025-09-09 05:25:08 -07:00
Zhang Yulong	2359c8d21c	update ci (#3962) ... Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-09-09 10:09:13 +08:00
bukejiyu	e52ce1c4b1	cache feature (#3857)	2025-09-07 18:52:46 +08:00
Zhang Yulong	349aa6348b	add cache queue port (#3904) ... * add cache queue port * add cache queue port * add cache queue port	2025-09-05 21:17:06 +08:00
Yuan Xiaolan	2cf55168ca	load hadamard_block_size from config (#3797)	2025-09-05 17:07:58 +08:00
AIbin	41aee08982	【Inference Optimize】Update MergedReplicatedLinear for DSK qkv_a_proj_with_mqa. (#3673) ... * support MergedReplicatedLinear * update MergedReplicatedLinear to support DSK_wint4 V1_load * update model name * update linear class * fix * fix v0 moe_bias load --------- Co-authored-by: bukejiyu <52310069+bukejiyu@users.noreply.github.com>	2025-09-04 21:16:05 -07:00
co63oc	6ac7cea81b	fix test_load_mtp (#3780)	2025-09-02 10:21:02 +08:00
YuanRisheng	6566e29807	Add loader test for mtp (#3724) ... * add test for mtp * fix unittest * fix	2025-09-01 10:55:49 +08:00
lizexu123	455205f991	[Features] support hugging face qwen3 moe (#3649) ... * split ut * qwen3-30B-A3B * fix * add test * add test_torch_model.py * fix test_torch_model.py * delete print * fix moe * delete init.py * fix * fix --------- Co-authored-by: bukejiyu <395822456@qq.com> Co-authored-by: bukejiyu <52310069+bukejiyu@users.noreply.github.com>	2025-08-30 15:26:05 +08:00
Yuan Xiaolan	c71ee0831c	add w4afp8 offline script (#3636)	2025-08-29 17:56:05 +08:00
周周周	17b414c2df	MoE Default use triton's blockwise fp8 in TP Case (#3678)	2025-08-29 11:07:30 +08:00
bukejiyu	73cf6096da	fix (#3676) ... * fix * update	2025-08-28 17:06:32 +08:00
bukejiyu	3200a80de3	[v1 loader]support fp8 (#3593) ... * support fp8 * update ci	2025-08-26 02:42:46 -07:00
lizexu123	c43a4bec00	[Features] support hugging face qwen3 dense and qwen2 model (#3574) ... * support qwen2 and qwen3 hugging face * fix moe * defualt_v1 loader * hugging_face_format deprecated * modify hugging_face_foramt to model_format * model_format auto * fix environemt * fix bug * fix qwen3-0.6 bug * model_format is str * fix	2025-08-26 10:54:53 +08:00
bukejiyu	bdbac0aa3d	support qwen2 weight only (#3571)	2025-08-24 11:14:34 +08:00
bukejiyu	77514e3e1e	[V1 Loader] support weight_only (#3413) ... * support wint4/wint8 * delete smoe case * update ci * print log	2025-08-23 13:13:41 +08:00
YuanRisheng	85fbf5455a	[V1 Loader]Ernie VL support loader v1 (#3494) ... * ernie vl support new loader * add unittest * fix test	2025-08-22 11:16:57 +08:00