80 Commits

Author	SHA1	Message	Date
fxyfxy777	4c92035f2d	[Feature] Unify fp8 block_wise quant ops (#5991) ... * quant stash * blockwise_quant * precommit * rm tensor.cut * tp ok * add swiglu * rm outdate code * fix activate ut * change baseline * fix baseline error	2026-01-15 05:50:37 -08:00
freeliuzc	49617d9832	[Feature]Support tag phase token enforce generation (#6034) ... * support tag phase token enforce generation * optimize note and some feature * fix sampler unit test --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2026-01-15 03:59:55 -08:00
lizexu123	6619298b50	【Optim】Optimize grid dimensions using max_tokens_per_expert for MoE models (#6007) ... * update w4afp8 * build.sh ok * support cuda_graph * fix * add test * fix max_tokens_per_expert * >=70 * fix * compute_max_tokens_from_prefix_sum in w4afp8 * compute_max_tokens use cub	2026-01-15 19:18:42 +08:00
Daci	e10b51b8c6	[Feature] get_output_kv_signal blocking read mode & send_first_token (#5836) ... * get_output_kv_signal blocking read mode * send first token before recycle * xpu get_output_kv_signal blocking read mode --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com>	2026-01-15 14:11:03 +08:00
chenjian	74d0f1c01f	[Optim] Robust sync status when preempted happens (#5796) ... * [Bug fix] Sync status for caching output cache * fix * fix * fix bug * fix * fix * support xpu * fix * fix * fix * fix * fix * fix ci * fix ci * fix xpu --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2026-01-14 12:07:33 +08:00
周周周	a3f0696e35	[BugFix] fix compile error in sm89 (#5809)	2025-12-29 16:55:52 +08:00
Ryan	09229d8953	change count_tokens_per_expert_func declaration: Tensor -> vector<Tensor> (#5794)	2025-12-26 19:02:28 +08:00
周周周	03363cab4c	make flash_mask attention pybind (#5783)	2025-12-26 14:31:35 +08:00
freeliuzc	9018ccf74e	[Speculative Decoding] Fix attn_mask_offset for multi-step MTP in mixed and PD-split modes (#5738) ... * fix attn_mask_offset in mtp with multi-step and pd-split-mode * fix xpu operater register * update pmtp multi-step mtp strategy in d-split -mode * add note * fix xpu register	2025-12-25 01:54:59 -08:00
Yuanle Liu	cdc0004894	Revert "[Feature] add ue8m0 for per_token_quant_fp8 (#5563)" (#5611) ... This reverts commit 73e1d6aa90.	2025-12-17 13:59:06 +08:00
Yuanle Liu	867803ae10	[BugFix] fix speculate_limit_thinking_content_length (#5590) ... * fix speculate_limit_thinking_content_length * update	2025-12-16 04:31:45 -08:00
fxyfxy777	73e1d6aa90	[Feature] add ue8m0 for per_token_quant_fp8 (#5563) ... * ue8m0 * add default arg --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-12-16 18:40:12 +08:00
lizexu123	95eab9f9ee	[Feature] support stop_token_ids (#5399) ... * support stop_token_ids * fix * delete chinese * support both * delete print	2025-12-09 17:49:12 +08:00
周周周	31410415db	FA3 support qwen3 (#5441)	2025-12-09 16:16:16 +08:00
K11OntheBoat	8d99bac532	Remove CUDA ERROR 9 of inputs of get_padding_offset kernel (#5440) ... Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-12-09 14:17:30 +08:00
周周周	2aea8a3a60	[Others] Remove useless code (#5404)	2025-12-08 13:59:46 +08:00
K11OntheBoat	2e1680838f	[PD Disaggregation] Support PD deployment of DeepSeekv3. (#5251) ... * Support deepseekv3 cache transfer for PD deploy * clean some log info --------- Co-authored-by: K11OntheBoat <“ruianmaidanglao@163.com”>	2025-12-02 14:11:50 +08:00
lizhenyun01	aba4fc657f	[Feature] support flash_mask_attention backend (#5134) ... * [Feature] suppert flash_mask_attention backend * fix unittest * clean code	2025-11-28 10:12:16 +08:00
xiaoxiaohehe001	6ca2651995	[Feature] Support noaux for eplb (#5143) ... * support noaux eplb * noaux_eplb * noaux_eplb * noaux_eplb	2025-11-21 14:10:32 +08:00
周周周	6fa34102e8	[Others]get_block_shape_and_split_kv_block clean code (#5123)	2025-11-20 16:40:04 +08:00
yangjianfengo1	ae7bee8122	【New Feature】W4afp8 supports per group quantization (#4987) ... * w4afp8 支持per group * code style * fix transpose * revert fast hardmard --------- Co-authored-by: yuanxiaolan <yuanxiaolan01@baidu.com> Co-authored-by: plusNew001 <95567040+plusNew001@users.noreply.github.com>	2025-11-13 19:17:27 +08:00
ming1753	cba185f1fe	[Feature] Optim PaddleOCR-VL (#4873) ... * [Feature] Optim PaddleOCR-VL * fix bug	2025-11-07 14:56:44 +08:00
YuBaoku	819b2dbbae	Revert "【New Feature】W4afp8 supports per group quantization (#4272)" (#4854) ... This reverts commit 93fcf7e4ec.	2025-11-06 17:48:28 +08:00
yangjianfengo1	93fcf7e4ec	【New Feature】W4afp8 supports per group quantization (#4272) ... * w4afp8 支持per group * code style * 精度完成 * revert append attn utils * ffn1 动态量化 * ffn2 支持动态量化 * code style * code style * 修改单测 * 修改单测 * fix bug * Implement conditional parameter creation for layers Add parameter creation for up_gate_proj_in_scale when ep_size > 1. * code style * fix conflict * code style * code style * 修复w4aint8 精度 * fix ci --------- Co-authored-by: yuanxiaolan <yuanxiaolan01@baidu.com>	2025-11-05 21:00:23 +08:00
周周周	937eb3c6ed	[get_padding_offset.] clean get_padding_offset.cu (#4777) ... [get_padding_offset.] clean get_padding_offset.cu (#4777)	2025-11-05 10:47:40 +08:00
freeliuzc	11398790d3	[Speculative Decoding][MTP]Support attn mask offset (#4641) ... * [MTP]Merge support attn (#4591) * support mask_offset in speculate decoding * fix dummpy run output * add unit test * fix unit test import * support attn_mask_offset in mtp mode * add update_attn_mask op * fix unit test && fix code-style	2025-11-03 10:08:01 +08:00
Yuanle Liu	b301bd6c31	[BugFix] fix thinking bug (#4710) ... * fix thinking bug * fix ut * update * fix	2025-10-31 22:00:31 +08:00
GoldPancake	1f3ce65b58	[Feature] support mtp distribution equivalence verification (#4699)	2025-10-31 11:45:04 +08:00
Ryan	07956a87b3	[Graph Optimization] Fix IR graph dependency error exposed after enabling SOT by updating the return value of TextImageGatherScatter (#4610) ... * fix TextImageGatherScatter in sot * fix codestyle	2025-10-28 18:31:23 +08:00
Lucas	5c6105f4a2	[XPU] bind some OPs for VL model with pybind (#4522)	2025-10-27 10:50:08 +08:00
Yuanle Liu	cef3164c3b	Optimizing the performance of think length limit using custom operators (#4279) ... * delete impl * delete min_length&max_length * support limit thinking content strategy * fix * fix * fix * update * fix set_value_by_flags_and_idx * fix * fix * fix * fix * update * fix * fix * fix typo * fix ci * fix * fix * support mtp * fix * fix * update * update	2025-10-20 21:09:13 +08:00
GoldPancake	47595a2480	[Feature] support mtp logprob (#4464) ... * support mtp logprob * fix unitest	2025-10-20 15:18:12 +08:00
Haonan Luo	1b9f351d21	Support GPT-OSS-BF16 (#4240) ... * [Feature] AppendAtten support sinks & HEAD_DIM=64 * fix bug * fix bug * fix bug * fix bug * [Feature] support gpt-oss * fix bug * add mask * support-gpt-oss * support-gpt-oss * fix long seq * support wint8 * support wint8 * support wint8 * update test * change sliding windows init pos --------- Co-authored-by: ming1753 <ideaminghp@163.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com> Co-authored-by: ming1753 <61511741+ming1753@users.noreply.github.com>	2025-10-20 14:44:58 +08:00
yzwu	4b661512ca	[Iluvatar GPU] Adapt VL model (#4313)	2025-10-17 16:13:38 +08:00
Ryan	49cea8fb1c	[SOT][Cudagraph] Remove BreakGraph of #3302 && update CustomOp (#3694) ... * rm inplace info && to(gpu) * update append_attention * unpin paddle version * add full_cuda_graph=False * add blank line --------- Co-authored-by: SigureMo <sigure.qaq@gmail.com>	2025-10-17 10:57:55 +08:00
freeliuzc	582aebd48b	[MTP]support mtp chunk_prefill_v1 (#4366) ... * support mtp chunk_prefill_v1 * fix mtp chunkprefill output, fix unit test * fix unit test * fix save_output	2025-10-15 13:21:32 +08:00
Sunny-bot1	a751d977bc	[Optimization] Fuse get_max_len and get_kv_max_len (#4369) ... * opt split_q_block * fuse max_lens and max kv_len	2025-10-13 20:35:00 +08:00
RAM	aa27b03bc0	[Executor]CUDAGraph support Speculate Decode (#3769) ... * success run ngram * Revert "[Code Simplification] remove cum_offsets (#3410)" This reverts commit 32b39620bc. * success run ngram5 tp4 42bs * success run ngram5 tp4 42bs * mtp draft commit * add decorator for target model * enable draft model in cudagraph v0.5 * revert revrt cum_offset * enable target model in cudagraph v0.9 And clean debug code * Revert "success run ngram" This reverts commit 8351e83993. * add reverted code * enable target model in cudagraph v0.9 * solve comment * fix bid < 0 * Enable Target Model Padding And Draft Model in cudagraph * solve problem * delete rebuild padding debug note * fast compile * Add capture list for mtp * success run 256 tp1 mtp * Enable Lite TP2 Bsz256 * realy enable tp2 bsz 256 * fix problem * Solve problem for Draft model in cudagraph * Solve comment * replace emptytensor as zeros * Solve comments * Revert "fast compile" This reverts commit 834639a7ff. * fix bug * fix merge bug * fix typo * fix bug --------- Co-authored-by: lizexu <2694294196@qq.com> Co-authored-by: littledgg <1658565283@qq.com> Co-authored-by: zeroRains <linjunlu@zerorains.top> Co-authored-by: gongshaotian <gstain5555@outlook.com>	2025-10-09 21:18:29 +08:00
chen	81959c7d88	[NewFeature]custom_allreduce support cudagraph recapture (#4305) ... * custom_allreduce support cudagraph recapture * add shut_down/restart default group	2025-09-29 15:56:54 +08:00
chen	1a6283424e	Fix noaux_tc cuda Error 700 in CUDAGraph (#4174)	2025-09-23 18:41:33 +08:00
AIbin	a7392a0ff9	【Inference Optimize】DeepSeek-V3-model MLA Optimize (#3886) ... * support MLA chunk_size auto search & cuda_graph	2025-09-11 10:46:09 +08:00
Ayakouji	453487d5b0	[Feat] ernie4_5_vl_moe support CudaGraph (#3226) ... * delete dynamic control flow for decode * coda-style * fix scatter/gather typos and use input stream instead default stream * support 0-Size Tensor * update runner and model * using static mem address as input * fix mem leak * refine code * update mm_buffer * fix typo * fix buffersize * fix unk token * refine code * refine * support other arch * open cudagraph in vlci * fix * update * update * update * fix cmd * update --------- Co-authored-by: aquagull <hongyuh@qq.com> Co-authored-by: Yuanle Liu <yuanlehome@163.com>	2025-09-10 13:11:57 +08:00
Jundong Liu	3d0aaa5923	[Excutor] Experiment Feature-Support Prefill in cudagraph (#3459) ... * Support prefill in Cudagraph * Refactor GetBlockShapeAndSplitKVBlock Kernel V2 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.1 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.2 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.3 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.4 * Refactor GetBlockShapeAndSplitKVBlock Kernel V2.5 * Solve problem about encoder_num_blocks_x_cpu * Add early-exit mechanism for attention kernel * fix test case about append-attention * Update testcode, Add annotations to related tensors * move get_input_length_list * solve test_code * Add annotations about early-exit for attention kernel * Add annotations about early-exit for attention kernel2 * solve comment * solve mtp --------- Co-authored-by: RAM <gstian5555@outlook.com>	2025-09-08 13:12:24 +08:00
Yuan Xiaolan	2cf55168ca	load hadamard_block_size from config (#3797)	2025-09-05 17:07:58 +08:00
freeliuzc	88d44a2c93	support mtp in v1_scheduler mode (#3695)	2025-09-04 17:39:59 +08:00
co63oc	5441538173	rename fused_get_rope.cu (#3752) ... * rename fused_get_rope.cu * fix * fix typos * fix * fix	2025-09-03 10:54:34 +08:00
co63oc	d6369b4d51	fix typos (#3684)	2025-09-01 17:50:17 +08:00
Sunny-bot1	fe5d09f9ee	[FIX]Fix Machete compile via ENABLE_MACHETE (#3727) ... * add ENABLE_MACHETE * fix * revert * update * pre_commit * fix * fix --------- Co-authored-by: Ayakouji <yuhongh@qq.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: aquagull <hongyuh@qq.com>	2025-08-30 17:50:17 +08:00
yangjianfengo1	3754a9906d	[Feature] block sparse attention (#3668) ... * 支持稀疏attn * fix bug * code style * fix moba attn get kv shape * 修复a100编译 * codestyle * code style * code style * code style * fix conflict * 增加单侧 * code style * 增加eblite 加载时间 * fix bug * for ci * for ci * for ci * for ci * 支持mlp block size 128 * 增加小算子单测 * fix 单测 mlp * 将环境变量加入到config里面 * fix rollout config * 修复显存 * add test server * add test server * fix mlp 最后一层使用full attn	2025-08-29 19:46:30 +08:00
Liumengyuan	e93d4cfcdd	Add with_output version AppendAttention (#3302) ... * get use_output from fd_config * add clear TODO description * add mask_offset para to align with develop * fix bug * fix use_output logic * fix sot bug	2025-08-28 17:10:18 +08:00