67 Commits

Author	SHA1	Message	Date
chen	29a313a402	[Optimization] Support FA2/FA3/FA4 with attn_mask_q (#6354) ... * support FA4 sm100 * flash attn backend support mask * flash attn backend run flashmask correct * add test for flash_attn_backend and flash_attn_func * check * add test for fa4 * requirements.txt add fa4 whl * check test on sm100 * fix CI conflict * add enable_torch_proxy for flash_mask * lazy import fa4 * check * fix tests import * check test_load_mpt import	2026-02-05 14:39:00 +08:00
fxyfxy777	36547cfdb3	[Feature] FD_USE_PHI_FP8_QUANT (#6320) ... * add ut * add use_fd_quant env * rm mask_per_token_quant * add make ops list * USE_FD_FP8_QUANT -> FD_USE_PHI_FP8_QUANT 默认是true * modify comments * use bool type * Add function declaration	2026-02-03 22:33:03 -08:00
fxyfxy777	2ada119a38	[Optimize] optimize mask_quant & swiglu (#6222) ... * optimize mask_quant op speed up 1.5 * fix calculate sequence * add fused * rm log * push kernel code * add ut * accuracy ok * add ue8m0 * add ut * add merge develop * rm ut of mask_per_token_quant	2026-02-02 13:52:38 +08:00
xiaozude	030647521a	[Metax] adapt to the latest develop (#6282)	2026-01-29 23:21:20 -08:00
sunxin	27f8799f04	[Model Runner] Refactor execute_model for GPU async scheduling (#6176)	2026-01-28 14:19:33 +08:00
freeliuzc	49617d9832	[Feature]Support tag phase token enforce generation (#6034) ... * support tag phase token enforce generation * optimize note and some feature * fix sampler unit test --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2026-01-15 03:59:55 -08:00
MingkunZhang	7ad5737560	[Metax] adapt to gemm interface on different versions of maca (#5905) ... Co-authored-by: root <root@lt-wks-10-0-180-15.pub.metax-tech.com>	2026-01-07 10:02:24 +08:00
Neil Zhu	272a371635	[Metax] optimize flash attention backend (#5876)	2026-01-06 09:52:09 +08:00
MingkunZhang	f732d7d2ad	[Metax] adapt prefix caching & cpu swap (#5844) ... Co-authored-by: root <root@lt-wks-10-0-180-15.pub.metax-tech.com>	2025-12-31 17:02:48 +08:00
周周周	a3f0696e35	[BugFix] fix compile error in sm89 (#5809)	2025-12-29 16:55:52 +08:00
Juncai	412867fd99	[Feature] Support KV Cache Storage (#5571) ... * Support Mooncake Store * up * up * add op * fix conflict * fix error * up for comments * avoid thread lock * up * fix unittest * fix unittest * remove debug info * consider tp_size > 1 * add default rdma_nics * add utils * up * fix error --------- Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-12-25 16:30:35 +08:00
yzwu	ac013803f3	[Iluvatar] Support V1_KVCACHE_SCHEDULER and paddleocr-vl rope mode (#5555)	2025-12-18 02:14:25 -08:00
FocusLuo	c3aaa7e441	[BugFix] Fixed build script issue on Intel HPU platforms (#5455) ... * [INTEL HPU] Fixed build script issue for non-gpu platforms Signed-off-by: Luo, Focus <focus.luo@intel.com> * [INTEL HPU] PR CI HPU will not use fixed version of fastdeploy_intel_hpu Signed-off-by: Luo, Focus <focus.luo@intel.com> --------- Signed-off-by: Luo, Focus <focus.luo@intel.com> Co-authored-by: YuBaoku <49938469+EmmonsCurse@users.noreply.github.com>	2025-12-11 16:36:37 +08:00
Neil Zhu	4403a21d4b	[Metax] refactor cutlass moe and optimize flash attention (#5361) ... * [Metax] refactor moe and flash attention backend --------- Co-authored-by: zhangchenyi_dl <16219492+zhangchenyidl@user.noreply.gitee.com>	2025-12-10 17:15:17 +08:00
xiaoxiaohehe001	6ca2651995	[Feature] Support noaux for eplb (#5143) ... * support noaux eplb * noaux_eplb * noaux_eplb * noaux_eplb	2025-11-21 14:10:32 +08:00
Neil Zhu	0edda75a56	[Metax] optimize cutlass moe and flash attention backend (#5128)	2025-11-20 16:12:35 +08:00
yzwu	3707af7a4f	[Iluvatar] add vl into ci and support v1 loader (#4774)	2025-11-11 10:50:17 +08:00
Neil Zhu	6de1ce3b25	[Metax] support ERNIE-4.5-VL-28B (#4820)	2025-11-07 04:55:49 -08:00
ming1753	cba185f1fe	[Feature] Optim PaddleOCR-VL (#4873) ... * [Feature] Optim PaddleOCR-VL * fix bug	2025-11-07 14:56:44 +08:00
Neil Zhu	c95d0740ec	[Metax] adapt cutlass moe for ernie-vl (#4685)	2025-11-03 17:44:27 +08:00
freeliuzc	11398790d3	[Speculative Decoding][MTP]Support attn mask offset (#4641) ... * [MTP]Merge support attn (#4591) * support mask_offset in speculate decoding * fix dummpy run output * add unit test * fix unit test import * support attn_mask_offset in mtp mode * add update_attn_mask op * fix unit test && fix code-style	2025-11-03 10:08:01 +08:00
ming1753	27746026c1	Skip building native architecture when specifying arch list (#4727)	2025-10-31 20:32:46 +08:00
Zhenghai Zhang	1712e1351b	【Hackathon 9th No.86】autogen MoeFastHardamardImplWrapper template_instantiation (#4592) ... * autogen MoeFastHardamardImplWrapper template_instantiation * fix codestyle * fix codestyle * add impl cu files	2025-10-30 10:28:36 +08:00
xiaozude	f7069b8057	[Metax] adapt DeepSeek (#4498)	2025-10-24 10:14:53 +08:00
RAM	775edcc09a	[Executor] Default use CUDAGraph (#3594) ... * add start intercept * Adjustment GraphOptConfig * pre-commit * default use cudagraph * set default value * default use cuda graph * pre-commit * fix test case bug * disable rl * fix moba attention * only support gpu * Temporarily disable PD Disaggregation * set max_num_seqs of test case as 1 * set max_num_seqs and temperature * fix max_num_batched_tokens bug * close cuda graph * success run wint2 * profile run with max_num_batched_tokens * 1.add c++ memchecker 2.success run wint2 * updatee a800 yaml * update docs * 1. delete check 2. fix plas attn test case * default use use_unique_memory_pool * add try-except for warmup * ban mtp, mm, rl * fix test case mock * fix ci bug * fix form_model_get_output_topp0 bug * fix ci bug * refine deepseek ci * refine code * Disable PD * fix sot yaml	2025-10-21 14:25:45 +08:00
Yuanle Liu	cef3164c3b	Optimizing the performance of think length limit using custom operators (#4279) ... * delete impl * delete min_length&max_length * support limit thinking content strategy * fix * fix * fix * update * fix set_value_by_flags_and_idx * fix * fix * fix * fix * update * fix * fix * fix typo * fix ci * fix * fix * support mtp * fix * fix * update * update	2025-10-20 21:09:13 +08:00
yzwu	4b661512ca	[Iluvatar GPU] Adapt VL model (#4313)	2025-10-17 16:13:38 +08:00
Zhenghai Zhang	6adfbe07ad	【Hackathon 9th No.86】autogen MultiQueryDecoderAttention template_instantiation -part (#4383) ... * split MultiQueryDecoderAttention template_instantiation * update comment * CI	2025-10-16 17:08:19 +08:00
Zhenghai Zhang	c46d5e48f8	【Hackathon 9th No.86】autogen MultiQueryAppendC8Attention template_instantiation -part (#4330) ... * split MultiQueryAppendC8Attention template_instantiation * update setup_ops.py * fix ci * fix bug	2025-10-10 15:07:48 +08:00
xiaozude	7c919070f7	[Metax] support cutlass moe & optimize flash attention (#4208)	2025-09-29 11:22:43 +08:00
李泳桦	6265f4385f	[feat] support prefix cache clearing when /clear_load_weight is called (#4008) ... * [feat] support clearing prefix cache (cherry-picked from release/2.1) * [fix] fix ipc suffix, use port instead * [fix] fix prefix caching not enabled * [fix] fix key/value_cache_scales indent * [fix] fix ep group all-reduce * [fix] fix clear/update lock not working when workers > 1 * [chore] add preemption triggered info log * [fix] fix code style * [fix] fix max_num_seqs config * [fix] do not force enable_prefix_caching=False in dynamic loading * [fix] fix ci * Revert "[fix] fix ci" This reverts commit 0bc6d55cc8. * [fix] initialize available_gpu_block_num with max_gpu_block_num * [fix] fix config splitwise_role * [fix] fix clearing caches synchronization and add more logs * [chore] print cache_ready_signal in log * [fix] fix scheduler_config.splitwise_role * [fix] fix cache_messager cache_ready_signal create=True * [fix] stop cache messager from launching in mixed deployment	2025-09-28 19:42:53 +08:00
fmiao2372	f1b5392e20	[Intel HPU] Support intel hpu platform (#4161) ... * [Intel HPU] Support intel hpu platform * fix some issues * apply precommit and move AttentionBackend_HPU * fix format issue * correct ops import * fix ci issue * update code in layers * fix code style issue * remove dense tp moe ep mode * fix enc_dec_block_num * fix rebase issue * rename hpu to gaudi in readme * rename ForwardMeta_HPU to HPUForwardMeta	2025-09-24 12:27:50 +08:00
yzwu	504461b6b5	[Iluvatar GPU] Optimize attention performance and fix moe load ckpt error (#3651)	2025-09-22 21:13:59 +08:00
YuanRisheng	d2d04c2d5e	[setup optimize]Support git submodule (#4033) ... * support git submodule * update setup * fix ci network * fix clone * revert clone linux * delete args * fix ci * update	2025-09-11 17:41:16 +08:00
zhupengyang	9d0074a91a	[xpu] add ep custom ops (#3911)	2025-09-10 12:22:50 +08:00
co63oc	5441538173	rename fused_get_rope.cu (#3752) ... * rename fused_get_rope.cu * fix * fix typos * fix * fix	2025-09-03 10:54:34 +08:00
co63oc	d6369b4d51	fix typos (#3684)	2025-09-01 17:50:17 +08:00
Sunny-bot1	fe5d09f9ee	[FIX]Fix Machete compile via ENABLE_MACHETE (#3727) ... * add ENABLE_MACHETE * fix * revert * update * pre_commit * fix * fix --------- Co-authored-by: Ayakouji <yuhongh@qq.com> Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: aquagull <hongyuh@qq.com>	2025-08-30 17:50:17 +08:00
yangjianfengo1	3754a9906d	[Feature] block sparse attention (#3668) ... * 支持稀疏attn * fix bug * code style * fix moba attn get kv shape * 修复a100编译 * codestyle * code style * code style * code style * fix conflict * 增加单侧 * code style * 增加eblite 加载时间 * fix bug * for ci * for ci * for ci * for ci * 支持mlp block size 128 * 增加小算子单测 * fix 单测 mlp * 将环境变量加入到config里面 * fix rollout config * 修复显存 * add test server * add test server * fix mlp 最后一层使用full attn	2025-08-29 19:46:30 +08:00
Sunny-bot1	479c8b85d3	[Optimize]support machete weight only gemm (#3561) ... * support machete weight only gemm * add generate * update * fix * change file location * add sm_version limit * fix * fix * fix ci * fix coverage * fix xpu	2025-08-28 09:49:58 +08:00
Liumengyuan	11803e0907	fix undefined cuPointerGetAttribute symbol error (#3628)	2025-08-27 20:24:59 +08:00
Jiang-Jia-Jun	c694fa2879	Revert "[Feature] block sparse attention (#3209)" (#3647) ... This reverts commit 646a0c2fd8.	2025-08-27 17:35:04 +08:00
yangjianfengo1	646a0c2fd8	[Feature] block sparse attention (#3209) ... * 支持稀疏attn * fix bug * code style * fix moba attn get kv shape * 修复a100编译 * codestyle * code style * code style * code style * fix conflict * 增加单侧 * code style * 增加eblite 加载时间 * fix bug * for ci * for ci * for ci * for ci * 支持mlp block size 128 * 增加小算子单测 * fix 单测 mlp * 将环境变量加入到config里面 * fix rollout config	2025-08-26 07:16:04 -07:00
Kane2011	2ae7ab28d2	[MetaxGPU] adapt to the latest fastdeploy on metax gpu (#3492)	2025-08-25 17:44:20 +08:00
yangjianfengo1	b047681c5d	【New Feature】支持Fp8 group Gemm 24稀疏 (#3463) ... * 支持24稀疏 * code style * 增加stmatrix 宏定义判断 * code style	2025-08-19 02:54:47 -07:00
chen	f0f00a6025	[OPs] Universal optimization and Fix early_stop cuda 700 (#3375) ... * delete nonzero * delete setup_ops_base.py * check if * check gcp infer_seed.cpu() * fix repetition_early_stopper_kernel cuda 700	2025-08-14 22:40:44 +08:00
Kane2011	b4fef2cf29	[MetaxGPU] Support FastDeploy on metax gpu (#3241) ... * [MetaxGPU] Support FastDeploy on metax gpu * Update metax_worker.py 1. change worker log; 2. remove custom allreduce, adapt it later; 3. remove cuda graph; * Update __init__.py 1. remove metax's key work comment * Update __init__.py 1. remove metax's key word comment; 2. add fused_moe_kernel_paddle import --------- Co-authored-by: yongqiangma <xing.wo@163.com>	2025-08-13 11:11:54 +08:00
yzwu	fbdd6b0663	[Iluvatar GPU] Optimze attention and moe performance (#3234)	2025-08-08 10:51:24 +08:00
yangjianfengo1	89397516a8	[New Feature] Support W4Afp8 MoE GroupGemm (#3171) ... * init * 增加多线程编译 * fix bug * fix bug * code style * 增加fp16 * 将print替换成assert * 修复stmatrix * 减小单测shape * 减小单测shape	2025-08-06 10:34:05 +08:00
yangjianfengo1	40f7f3e0d8	[New Feature] fa3 支持flash mask (#3184) ... * 支持flash mask * 修改test_flash_mask * 修改test.sh	2025-08-05 12:20:48 +08:00