Zero Rains 89a485b69f [Feature] Support using prefix-caching + cudagraph for inference (#2924) ...

* fix the bug in cudagraph+prefix-caching but still have some bug with profile

Change-Id: Ibf2ba3f2e3b08641d03f4b1391d7c862c3efa397

* add the signal to make sure cache manager launched

* fix judge condition

* reomove useless control

* update control stream

* update

* fix xpu

* change the do_profile flag

* update

* add new threads to init cache_manager

---------

Co-authored-by: RAM <gstian5555@outlook.com>

2025-07-22 00:59:45 -07:00

..

__init__.py

polish code with new pre-commit rule (#2923)

2025-07-19 23:19:27 +08:00

args_utils.py

[Feature] Support using prefix-caching + cudagraph for inference (#2924)

2025-07-22 00:59:45 -07:00

config.py

custom all reduce support cuda graph (#2938)

2025-07-21 22:52:03 +08:00

engine.py

[Feature] Support using prefix-caching + cudagraph for inference (#2924)

2025-07-22 00:59:45 -07:00

expert_service.py

polish code with new pre-commit rule (#2923)

2025-07-19 23:19:27 +08:00

kv_cache_interface.py

polish code with new pre-commit rule (#2923)

2025-07-19 23:19:27 +08:00

request.py

[Feature] Add return_token_ids, prompt_token_ids, and delete training, raw_request in request body (#2940)

2025-07-21 19:31:14 +08:00

resource_manager.py

polish code with new pre-commit rule (#2923)

2025-07-19 23:19:27 +08:00

sampling_params.py

[Feature] support min_p_sampling (#2872)

2025-07-20 23:17:59 -07:00