SGLang issues | Ecosyste.ms: OpenCollective

Provide an offline engine API

github.com/sgl-project/sglang - ByronHsu opened this pull request 3 months ago

Use ipc instead of tcp in zmq

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[doc] Chinese Documentation Translation Available for sglang

github.com/sgl-project/sglang - khum08 opened this issue 3 months ago

[Feature] Add `choices` in `/generate` endpoint and add `min_new_tokens` in `sgl.gen()`

github.com/sgl-project/sglang - TING2938 opened this issue 3 months ago

[Fix] Fix major performance bug in certain cases

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

Organize sampling batch info better

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

Add llama implementation with no tensor parallel linears

github.com/sgl-project/sglang - jerryzh168 opened this pull request 3 months ago

Print out what the model saw?

github.com/sgl-project/sglang - cinjon opened this issue 3 months ago

[FP8 KV Cache] Avoid KeyError at loading pre-quantized FP8 model with kv_scale

github.com/sgl-project/sglang - HaiShaw opened this pull request 3 months ago

[Bug] Exception: Capture cuda graph failed: Triton Error [CUDA]: device kernel image is invalid

github.com/sgl-project/sglang - a136214808 opened this issue 3 months ago

Move status check in the memory pool to CPU

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[Fix] Move ScheduleBatch out of SamplingInfo

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[Fix] do not maintain regex_fsm in SamplingBatchInfo

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[Performance, Hardware] MoE tuning on AMD MI300x GPUs

github.com/sgl-project/sglang - kkHuang-amd opened this pull request 3 months ago

[Fix] Fix all the Huggingface paths

github.com/sgl-project/sglang - tbarton16 opened this pull request 3 months ago

Simplify flashinfer dispatch

github.com/sgl-project/sglang - hnyls2002 opened this pull request 3 months ago

Llama3.2 vision model support

github.com/sgl-project/sglang - hnyls2002 opened this pull request 3 months ago

Dispatch flashinfer wrappers

github.com/sgl-project/sglang - hnyls2002 opened this pull request 3 months ago

[Refactor] Simplify io_struct and tokenizer_manager

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

Fix bugs of `logprobs_nums`

github.com/sgl-project/sglang - hnyls2002 opened this pull request 3 months ago

Organize Attention Backends

github.com/sgl-project/sglang - hnyls2002 opened this pull request 3 months ago

Support qwen2 vl model

github.com/sgl-project/sglang - yizhang2077 opened this pull request 3 months ago

[Fix, LoRA] fix LoRA with updates in main

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

Clean up batch data structures: Introducing ModelWorkerBatch

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

Rename InputMetadata -> ForwardBatch

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

Add support for Molmo-D-7B Model

github.com/sgl-project/sglang - BabyChouSr opened this pull request 3 months ago

Let ModelRunner take InputMetadata as input, instead of ScheduleBatch

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[Refactor] Simplify io_struct and tokenizer_manager

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

Process image in parallel

github.com/sgl-project/sglang - hnyls2002 opened this pull request 3 months ago

Move scheduler code from tp_worker.py to scheduler.py

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

fix ipv6 url when warm up model

github.com/sgl-project/sglang - cauyxy opened this pull request 3 months ago

[Fix] Fix AttributeError in Qwen2.5 LoRA: 'Qwen2ForCausalLM' object has no attribute 'get_hidden_dim'

github.com/sgl-project/sglang - mssongit opened this pull request 3 months ago

[Fix] Fix AttributeError in Qwen2.5(huggingface model) LoRA: 'Qwen2ForCausalLM' object has no attribute 'get_module_name'

github.com/sgl-project/sglang - mssongit opened this pull request 3 months ago

Improve process creation

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[Bug] ValueError: The memory capacity is unbalanced

github.com/sgl-project/sglang - chuangzhidan opened this issue 3 months ago

Make detokenizer_manager.py not asyncio

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

Organize image inputs

github.com/sgl-project/sglang - hnyls2002 opened this pull request 3 months ago

Multiple minor fixes

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[Event] Update meeting link

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

Add float8 dynamic quant to torchao_utils

github.com/sgl-project/sglang - jerryzh168 opened this pull request 3 months ago

[Feature] VLLM 6.0 support

github.com/sgl-project/sglang - arunpatala opened this issue 3 months ago

[Bug] IndexError: list index out of range

github.com/sgl-project/sglang - lvxianfeng-git opened this issue 3 months ago

[Feature] Support reward model LxzGordon/URM-LLaMa-3.1-8B

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

minor: fix config

github.com/sgl-project/sglang - hnyls2002 opened this pull request 3 months ago

[Feature] add support for llama 3.2

github.com/sgl-project/sglang - Stealthwriter opened this issue 3 months ago

[Bug] Unable to use gptq or awq with torch.compile (8*A40)

github.com/sgl-project/sglang - smallstepman opened this issue 3 months ago

[FIX] Catch syntax error of Regex Guide to avoid crash

github.com/sgl-project/sglang - du00cs opened this pull request 3 months ago

[bugfix]Add modelscope package to avoid docker image without modelscope

github.com/sgl-project/sglang - KylinMountain opened this pull request 3 months ago

Accuracy reduction of Lora

github.com/sgl-project/sglang - yileld opened this issue 3 months ago

Update Dockerfile

github.com/sgl-project/sglang - KylinMountain opened this pull request 3 months ago

[Bug] no module modelscope using docker compose to start sglang

github.com/sgl-project/sglang - KylinMountain opened this issue 3 months ago

How to study the code?

github.com/sgl-project/sglang - TJ949 opened this issue 3 months ago

[Feature] _get_pixel_values needs to return tgt_sizes

github.com/sgl-project/sglang - huangzl18883 opened this issue 3 months ago

[Fix] Ignore model import error

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

Release v0.3.2

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

Revert "kernel: use tensor cores for flashinfer gqa kernels"

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

[Fix] Fix clean_up_tokenization_spaces in tokenizer

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[Bug] tensor parallel run error

github.com/sgl-project/sglang - jerryzh168 opened this issue 3 months ago

Add support for tie_word_embeddings when loading weights + support for SmolLM

github.com/sgl-project/sglang - TianyiQ opened this pull request 3 months ago

[CI] Update nightly eval

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

[Bug] LLaVa-next does not work for single image processing

github.com/sgl-project/sglang - ThomasBenzshawel opened this issue 3 months ago

AWQ performance tracking

github.com/sgl-project/sglang - zhyncs opened this issue 3 months ago

Possible timing side-channels caused by shared prefix

github.com/sgl-project/sglang - Unik-lif opened this issue 3 months ago

Simplify bench_latency.py

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

Update test_srt_backend.py

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

[Bug] radixcache stack_overflow

github.com/sgl-project/sglang - luzengxiangcn opened this issue 3 months ago

[CI] Move AMD test to a separate file

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

debug radixcache stack_overflow

github.com/sgl-project/sglang - luzengxiangcn opened this pull request 3 months ago

Speculative decoding with EAGLE2

github.com/sgl-project/sglang - yukavio opened this pull request 3 months ago

MoE torch compile

github.com/sgl-project/sglang - ispobock opened this pull request 3 months ago

Fix the overhead due to penalizer in bench_latency

github.com/sgl-project/sglang - merrymercy opened this pull request 3 months ago

Fix RuntimeEndpoint.select method

github.com/sgl-project/sglang - jeffrey-fong opened this pull request 3 months ago

minor: add mla fp8 test

github.com/sgl-project/sglang - zhyncs opened this pull request 3 months ago

[Community] Add open collective sponsor link to README

github.com/sgl-project/sglang - Ying1123 opened this pull request 3 months ago

Update dockerfile to include datamodel_code_generator

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Add AMD tests to CI

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

[API, Feature] Support response prefill for openai API

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

Add a unit test for data parallelism

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Better unit tests for adding a new model

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Development Roadmap (2024 Q4)

github.com/sgl-project/sglang - Ying1123 opened this issue 4 months ago

doc: update backend

github.com/sgl-project/sglang - zhyncs opened this pull request 4 months ago

[Bug] tp-4 start timeout

github.com/sgl-project/sglang - siddhatiwari opened this issue 4 months ago

Add MLA gsm8k eval

github.com/sgl-project/sglang - ispobock opened this pull request 4 months ago

chore: bump v0.3.1.post3

github.com/sgl-project/sglang - zhyncs opened this pull request 4 months ago

Fix triton head num

github.com/sgl-project/sglang - ispobock opened this pull request 4 months ago

fix incorrect links in documentation

github.com/sgl-project/sglang - rchen19 opened this pull request 4 months ago

[Feature, Hardware] Enable SGLang on XPU GPUs via PyTorch

github.com/sgl-project/sglang - liangan1 opened this pull request 4 months ago

[Bug] Deepseek-V2.5 capture cuda graph failed

github.com/sgl-project/sglang - halexan opened this issue 4 months ago

[Bug] The sglang cannot reach the preset concurrency level.

github.com/sgl-project/sglang - rangehow opened this issue 4 months ago

Add OLMoE

github.com/sgl-project/sglang - Muennighoff opened this pull request 4 months ago

minor: add quant eval compared with base

github.com/sgl-project/sglang - zhyncs opened this pull request 4 months ago

[Bug] The engine hangs after requesting health_generate 190 times.

github.com/sgl-project/sglang - unix1986 opened this issue 4 months ago

Fix env vars in bench_latency

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Performance] Add triton kernels for LoRA

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

Release v0.3.1.post2

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Fix padding in the cuda graph

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Bug] illegal memory access encountered

github.com/sgl-project/sglang - wonderisland opened this issue 4 months ago

[Bug] enable-mixed-chunk may cause the regex request get wrong result and output_token_logprobs

github.com/sgl-project/sglang - liuteng opened this issue 4 months ago

Debug schedule optimization

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

fix: creat new dict everytime for putting new frame

github.com/sgl-project/sglang - Luodian opened this pull request 4 months ago