SGLang issues | Ecosyste.ms: OpenCollective

[Bug] oom,torch.OutOfMemoryError: seems to only use one gpu on A800-80G,available 40g on each card

github.com/sgl-project/sglang - chuangzhidan opened this issue 4 months ago

[WIP] Prometheus Metrics

github.com/sgl-project/sglang - binarycrayon opened this pull request 4 months ago

[Question]Why is the default value of max_prefill_tokens 16384?

github.com/sgl-project/sglang - wjj19950828 opened this issue 4 months ago

Support double sparsity

github.com/sgl-project/sglang - andy-yang-1 opened this pull request 4 months ago

[Event] Add public meeting invite to README

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

Fuse top_k and top_k in the sampler

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Pr fix max workers

github.com/sgl-project/sglang - wellhowtosay opened this pull request 4 months ago

[Bug] OOM when runing `bench_serving` with DeepSeekCoder-V2-Lite.

github.com/sgl-project/sglang - zh-zheng opened this issue 4 months ago

Fix oom issues with fp8 for llama

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Bugfix] Enable SGLang on AMD GPUs via PyTorch for ROCm (#1419)

github.com/sgl-project/sglang - HaiShaw opened this pull request 4 months ago

Add bench_server_latency.py

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Fix schedule bug

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

fix schedule bug

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

Fixed n>1 causing list index out of range with VLM

github.com/sgl-project/sglang - jasonyux opened this pull request 4 months ago

Fix attention backend

github.com/sgl-project/sglang - ispobock opened this pull request 4 months ago

Enable MLA by default

github.com/sgl-project/sglang - ispobock opened this pull request 4 months ago

[Bug] Performance issue on MoE with torch.compile

github.com/sgl-project/sglang - ispobock opened this issue 4 months ago

Release 0.3.1.post1

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Add OLMoE model

github.com/sgl-project/sglang - janimo opened this pull request 4 months ago

[Bug] The latest Sglang docker image cannot start online services

github.com/sgl-project/sglang - CedricHwong opened this issue 4 months ago

Fix torch compile for deepseek-v2

github.com/sgl-project/sglang - ispobock opened this pull request 4 months ago

Simplify sampler and its error handling

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Clean up model loader

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Bug] Llama 405B FP8 causes OOM on 16xA40

github.com/sgl-project/sglang - sumukshashidhar opened this issue 4 months ago

Add constrained_json_whitespace_pattern to ServerArgs

github.com/sgl-project/sglang - zifeitong opened this pull request 4 months ago

[Feature] Add initial support for sequence parallelism

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

[Feature] Expert parallelism support

github.com/sgl-project/sglang - chongli-uw opened this issue 4 months ago

[Bug] Nonsense and slow output under high concurrency

github.com/sgl-project/sglang - tongyx361 opened this issue 4 months ago

[Feature] Support LoRA path renaming and add LoRA serving benchmarks

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

Revert "[Minor] Raise exception for wrong import (#1409)"

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

Remove deprecated configs

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Release v0.3.1

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Update backend.md

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Fix] Fix logprob and normalized_logprob

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Add libibverbs-dev to Dockerfile

github.com/sgl-project/sglang - Aphoh opened this pull request 4 months ago

fix: resolve nightly eval

github.com/sgl-project/sglang - zhyncs opened this pull request 4 months ago

Add pytorch sampling backend ut

github.com/sgl-project/sglang - ispobock opened this pull request 4 months ago

[Bug] missing max_workers param when initiate ProcessPoolExecutor

github.com/sgl-project/sglang - wellhowtosay opened this issue 4 months ago

[Bug] MLA models can't use enable-torch-compile. Can be fix by suppressing errors.

github.com/sgl-project/sglang - Achazwl opened this issue 4 months ago

Enable torch.compile for triton backend

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Bug] deepseek-v2 fp8 cuda graph errror

github.com/sgl-project/sglang - fengyang95 opened this issue 4 months ago

[Feature, Hardware] Enable SGLang on AMD GPUs via PyTorch for ROCm

github.com/sgl-project/sglang - HaiShaw opened this pull request 4 months ago

[Feature] Support AMD GPU via PyTorch for ROCm

github.com/sgl-project/sglang - HaiShaw opened this issue 4 months ago

Add torchao quant for mixtral and qwen_moe

github.com/sgl-project/sglang - jerryzh168 opened this pull request 4 months ago

fallback to round robin scheduler

github.com/sgl-project/sglang - qeternity opened this pull request 4 months ago

[Bug] AttributeError: 'MiniCPM3ForCausalLM' object has no attribute 'get_module_name'

github.com/sgl-project/sglang - Lixtt opened this issue 4 months ago

[Bug] OpenAI batch API gets stuck

github.com/sgl-project/sglang - dmakhervaks opened this issue 4 months ago

ci: fix finish

github.com/sgl-project/sglang - zhyncs opened this pull request 4 months ago

[Bug] triton attention-backend bug

github.com/sgl-project/sglang - 81549361 opened this issue 4 months ago

Update pr-test.yml

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Balance test in CI

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Update pr-test.yml

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Minor] Raise exception for wrong import

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

[CI] Include triton backend and online serving benchmark into CI

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Make stop reason a dict instead of str

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Minor, CI] remove lora test from minimal suite

github.com/sgl-project/sglang - Ying1123 opened this pull request 4 months ago

[Bug] RuntimeError: Failed to allocate memory for batch_prefill_tmp_v with size 458752000 and alignment 16 in AlignedAllocator

github.com/sgl-project/sglang - josephydu opened this issue 4 months ago

[Bug] ImportError : cannot import name 'gemma_fused_add_rmsnorm' from 'flashinfer.norm'

github.com/sgl-project/sglang - luo647 opened this issue 4 months ago

kernel: use tensor cores for flashinfer gqa kernels

github.com/sgl-project/sglang - yzh119 opened this pull request 4 months ago

[Minor Fix] Fix llava modalities issue for single-image

github.com/sgl-project/sglang - kcz358 opened this pull request 4 months ago

Support cuda graph in the triton attention backend

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Bug] LLaVA performance inconsistent with the result

github.com/sgl-project/sglang - kcz358 opened this issue 4 months ago

Fix README format

github.com/sgl-project/sglang - Achazwl opened this pull request 4 months ago

[Bug] This modeling file requires the following packages that were not found in your environment: datamodel_code_generator. Run `pip install datamodel_code_generator`

github.com/sgl-project/sglang - cicicji opened this issue 4 months ago

Add Support for XVERSE Models (Dense and MoE) to sglang

github.com/sgl-project/sglang - hxer7963 opened this pull request 4 months ago

[Feature] support awq of deepseek-v2 or deepseek-v2.5

github.com/sgl-project/sglang - tutu329 opened this issue 4 months ago

[Feature] need DeepSeek-v2 or deepseek-v2.5 awq support

github.com/sgl-project/sglang - tutu329 opened this issue 4 months ago

Remove synchronization in cuda graph replay

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

Add no commit to main rule

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

Optimize conflicts between CUDA graph and vocab mask tensors

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

[Bug] 'LlamaTokenizerFast' object has no attribute 'tokenizer'

github.com/sgl-project/sglang - zwc163 opened this issue 4 months ago

Improve error reporting during server launch

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Fix] Fix --disable-flashinfer

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Feature] Support torch profiler

github.com/sgl-project/sglang - danielhua23 opened this issue 4 months ago

[Feature] Can centos7 use this project?

github.com/sgl-project/sglang - luo647 opened this issue 4 months ago

[Bug] requests.exceptions.JSONDecodeError:

github.com/sgl-project/sglang - eyuansu62 opened this issue 4 months ago

remove assertion in triton attention and add an unit test

github.com/sgl-project/sglang - ByronHsu opened this pull request 4 months ago

[Feature] Support RM API

github.com/sgl-project/sglang - UbeCc opened this issue 4 months ago

Rewrite mixed chunked prefill

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

[Bug] too many processes

github.com/sgl-project/sglang - wellhowtosay opened this issue 4 months ago

Refactor attention backend

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Deprecate --disable-flashinfer and introduce --attention-backend

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Minor] move triton attention kernels into a separate folder

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

Organize flashinfer indices update

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

[Do not merge] Test torchao

github.com/sgl-project/sglang - jerryzh168 opened this pull request 4 months ago

Fix vocab mask update bug

github.com/sgl-project/sglang - hnyls2002 opened this pull request 4 months ago

[Minor] improve kill scripts and torchao import

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

[Feature] 4-bit quantized prefix cache

github.com/sgl-project/sglang - josephrocca opened this issue 4 months ago

Fix CORS compatibility with OpenAI, vLLM, TGI, LMDeploy

github.com/sgl-project/sglang - josephrocca opened this pull request 4 months ago

deepseek-v2 torch.compile error

github.com/sgl-project/sglang - cdj0311 opened this issue 4 months ago

Support MiniCPM3

github.com/sgl-project/sglang - Achazwl opened this pull request 4 months ago

fix bug of `undefined is_single` in meth `create_abort_task`

github.com/sgl-project/sglang - wcsjtu opened this pull request 4 months ago

deepseek-v2 enable-mla 4x slower

github.com/sgl-project/sglang - cdj0311 opened this issue 4 months ago

[Docs] Improve documentations

github.com/sgl-project/sglang - merrymercy opened this pull request 4 months ago

BaiChuan2 Model

github.com/sgl-project/sglang - blacker521 opened this pull request 4 months ago

SGLang Discussion WeChat Group

github.com/sgl-project/sglang - qingkelab opened this issue 4 months ago

[Bug] Unable to see logprobs for prompt/input

github.com/sgl-project/sglang - dmakhervaks opened this issue 4 months ago

[Bug] Mixed chunked prefill is not compatible with vocab tensor mask

github.com/sgl-project/sglang - hnyls2002 opened this issue 4 months ago

Support OpenAI API json_schema response format

github.com/sgl-project/sglang - zifeitong opened this pull request 4 months ago

[Bug] sgLang v0.3 breaks TP8 Llama 3.1 405B FP8 on 8xH100

github.com/sgl-project/sglang - jischein opened this issue 4 months ago