SGLang issues | Ecosyste.ms: OpenCollective

[Bug] canot load Gemma2 awq

github.com/sgl-project/sglang - Foreist opened this issue about 2 months ago

[Bug] big TPOT and ITL when running the offline benchmark

github.com/sgl-project/sglang - TraceIvan opened this issue about 2 months ago

Use native fp8 format on MI300X

github.com/sgl-project/sglang - HaiShaw opened this pull request about 2 months ago

minor: add dataset dump and questions shuffle

github.com/sgl-project/sglang - zhyncs opened this pull request about 2 months ago

Expose max total num tokens from Runtime & Engine API

github.com/sgl-project/sglang - henryhmko opened this pull request about 2 months ago

minor: update gsm8k eval

github.com/sgl-project/sglang - zhyncs opened this pull request about 2 months ago

[Bug] disk cache io error when simultaneously loading lots of sglang offline engine

github.com/sgl-project/sglang - LeeSureman opened this issue about 2 months ago

Use cuda event wait and synchronization instead of busy waiting

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix: incorrect top_logprobs in chat completion

github.com/sgl-project/sglang - ajwaitz opened this pull request about 2 months ago

[Feature, Performance] kv cache performance improvement

github.com/sgl-project/sglang - HaiShaw opened this issue about 2 months ago

Simplify logits penalizer

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Allow passing extra request body to bench_offline_throughput.py

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[Bug] Qwen-2.5-Math-7B-Instruct and Llama-3.1-8B-Instruct Produce Nonsensical Results

github.com/sgl-project/sglang - Broyojo opened this issue about 2 months ago

Fix chunked prefill with output logprob

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

feat(srt): support prefill and generate with `input_embeds`

github.com/sgl-project/sglang - XuehaiPan opened this pull request about 2 months ago

Add simple CPU offloading support.

github.com/sgl-project/sglang - janimo opened this pull request about 2 months ago

[Feature] TorchAO support for Qwen 32B

github.com/sgl-project/sglang - grahama1970 opened this issue about 2 months ago

Rename layer_idx to layer_id for consistency

github.com/sgl-project/sglang - janimo opened this pull request about 2 months ago

docs: fix module docstrings and copyright headers

github.com/sgl-project/sglang - XuehaiPan opened this pull request about 2 months ago

[Performance] why so many bubbles between steps when running llava-one-vision?

github.com/sgl-project/sglang - sleepwalker2017 opened this issue about 2 months ago

support set role as 'tool'

github.com/sgl-project/sglang - yukavio opened this pull request about 2 months ago

Simplify flashinfer indices update for prefill

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[feat] Add session control

github.com/sgl-project/sglang - Ying1123 opened this pull request about 2 months ago

Crash the CI jobs on model import errors

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Tune the threshold for accuracy tests in CI

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix cuda illegal memory access in overlap mode

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

feat: update torch 2.5.1

github.com/sgl-project/sglang - zhyncs opened this pull request about 2 months ago

[Minor] Fix styles for overlap mode

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Enable overlap by default

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Rename arguments `--disable-nan-detection` to `--enable-nan-detection`

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Deprecate --disable-flashinfer and --disable-flashinfer-sampling

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Remove monkey_patch_vllm_dummy_weight_loader

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Revert "chore: update torch v2.5.1"

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

add phi-3 small support

github.com/sgl-project/sglang - Tushar-ml opened this pull request about 2 months ago

Support cuda graph for DP attention

github.com/sgl-project/sglang - ispobock opened this pull request about 2 months ago

[Feature] Add Dockerfile.dev for development purposes

github.com/sgl-project/sglang - zhyncs opened this issue about 2 months ago

[Bug] torch 2.5.1 upgrade performance issue

github.com/sgl-project/sglang - zhyncs opened this issue about 2 months ago

Add log input text when using openai chat api

github.com/sgl-project/sglang - ccjincong opened this pull request about 2 months ago

[Bug] lmms-lab/llava-onevision-qwen2-7b-ov-chat. Missing file dependencies, no preprocessor_config.json processor_config.json

github.com/sgl-project/sglang - zhangucan opened this issue about 2 months ago

[Performance] Update xgrammar-related constrained decoding

github.com/sgl-project/sglang - DarkSharpness opened this pull request about 2 months ago

Add support for Qwen2-VL-based embedding models

github.com/sgl-project/sglang - james-p-xu opened this pull request about 2 months ago

[TEST] flashinfer version upgrade to v0.2.0

github.com/sgl-project/sglang - james-p-xu opened this pull request about 2 months ago

Launch dp ranks in parallel

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Input_embeds support

github.com/sgl-project/sglang - RinRin-32 opened this pull request about 2 months ago

Fix illegal memory access in overlap mode & Use more fused triton kernels for building meta data

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix weight update for data parallelism

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Add get_amdgpu_memory_capacity()

github.com/sgl-project/sglang - HaiShaw opened this pull request about 2 months ago

Fix core (MI300X) with --enable-overlap

github.com/sgl-project/sglang - HaiShaw opened this pull request about 2 months ago

fix a small typo in docs

github.com/sgl-project/sglang - BBuf opened this pull request about 2 months ago

Release v0.3.5.post2

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[BUG] Problems with jump forward decoding

github.com/sgl-project/sglang - merrymercy opened this issue about 2 months ago

[Fix] Adjust default chunked prefill size and cuda graph max bs according to GPU memory capacity

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix json benchmark

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix the default arguments of bench_offline_throughput.py & simplify detokenizer manager

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Add support for GPT-J

github.com/sgl-project/sglang - danilotpnta opened this pull request about 2 months ago

[Bug] torch compile warning

github.com/sgl-project/sglang - fengyang95 opened this issue about 2 months ago

Expose no_stop_trim and skip_special_tokens in openai api

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

fix: align enable_overlap_scheduler naming between code and docs

github.com/sgl-project/sglang - w1ndseeker opened this pull request about 2 months ago

[Bug] Qwen2VL Crashes on some inputs

github.com/sgl-project/sglang - jakep-allenai opened this issue about 2 months ago

Fix outlines version

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

regex stopping condition

github.com/sgl-project/sglang - jancervenka opened this pull request about 2 months ago

Fix unit tests

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix torch.compile for MoE

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[Feature] Support Qwen2-VL based embedding model

github.com/sgl-project/sglang - VoVAllen opened this issue about 2 months ago

Github runner instructions for AMD

github.com/sgl-project/sglang - HaiShaw opened this pull request about 2 months ago

benchmark json schema

github.com/sgl-project/sglang - DarkSharpness opened this pull request about 2 months ago

[Bug] Does Mixtral currently not support torch compile?

github.com/sgl-project/sglang - sitabulaixizawaluduo opened this issue about 2 months ago

chore: open lto and optimization in release profile

github.com/sgl-project/sglang - ethe opened this pull request about 2 months ago

Add download_dir ServerArgs property

github.com/sgl-project/sglang - pjyi2147 opened this pull request about 2 months ago

set content to empty string

github.com/sgl-project/sglang - chottolabs opened this pull request about 2 months ago

[BUG] Jump forward w/ outlines backend slightly changes the decoding results

github.com/sgl-project/sglang - merrymercy opened this issue about 2 months ago

Fix dependency and error message for xgrammar

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Do not let invalid grammar crash the server

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Release v0.3.5.post1

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[Bug] Have any suggestions for setting hyperparameters for inference acceleration？

github.com/sgl-project/sglang - 948024326 opened this issue about 2 months ago

Fix grammar backend for tensor parallelism

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[WIP] [Router] Multi Tenant Radix Tree

github.com/sgl-project/sglang - ByronHsu opened this pull request about 2 months ago

Refactor grammar backend

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[BUG] xgrammar does not follow the constraint

github.com/sgl-project/sglang - merrymercy opened this issue about 2 months ago

[WIP] Use FlashInfer RoPE

github.com/sgl-project/sglang - james-p-xu opened this pull request about 2 months ago

fix test_embedding_models prompt length too long's bug

github.com/sgl-project/sglang - BBuf opened this pull request about 2 months ago

fix a bug in v1_embeeding_request

github.com/sgl-project/sglang - BBuf opened this pull request about 2 months ago

Fix finish reason

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

poke test

github.com/sgl-project/sglang - ByronHsu opened this pull request about 2 months ago

Filter empty prompt in random bench serving

github.com/sgl-project/sglang - ispobock opened this pull request about 2 months ago

cleanup rust folder

github.com/sgl-project/sglang - ByronHsu opened this pull request about 2 months ago

Fix weight loading for tied word embedding when TP > 1

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix a typo in io_struct.py

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[Feature] Regex stop condition

github.com/sgl-project/sglang - SinanAkkoyun opened this issue about 2 months ago

[Minor] Remove unused imports

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

fix sglang_router not found

github.com/sgl-project/sglang - ByronHsu opened this pull request about 2 months ago

Bump router to 0.0.3

github.com/sgl-project/sglang - ByronHsu opened this pull request about 2 months ago

run rust test on ubuntu instead of 1-gpu-runner

github.com/sgl-project/sglang - ByronHsu opened this pull request about 2 months ago

release router from py38 to py312

github.com/sgl-project/sglang - ByronHsu opened this pull request about 2 months ago

Fix rust unit test and pypi token

github.com/sgl-project/sglang - ByronHsu opened this pull request about 2 months ago

Add Engine::encode example

github.com/sgl-project/sglang - james-p-xu opened this pull request about 2 months ago

support echo=true and logprobs in openai api when logprobs=1 in lm-evaluation-harness

github.com/sgl-project/sglang - BBuf opened this pull request about 2 months ago

support parallel grammar preprocessing

github.com/sgl-project/sglang - DarkSharpness opened this pull request about 2 months ago

support internlm2-reward

github.com/sgl-project/sglang - RangiLyu opened this pull request about 2 months ago

[Feature] Does sglang support only input embeds?

github.com/sgl-project/sglang - OswaldoBornemann opened this issue about 2 months ago