SGLang issues | Ecosyste.ms: OpenCollective

[Performance]: Process affinity to CPU cores with multiple sockets support

github.com/sgl-project/sglang - HaiShaw opened this pull request about 1 month ago

Replace prob based with threshold based load balancing

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

Allow overwrite flashinfer use_tensorcore

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

[Feature] How to accelerate constrained decoding when regex needs to change with input?

github.com/sgl-project/sglang - GrittyChen opened this issue about 1 month ago

[Fused moe] add tuning fused configs for qwen2 57b and mixtral 8x7b

github.com/sgl-project/sglang - BBuf opened this pull request about 1 month ago

[Bug] cannot import name 'CachedGrammarCompiler' from 'xgrammar' (version 0.3.6)

github.com/sgl-project/sglang - Quang-elec44 opened this issue about 1 month ago

test select concurrency

github.com/sgl-project/sglang - qeternity opened this pull request about 1 month ago

Fix docs

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

Rename triton_fused_moe -> fused_moe_triton

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

Balance CI tests

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

fix: use torch.sum for compatible

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

[Bug] FusedMoE compatible with vllm 0.6.3.post1

github.com/sgl-project/sglang - zhyncs opened this issue about 1 month ago

Update CI threshold & Improve code style

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

Fix mixed chunked prefill in overlap mode

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

fix: resolve end-of-file-fixer

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

feat: update other MoE models deps

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

feat: update gitignore and add tuning config for FusedMoE

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

Simplify `Scheduler.update_running_batch`

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

feat: remove the dependency on FusedMoE

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

Merged three native APIs into one: get_server_info

github.com/sgl-project/sglang - henryhmko opened this pull request about 1 month ago

[Bug] llava use image hash as token，leading to cache bug

github.com/sgl-project/sglang - zwc163 opened this issue about 1 month ago

Speculative EAGLE2

github.com/sgl-project/sglang - yukavio opened this pull request about 1 month ago

Byhsu/fairness router

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

Improve sglang router

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

add prefix match for certain tenant

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

Add more api routes (completion, health, etc) to the router

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

[Draft] Resolving integration differences after XGrammar lauch refactoring

github.com/sgl-project/sglang - gittb opened this pull request about 1 month ago

fix dp_rank env

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

update router doc

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

Bump sglang-router to 0.0.5

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

[Bug] Error when using LLAVA 1.5 for llava bench

github.com/sgl-project/sglang - pspdada opened this issue about 1 month ago

fix: resolve bench_serving args

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

Fix dp print message

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

[CI] Fix test cases

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

Add concurrency option for benchmark

github.com/sgl-project/sglang - cermeng opened this pull request about 1 month ago

Add concurrency option in benchmark

github.com/sgl-project/sglang - cermeng opened this pull request about 1 month ago

Fix grid size in Triton decoding kernel

github.com/sgl-project/sglang - ispobock opened this pull request about 1 month ago

[Bug] Error when launching llava1.5

github.com/sgl-project/sglang - pspdada opened this issue about 1 month ago

deps(flashinfer): fix `is_flashinfer_available()` and make `flashinfer` optional dependency

github.com/sgl-project/sglang - XuehaiPan opened this pull request about 1 month ago

[Feature] Support LLaMA-3.2 finetuned with Sentence Transformers !

github.com/sgl-project/sglang - thusinh1969 opened this issue about 1 month ago

Revert "Only stream output on tp rank 0"

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

EAGLE2: general part [2]

github.com/sgl-project/sglang - yukavio opened this pull request about 1 month ago

EAGLE2: Eagle related part [1]

github.com/sgl-project/sglang - yukavio opened this pull request about 1 month ago

feat(pre-commit): trim unnecessary notebook metadata from git history

github.com/sgl-project/sglang - XuehaiPan opened this pull request about 1 month ago

fix: add xgrammar dependency

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

minor: update gsm8k threshold

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

Only stream output on tp rank 0

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

add profile in offline benchmark & update doc

github.com/sgl-project/sglang - bjmsong opened this pull request about 1 month ago

[minor] Clean up unused imports

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

Add initial support for intel Gaudi accelerators

github.com/sgl-project/sglang - ankurneog opened this pull request about 1 month ago

chore: bump v0.3.6

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

Online weight update [WIP]

github.com/sgl-project/sglang - zhaochenyang20 opened this pull request about 1 month ago

Rename sglang.bench_latency to sglang.bench_one_batch

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

[Bug] Unable to load GPTQ Mixtral 8x7 v0.1 with SGLang

github.com/sgl-project/sglang - DhruvaBansal00 opened this issue about 1 month ago

Turn off autotune for scaled mm for fp8 dynamic quant in torchao

github.com/sgl-project/sglang - jerryzh168 opened this pull request about 1 month ago

[router] add base_gpu_id server args & merged radix tree python reference

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

[router] cache-aware load-balancing router v1

github.com/sgl-project/sglang - ByronHsu opened this pull request about 1 month ago

[Feature] Inference example code for Qwen2-VL

github.com/sgl-project/sglang - YuanLiuuuuuu opened this issue about 1 month ago

[Bug] Qwen2-VL-7B with sglang Performance Degradation on MME benchmark

github.com/sgl-project/sglang - Mr-Loevan opened this issue about 1 month ago

ROCm: Fix MoE padding for none FP8 cases

github.com/sgl-project/sglang - HaiShaw opened this pull request about 1 month ago

Benchmark with Pytorch Profiler easily

github.com/sgl-project/sglang - bjmsong opened this pull request about 1 month ago

[Feature] Support for rerank models

github.com/sgl-project/sglang - dinhanhx opened this issue about 1 month ago

[Feature] Is Yarn supported in sglang?

github.com/sgl-project/sglang - klykq111 opened this issue about 1 month ago

Error out when torchao-config option is not recognized

github.com/sgl-project/sglang - jerryzh168 opened this pull request about 1 month ago

Fix #2037 - Context length check does not take into out pad tokens for visual models

github.com/sgl-project/sglang - jakep-allenai opened this pull request about 1 month ago

Enable overlap scheduler by default for the triton attention backend

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

Move test_session_id.py to playground

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

Allow skipping warmup in bench_offline_throughput.py

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

[Bug] RuntimeError: Failed to allocate memory for batch_prefill_tmp_v with size 435814400 and alignment 16 in AlignedAllocator

github.com/sgl-project/sglang - yuki252111 opened this issue about 1 month ago

feat: use cascade attention kernel (single level)

github.com/sgl-project/sglang - james-p-xu opened this pull request about 1 month ago

Update nightly-eval.yml

github.com/sgl-project/sglang - merrymercy opened this pull request about 1 month ago

[Bug] canot load Gemma2 awq

github.com/sgl-project/sglang - Foreist opened this issue about 1 month ago

[Bug] big TPOT and ITL when running the offline benchmark

github.com/sgl-project/sglang - TraceIvan opened this issue about 1 month ago

Use native fp8 format on MI300X

github.com/sgl-project/sglang - HaiShaw opened this pull request about 1 month ago

minor: add dataset dump and questions shuffle

github.com/sgl-project/sglang - zhyncs opened this pull request about 1 month ago

Expose max total num tokens from Runtime & Engine API

github.com/sgl-project/sglang - henryhmko opened this pull request about 1 month ago

minor: update gsm8k eval

github.com/sgl-project/sglang - zhyncs opened this pull request about 2 months ago

[Bug] disk cache io error when simultaneously loading lots of sglang offline engine

github.com/sgl-project/sglang - LeeSureman opened this issue about 2 months ago

Use cuda event wait and synchronization instead of busy waiting

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix: incorrect top_logprobs in chat completion

github.com/sgl-project/sglang - ajwaitz opened this pull request about 2 months ago

[Feature, Performance] kv cache performance improvement

github.com/sgl-project/sglang - HaiShaw opened this issue about 2 months ago

Simplify logits penalizer

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Allow passing extra request body to bench_offline_throughput.py

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[Bug] Qwen-2.5-Math-7B-Instruct and Llama-3.1-8B-Instruct Produce Nonsensical Results

github.com/sgl-project/sglang - Broyojo opened this issue about 2 months ago

Fix chunked prefill with output logprob

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

feat(srt): support prefill and generate with `input_embeds`

github.com/sgl-project/sglang - XuehaiPan opened this pull request about 2 months ago

Add simple CPU offloading support.

github.com/sgl-project/sglang - janimo opened this pull request about 2 months ago

[Feature] TorchAO support for Qwen 32B

github.com/sgl-project/sglang - grahama1970 opened this issue about 2 months ago

Rename layer_idx to layer_id for consistency

github.com/sgl-project/sglang - janimo opened this pull request about 2 months ago

docs: fix module docstrings and copyright headers

github.com/sgl-project/sglang - XuehaiPan opened this pull request about 2 months ago

[Performance] why so many bubbles between steps when running llava-one-vision?

github.com/sgl-project/sglang - sleepwalker2017 opened this issue about 2 months ago

support set role as 'tool'

github.com/sgl-project/sglang - yukavio opened this pull request about 2 months ago

Simplify flashinfer indices update for prefill

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

[feat] Add session control

github.com/sgl-project/sglang - Ying1123 opened this pull request about 2 months ago

Crash the CI jobs on model import errors

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Tune the threshold for accuracy tests in CI

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Fix cuda illegal memory access in overlap mode

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

feat: update torch 2.5.1

github.com/sgl-project/sglang - zhyncs opened this pull request about 2 months ago

[Minor] Fix styles for overlap mode

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago

Enable overlap by default

github.com/sgl-project/sglang - merrymercy opened this pull request about 2 months ago