this post was submitted on 23 Aug 2026
76 points (81.7% liked)
Technology
87627 readers
3697 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
This is true. But VLLM's quantization support (as outlined in the article) is quite primitive.
Thats not what its built for. Its made for maximum speed with many concurrent users, not cramming LLMs into a desktop rig.
If you are running an LLM for personal use, you are usually better off with a backend optimized for higher fidelity quantization, like MLX DWQs, exl3, ik llama GGUFs and stuff like that.
If you want raw speed on a desktop, use one of the ninfer project branches.