Technology

87515 readers

2766 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related news or articles.
Be excellent to each other!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
Check for duplicates before posting, duplicates may be removed
Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 3 years ago

MODERATORS

L3s@lemmy.world

enu@lemmy.world

technopagan@lemmy.world

L4s@lemmy.world

L3s@hackingne.ws

Bytedance is training an AI model with up to ten trillion parameters: That's three times the size of Moonshot's Kimi K3, currently the largest Chinese model. (the-decoder.com)

submitted 2 weeks ago by eicker@lemmy.world to c/technology@lemmy.world

6 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] Armand1@lemmy.world 7 points 2 weeks ago (1 children)

Having run models locally, RAM use seems to be almost directly proportional to number of parameters. 8 Billion parameters requires approx 8GB of VRAM at 1/4 precision.

Therefore, if this pattern holds you somehow need 10 Terabytes of VRAM at 4K and 40 Terabytes at full precision.

I think I saw some estimates that Claude's Opus models may be and Opus model equivalents may be at around 100B parameters (100-400GB VRAM).

TLDR its clear why RAM is so expensive.

[–] eager_eagle@lemmy.world 3 points 2 weeks ago

for inference you're only counting active parameters towards VRAM, and some labs / models don't train at 32b precision, or even use the same precision for different parts of the network