Ask HN: Workstation/rig config for local LLM finetuning/training and inference?
I need help purchasing/putting together a rig that's powerful enough for training LLMs from scratch, finetuning models, and inferencing them.
Many people on this sub showcase their impressive GPU clusters, often usnig 3090/4090. But I need more than that—essentially the higher the VRAM, the better.
Here's some options that have been announced, please tell me your recommendation even if it's not one of these:
- Nvidia DGX Station
- Dell Pro Max with GB300 (Lenovo and HP offer similar products)
The above are not available yet, but it's okay, I'll need this rig by August.
Some people suggest AMD's MI300x or MI210. MI300x comes only in x8 boxes, otherwise it's an attractive offer!
Budget ≤ $30,000
Hey there!
How long do you plan to use this setup to reach your goals? I’m asking because this is precisely why I started Skorppio—we rent out high-performance workstations and servers for tasks like training AI models. It’s flexible, with no contracts, and can save you money compared to buying. Best of all its on-prem and you get secure 24/7 access to your data.
With a $30k budget, you’re likely looking at high-end consumer GPUs like the upcoming RTX 5090. But keep in mind, those kinds of GPUs can use a lot of power and generate a lot of heat.
For example, the RTX 5090 uses about 575 watts for 32GB of VRAM. The RTX 6000 Pro uses around 600 watts, but gives you 96GB of VRAM. So you get a lot more power and memory for about the same energy cost. Its not just about cost, its about the access to power. A typical household / office outlet will offer around 1575W @ 120v. With just a single GPU workstation you could come close to saturating that power. If you built a mGPU rig, like dual or quad 5090, you would be into 2-3 PSU's at 1000W each PSU. Do you have 3x 15Amp 120V circuits laying around? Most individuals dont, even most office environments (outside a data closet) dont. Beyond power availability, youre also going to run into major cooling issues. We have bespoke systems that use PCIE daughter boards to house 4x 5090's it also makes the rig 4x physically larger, but does help with the ability to keep temps at optimal levels.
At Skorppio, we already have RTX 6000 Pro / Max GPUs, NVIDIA Spark systems (both Founders and Dell), and dual or quad RTX 5090 Threadripper Pro workstations on the way.
About the DGX Station—while it sounds great, Dell told us it likely won’t be available until late this year. There’s no official launch date yet, so understand that's probably not even a candidate for you.
Hope this helps you figure out the best direction! Let me know if you want to dig deeper into any of this.