You can, but it will be super slow. It would be more afforable to buy an Intel CPU and 64 GB RAM, given that the 64 GB version is 2700. Even a couple of used 3090s will be cheaper.
The top-end Jetson is essentially 1/5th of a 3090. But it is likely way more than 5x slower. It has 1/5 of the CUDA cores, 1/5 of the Tensor Cores, a bit more than 1/5 of bandwidth. It’s more realistic to say that it is 7-10x slower. It might be even worse given its measly ARM cpu might not be able to utilize that hardware as well as an x86 one could. Yet you can buy 3 used 3090s, maybe even 4 for the same price! The catch is that it’s not meant for DL. It’s an edge device. A fairly overpriced one, at that.
Oh and of course, all of these questions are kind of pointless outside of a scenario where cloud is not an option since pay-as-you-go A100 and H100 will be cheaper than any of these. Ignoring those is basically a luxury.
edge consumer hardware LLM options
hardware options
- gemma-2b-it-q4
- Turing Pi RK1 32gb (€300)
- Turing Pi 2.5 Cluster Board (€259)
- smaug-72b
- 2x 3090 (2x€660)
- 2x 4090 (2x€2060)
- or 2x7900xtx (2x €970