Quantisation
A large model has billions of parameters, and each one originally takes up 16 bits. Quantisation shrinks them to 4 to 8 bits with only a small loss of quality. A model with 7 to 8 billion parameters then takes up around 5 GB and runs comfortably even on a laptop.
What you need
- A Mac with an M-series chip is surprisingly good for larger models thanks to its shared memory.
- A PC with a graphics card with plenty of VRAM is the fastest.
- Even integrated graphics (for example Intel Arc) can handle smaller models with the right software.
- More memory means a bigger model. With today's RAM prices that is expensive.
Why run it locally
Privacy, no monthly fees, and it works without the internet. It is handy for working with sensitive documents, for home automation or for experiments. The most powerful models, however, remain in the cloud.