Cloud-hosted frontier models carry non-trivial privacy, latency, and subscription liabilities. With the release of Llama 3.2 (1B/3B), Microsoft's Phi-4 (14B), and Google's Gemma 2 (2B/9B), running high-capability reasoning models entirely on local developer workstations and edge devices is now an enterprise reality.
01 Precision vs Accuracy: The Quantization Frontier
Quantizing FP16 weights down to 4-bit and 8-bit integers is essential to fit models into workstation Unified Memory:
- Q4_K_M: Provides the ideal balance between memory footprint (approx. 0.65x model size in GB) and perplexity retention (>98% of FP16 benchmark scores);
- AWQ (Activation-aware Weight Quantization): Protects the critical 1% salient weights, delivering near-lossless reasoning for mathematical tasks.