Gemma 2 makes hardware fit a central part of open-model selection

A second Gemma generation

Google released Gemma 2 on June 27, 2024, with 9B and 27B variants. The announcement emphasized a redesigned architecture, inference efficiency, and integration with common developer tools.

For teams exploring self-hosting, the release created another opportunity to compare capability with the practical limits of available hardware.

The smallest fitting model is not always the cheapest

A model that fits in memory can still be too slow at the concurrency a service needs. Conversely, a larger model may reduce retries or manual correction enough to justify its operating cost. Neither conclusion follows from parameter count alone.

Include input length and generation length in the test. A short demonstration prompt does not reveal how the service behaves when many users submit realistic documents at once.

A sensible local trial

Keep the runtime, quantization settings, and evaluation inputs fixed while comparing model sizes. Measure peak memory, time to first useful output, and accepted results per hour. Reserve some capacity for maintenance and traffic spikes rather than planning around a fully occupied accelerator. The resulting evidence will be more useful to a budget owner than an isolated benchmark score or a theoretical fit calculation.