Groq

Groq's pitch is a single number: tokens per second. By running open-weight models on custom LPU hardware rather than GPUs, Groq delivers inference fast enough to change what applications feel possible — real-time voice, instant agent loops, streaming that arrives faster than you can read.

What is Groq?

Groq builds Language Processing Units, inference-specific silicon, and sells access through GroqCloud. The paid API runs open-source models at roughly 280 to over 1,000 tokens per second, typically 10–20x cheaper than equivalent closed-model APIs. Artificial Analysis measurements from April 2026 put Qwen3 32B on Groq at 0.14ms time-to-first-token with 627 tokens per second of throughput. The catalogue is entirely open-source — there is no GPT, Claude or Gemini here, which is the trade you make for the speed and price. The corporate picture changed sharply in December 2025, when NVIDIA struck a $20 billion deal that brought founder Jonathan Ross, president Sunny Madra and around 90% of Groq's engineering staff to NVIDIA; the Groq 3 LPU was subsequently unveiled inside NVIDIA's Vera Rubin platform at GTC 2026.

Key Features

• LPU inference: Purpose-built hardware delivering 280–1,000+ tokens per second.
• Sub-millisecond latency: Time-to-first-token low enough for real-time voice and agent loops.
• Very low pricing: From $0.05 per million input tokens, with a 50% batch discount.
• Open-model catalogue: Qwen, Llama, DeepSeek, Gemma, Mixtral and other open-weight families.
• Free tier: Rate-limited but sufficient to evaluate latency against your own workload.

Who Should Use Groq?

Groq is the right choice when latency is the product: voice assistants, live translation, interactive agents, anything where a two-second pause breaks the experience. It is also compelling for cost-sensitive high-volume work, given the batch discount. The limits are worth stating plainly — you cannot run frontier closed models here, so if your application depends on GPT-5.6 or Claude Opus 5 quality, Groq is not a substitute. And after the NVIDIA deal absorbed most of the engineering team, it is reasonable to watch how independently the platform evolves.

Pricing

There is a free tier with rate limits. Paid usage is pay-as-you-go, starting around $0.05 per million input tokens on the smallest models and reaching roughly $3.00 per million on the largest, with a 50% discount for batch processing. That places Groq among the cheapest inference options anywhere, and the speed comes at no premium.

Pros and Cons

Bottom Line

If latency is what your application sells, Groq is still the fastest way to serve an open-weight model, and the pricing makes it hard to argue with for high-volume work. Use it for real-time and streaming experiences, and pair it with a frontier API for the harder reasoning your product also needs. Keep an eye on the roadmap given how much of the team NVIDIA absorbed.