Understanding NVIDIA GPU generations Ampere's A100, Hopper's H100 and H200, and now Blackwell's B200 and GB200 is essential before you commit budget to AI training or inference hardware. Each generation didn't just get "faster"; it targeted a different bottleneck, moving from mixed-precision training and transformer acceleration toward larger memory, lower-precision inference, and faster multi-GPU communication. This guide walks through what changed at each step and which generation actually fits your workload, whether you're buying outright or looking to on demand.
NVIDIA GPU Generations at a Glance
| Generation | Representative GPU | Main Advance | Typical Strength |
|---|---|---|---|
| Ampere | A100 | TF32, BF16, sparsity, MIG | General AI training and HPC |
| Hopper | H100 | FP8 and Transformer Engine | LLM training and inference |
| Hopper Refresh | H200 | Larger, faster memory | Large-model inference |
| Blackwell | B200 / GB200 | FP4, dual-die design, NVLink 5 | Large-scale AI, trillion-parameter models |
The naming can be confusing: the letter marks the architecture (A = Ampere, H = Hopper, B = Blackwell), but the individual GPU, server, and full CPU-GPU platform can all carry different names. The B200 is a standalone Blackwell GPU, GB200 pairs Grace CPUs with Blackwell GPUs, and GB200 NVL72 is a complete rack-scale system memory, power, and NVLink specs shift depending on which one you mean.
NVIDIA A100: The GPU That Made Mixed-Precision AI Practical
Built on Ampere, the A100 became a defining accelerator of the early large-scale deep-learning era not just for raw performance, but for making mixed-precision computing easy to adopt without major code changes.
Key A100 capabilities:
. Third-generation Tensor Cores
. TF32 for FP32-style workloads at Tensor Core speed
. FP16, BF16, and INT8 support
. Structured sparsity
. Up to 80 GB of HBM2e memory
. Multi-Instance GPU (MIG) for multi-tenant use
. Up to 600 GB/s NVLink bandwidth
TF32 let existing FP32 training workloads gain Tensor Core acceleration with minimal rework, while BF16/FP16 cut memory use and lifted throughput further. This made the A100 a strong fit for deep-learning training, scientific computing, HPC, LLM fine-tuning, and multi-tenant cloud setups. Its main limitation today is memory capacity and bandwidth that now trail Hopper and Blackwell but it remains a cost-effective, well-supported option for workloads that don't need the newest generation.
NVIDIA H100: Hopper Brings Dedicated Transformer Acceleration
The H100 marked a real architectural shift toward hardware built specifically for transformer-based AI, adding fourth-generation Tensor Cores and NVIDIA's Transformer Engine to dynamically select numerical precision during computation.
Representative H100 specs:
. 80 GB HBM3 memory (SXM)
. 3.35 TB/s memory bandwidth
. Up to 3,958 TFLOPS FP8 Tensor Core performance (SXM)
. Up to 900 GB/s fourth-generation NVLink bandwidth
. Up to 7 MIG instances
Where the A100 made mixed-precision AI practical, the H100 made transformer-specific acceleration central to the architecture with FP8 support driving meaningful gains in LLM training, high-throughput inference, generative AI, and distributed training. FP8 boosts throughput and cuts data movement, though preserving accuracy still requires proper scaling and mixed-precision handling.
NVIDIA H200: More Memory for Larger Models
The H200 isn't a new architecture it's still Hopper but it substantially upgrades the memory subsystem: 141 GB of HBM3e and roughly 4.8 TB/s of bandwidth, on top of the H100's existing Tensor Core and Transformer Engine capabilities.
Memory, not just compute, is often the real constraint in modern AI workloads model weights, activations, KV cache, and runtime overhead all compete for VRAM, especially during LLM inference. A model that barely fits an 80 GB H100 has considerably more headroom on a 141 GB H200, supporting larger batches and longer context windows without adding GPUs (exact fit still depends on precision, KV-cache size, and framework overhead).
H200 is particularly well suited to:
. Large language-model inference
. Long-context workloads
. Large-batch inference
. Retrieval-augmented generation
. Models that don't fit comfortably on 80 GB GPUs
NVIDIA Blackwell: B200 and GB200
Blackwell is NVIDIA's generation after Hopper, built around the demands of large-scale generative AI emphasizing lower-precision inference, faster GPU-to-GPU communication, and scaling across very large systems rather than compute alone.
Key Blackwell technologies:
. Second-generation Transformer Engine
. FP4/NVFP4 support for efficient inference
. Fifth-generation NVLink
. Multi-die GPU architecture
. 180 GB HBM3e per B200 GPU (HGX-class)
. 1.8 TB/s GPU-to-GPU NVLink bandwidth (HGX B200)
Blackwell's dual-die design connects two dies via a high-speed chip-to-chip link that functions as one unified GPU from a software standpoint.
B200 vs GB200 what's the difference?
| Product | What It Is |
|---|---|
| B200 | Blackwell GPU accelerator |
| GB200 | Grace CPU + two B200 GPUs |
| DGX B200 | Eight-B200 GPU server |
| GB200 NVL72 | Rack-scale system: 72 Blackwell GPUs + 36 Grace CPUs |
GB200 NVL72 is significant because it shifts the conversation from individual GPUs to a single, tightly interconnected 72-GPU AI system built for extremely large mixture-of-experts and trillion-parameter models.
The Evolution of AI Precision: FP32 to FP4
Precision support is one of the clearest threads connecting these generations: FP32 → FP16/BF16 → FP8 → FP4.
A100 brought TF32, FP16, BF16, and INT8 into a widely adopted platform. Hopper (H100/H200) added dedicated FP8 via the Transformer Engine. Blackwell extends this further with FP4/NVFP4-oriented acceleration. Each step is an expansion of the precision toolbox rather than a replacement lower precision moves and stores fewer bits per operation, raising throughput and cutting memory use, but it also raises the bar on maintaining accuracy through dynamic scaling, higher-precision accumulation, and quantization-aware techniques.
This is why picking a GPU on advertised FLOPS alone can mislead the workload's supported precision, memory bandwidth, and communication needs matter just as much, and peak numbers are sometimes quoted for sparse rather than dense workloads.
GPU Memory and Bandwidth Across Generations
Two separate things matter here: memory capacity (how much fits on the GPU) and memory bandwidth (how fast that data moves).
| GPU | Architecture | Memory | Memory Bandwidth | Key Precision |
|---|---|---|---|---|
| A100 | Ampere | Up to 80 GB HBM2e | Up to 2.0 TB/s | TF32, BF16, FP16 |
| H100 | Hopper | 80 GB HBM3 (SXM) | 3.35 TB/s | FP8, BF16, FP16 |
| H200 | Hopper | 141 GB HBM3e | 4.8 TB/s | FP8, BF16, FP16 |
| B200 | Blackwell | 180 GB HBM3e | Up to 8.0 TB/s | FP4/NVFP4, FP8 |
Specifications vary by form factor and system configuration; figures above reflect representative SXM/HGX-class configurations.
For LLM inference specifically, memory capacity can decide whether a model runs on one GPU, several GPUs, or a full multi-GPU cluster which is exactly why H200 and B200 deliver real-world gains even where arithmetic throughput isn't the only differentiator.
NVLink and the Shift to Multi-GPU AI
As models grow, GPU-to-GPU communication becomes as important as compute:
| Generation | Representative NVLink Bandwidth |
|---|---|
| A100 | Up to 600 GB/s |
| H100 / H200 | Up to 900 GB/s |
| B200 | ~1.8 TB/s per GPU (HGX) |
Faster interconnects reduce overhead when a model is split across GPUs. Blackwell pushes this furthest with systems like GB200 NVL72, which creates a 72-GPU NVLink domain especially valuable for mixture-of-experts models and other architectures with heavy GPU-to-GPU communication.
Which NVIDIA GPU Should You Choose?
| Workload | Sensible Choice |
|---|---|
| Existing infrastructure, broad compatibility | A100 |
| High-performance transformer training | H100 |
| Large-model inference, long context | H200 |
| High-end Blackwell throughput, FP4 inference | B200 |
| Rack-scale trillion-parameter AI | GB200 NVL72 |
These are guidelines, not rules. An A100 can still be the more cost-effective pick for smaller workloads, particularly when"GPU rental" pricing favors the older generation, and an H100 may make more sense than a newer GPU when your stack is already optimized for Hopper. The right choice ultimately comes down to workload, model size, precision, memory needs, software stack, scaling requirements, power envelope, and budget check current pricing "GPU cloud pricing in India" across generations before deciding.
The Main Takeaway
The path from A100 to H100, H200, and Blackwell isn't just a series of faster GPUs each generation solves a different bottleneck. A100 made mixed-precision AI broadly practical. H100 shifted focus to transformer workloads with FP8. H200 tackled growing memory demands with a major capacity and bandwidth bump. Blackwell pushes further into FP4 inference, multi-die GPUs, and rack-scale systems for the largest AI models.
Headline FLOPS shouldn't be the only factor in your decision memory capacity, bandwidth, precision support, interconnect speed, software compatibility, and total cost usually matter more. The right GPU isn't necessarily the newest one; it's the one that balances performance, memory, compatibility, and cost for your specific workload.
Frequently Asked Questions
Q1, What is the difference between A100, H100, and H200?
A100 (Ampere) established mixed-precision AI training at scale. H100 (Hopper) added FP8 and the Transformer Engine for transformer-specific acceleration. H200 keeps the same Hopper compute but roughly doubles memory to 141 GB with higher bandwidth, mainly benefiting large-model inference.
Q2. Is the H200 better than the H100 for LLM inference?
For memory-bound inference workloads long context, large batches, or models that barely fit an 80 GB GPU yes, the H200's 141 GB and higher bandwidth typically outperform the H100 in practice, even though the core compute architecture is the same.
Q3. What is the difference between B200 and GB200?
B200 is a standalone Blackwell GPU. GB200 pairs a Grace CPU with two B200 GPUs on one module. GB200 NVL72 is a full rack-scale system combining 72 Blackwell GPUs and 36 Grace CPUs into one large NVLink domain.
Q4. Do I need the newest NVIDIA GPU for AI training?
Not always. Smaller or memory-light workloads can run cost-effectively on an A100 or H100, especially when renting makes older generations significantly cheaper per hour. The newest GPU is worth it mainly for large models, long-context inference, or FP4-optimized workloads.
Q5. What is FP4 and why does Blackwell use it?
FP4 is a 4-bit floating-point format that reduces the data moved and stored per operation, increasing inference throughput and lowering memory use. Blackwell adds dedicated FP4/NVFP4 support through its Transformer Engine to make large-scale inference more efficient, with accuracy maintained through techniques like dynamic scaling.
