How Much Does It Cost to Train an LLM? GPU-Hours and Dollars, Estimated
Every foundation-model training cost estimate — from a 300M-parameter experiment to the numbers circulated for GPT-4 — comes down to the same three inputs: how many floating-point operations the run needs, how fast the GPUs actually execute them, and what an hour of those GPUs costs. This page gives you the formula, works it through for the model sizes people most often ask about, and then lists everything the GPU-hour figure leaves out.
The GPU-hours formula
For a dense transformer, the compute needed to train is well approximated by:
where N is the parameter count and D is the number of training tokens. The 6 covers the forward pass (2 FLOPs per parameter per token) and the backward pass (about 4). To turn FLOPs into GPU-hours:
MFU (model FLOPs utilization) is the fraction of the chip's theoretical peak your training loop actually achieves. Well-tuned large runs reach 35–45%; a first attempt on a new stack often lands at 20–30%. Peak figures to plug in (dense, no sparsity):
| GPU | Peak BF16 (dense) | Effective at 40% MFU | Index median $/hr |
|---|---|---|---|
| NVIDIA V100 32GB | 125 TFLOPS (FP16) | 50 TFLOPS | legacy, ~$1.50 at the time of GPT-3 |
| NVIDIA A100 80GB | 312 TFLOPS | 125 TFLOPS | $2.46 |
| NVIDIA H100 80GB | 989 TFLOPS | 396 TFLOPS | $2.97 |
| NVIDIA H200 141GB | 989 TFLOPS | 396 TFLOPS | $4.95 |
| NVIDIA B200 192GB | ~2,250 TFLOPS | ~900 TFLOPS | $6.89 |
Then the bill is simply GPU-hours × $/GPU-hour. The examples below use the H100 at the index median of $2.97/hr unless stated otherwise; substitute the cheapest listing ($2.29) or a spot rate (~$1.00–1.80) to see the range.
Worked estimates by model size
| Model | Tokens | FLOPs (6ND) | H100-hours @ 40% MFU | Cost @ $2.97/hr | Cost @ $2.29/hr |
|---|---|---|---|---|---|
| 300M params (Chinchilla-optimal, 20 tokens/param) | 6B | 1.1 × 1019 | ≈ 8 | ≈ $25 | ≈ $18 |
| 1B params (20 tokens/param) | 20B | 1.2 × 1020 | ≈ 85 | ≈ $250 | ≈ $195 |
| 7B params (Llama-2-style) | 2T | 8.4 × 1022 | ≈ 59,000 | ≈ $175,000 | ≈ $135,000 |
| 70B params (Llama-3-style) | 15T | 6.3 × 1024 | ≈ 4.4 million | ≈ $13 million | ≈ $10 million |
| 175B params (GPT-3-style) | 300B | 3.2 × 1023 | ≈ 221,000 | ≈ $655,000 | ≈ $505,000 |
1B vs 300M: why "3× the parameters" is 10× the GPU-hours
Compute scales with N × D, and a compute-optimal token budget scales with N too, so training cost grows roughly with the square of model size when you also scale the data. The 1B model above needs about 11× the GPU-hours of the 300M model, not 3.3×. If you keep the dataset fixed instead, cost scales linearly with parameters.
7B on 2T tokens: the 300,000 GPU-hour question
The ~59,000 H100-hours in the table assumes 40% MFU. Meta reported about 184,000 A100-hours for Llama 2 7B, which the formula reproduces almost exactly at the A100's 312 TFLOPS peak (8.4 × 1022 ÷ (312 × 1012 × 0.4 × 3,600) ≈ 187,000). A quoted figure of 300,000 GPU-hours for a 7B pre-train therefore implies either A100s at a lower MFU, more tokens, or an all-in number that includes failed and ablation runs. At H100 rates, 300,000 GPU-hours costs roughly $690,000 at $2.29/hr, $890,000 at $2.97/hr, or $1.5 million on hyperscaler on-demand at $4.99/hr.
GPT-3 175B: the 2020 number and the same run today
GPT-3 was trained on roughly 300B tokens, which the formula puts at 3.2 × 1023 FLOPs. On V100s at a realistic 25–30% MFU that is 2.3–3.1 million V100-hours (the often-quoted "355 V100-years" is 3.1 million hours), and at the ~$1.50/hr cloud rate of the time gives the widely cited estimate of around $4.6 million. The same compute on H100s at 40% MFU is about 221,000 GPU-hours — roughly $650,000 at the index median, and under $300,000 on spot. That 10–15× fall in five years is the practical meaning of "compute is getting cheaper".
GPT-4 and the "25,000 GPUs" estimate
OpenAI has not published GPT-4's training configuration. The figure that circulates — around 25,000 A100s running for roughly 90–100 days — is an external estimate, not a confirmed number. Taken at face value it implies 55–60 million A100-hours; at $1.00–2.00 per A100-hour that is on the order of $60–120 million in GPU time alone, before failed runs, experiments and the engineering team. Treat any GPT-4 cost figure as a range built from those assumptions rather than a fact.
Fine-tuning is a different order of magnitude
Fine-tuning reuses a pre-trained model, so D is tiny by comparison — tens of millions to a few billion tokens rather than trillions. The cost is usually driven by memory (how many GPUs you need to hold the model) rather than by FLOPs.
- Full fine-tune of a 32B model on 100M tokens: 6 × 32 × 109 × 108 ≈ 1.9 × 1019 FLOPs — about 14 H100-hours of pure compute. But full fine-tuning with Adam needs roughly 16 bytes per parameter (~512 GB), so the job runs on an 8×H100 node; at 8 × $2.97/hr for a few hours of wall-clock, budget $100–300 per model, including data loading and evaluation overhead.
- LoRA / QLoRA fine-tune of the same 32B model: fits on 1–2 GPUs, so $30–100 per model is typical. Five 32B fine-tunes come in well under $1,000 either way — the expensive part becomes the evaluation runs and the people, not the GPUs.
- Serving those five models costs more over a year than training them did. See GPU training vs inference for why.
The VRAM sizing guide covers the memory side in detail: it decides how many GPUs the fine-tune needs, which matters more to the bill than the FLOPs do.
The full cost breakdown: what GPU-hours leave out
A "full" training budget is usually 1.5–3× the GPU-hour figure. The multipliers that get people:
| Line item | Typical size | Why it appears |
|---|---|---|
| Failed, restarted and ablation runs | +30–100% of the final run | Nobody trains the final configuration first. Large labs report the "final run" as a fraction of total compute spent. |
| MFU shortfall | up to 2× GPU-hours | A 20% MFU stack needs twice the hours of a 40% one for the same model. |
| Multi-node overhead | +5–20% | Communication-bound steps as the cluster grows; worse over Ethernet than InfiniBand. |
| Storage and checkpoints | $50–500/month per TB-class dataset | Training data, frequent checkpoints (a 70B model checkpoint is ~1 TB with optimizer state). |
| Data egress | $0.05–0.12/GB | Moving datasets in is free; moving checkpoints and models out is not. |
| Idle and setup time | +5–15% | Cluster provisioning, environment builds, and GPUs sitting idle between runs are billed at the full rate. |
| Evaluation and inference during training | +5–10% | Periodic eval, sampling, and reward-model passes. |
The non-GPU items are broken out further in hidden cloud GPU costs. The cheapest lever on the GPU line itself is the pricing tier: spot GPU instances cut the hourly rate 30–70% for exactly the checkpointed, restartable kind of job that pre-training is.
A five-line estimating procedure
- Fix N (parameters) and D (tokens). If you have not chosen D, 20 tokens per parameter is the compute-optimal starting point; production models are often trained on 10–100× that.
- Compute FLOPs = 6 × N × D.
- Pick a GPU and an MFU (use 30% unless you have measured better) and convert to GPU-hours.
- Multiply by the $/GPU-hour for the pricing tier you will really use — median on-demand from the price index, or a spot rate if the job can checkpoint.
- Multiply the total by 1.5–2 for the overheads above, then check the memory side with the GPU cost calculator, which also ranks the cheapest providers for the configuration.
Related reading
- GPU rental price index — the $/GPU-hour figures used above, with low / median / high per SKU.
- H100 vs A100 vs H200 vs B200 — which generation gives the lowest cost per training run.
- GPU carbon footprint calculator — the emissions that go with the GPU-hours.
- Glossary — FLOPS, MFU, HBM, mixed precision and the rest of the vocabulary.