Fine-Tuning VRAM Calculator (LoRA, QLoRA, Full)

View results · 11.9 GB

Training setup

Method

The frozen base is stored in 4-bit NF4 with double quantization; only the adapters are trained.

LoRA target modules
Adapter weights

PEFT creates adapter weights in fp32; TRL's SFTTrainer casts them to bf16 when the base model is quantized (QLoRA). Choose fp32 for plain PEFT.

Optimizer

Estimated GPU memory

Total VRAM
Trainable parameters
41.94M (0.52%)
Weights, gradients, optimizer
6.2 GB
Activations
0.8 GB

Smallest GPU on the list that fits: RTX 3060 12GB (12 GB), with 1.0 GB to spare.

Without gradient checkpointing: 21.1 GB.

Model: 8,030,261,248 parameters, 32 layers, hidden size 4,096, vocabulary 128,256.

PEFT's prepare_model_for_kbit_training upcasts the embeddings and output head to fp32: add 2.1 GB if your script calls it.

An estimate, not a measurement: activation memory depends on the framework and its kernels. Model data and GPU list checked on 2026-10-24.

Memory breakdown

Memory breakdown
PartRuleMemoryShare
Frozen base weights (4-bit NF4)0.5 B×Ψlin0.5\,\text{B} \times \Psi_{\text{lin}}3.49 GB29.3%
Quantization constants0.127 bit×Ψlin0.127\,\text{bit} \times \Psi_{\text{lin}}0.11 GB0.93%
Embeddings and output head (bf16)2 B×Ψemb2\,\text{B} \times \Psi_{\text{emb}}2.10 GB17.6%
LoRA adapter weights2 B×ΨLoRA2\,\text{B} \times \Psi_{\text{LoRA}}0.08 GB0.7%
Adapter gradients2 B×ΨLoRA2\,\text{B} \times \Psi_{\text{LoRA}}0.08 GB0.7%
Optimizer states (adapters)8 B×ΨLoRA8\,\text{B} \times \Psi_{\text{LoRA}}0.34 GB2.8%
Activations (checkpointed)2sbhL+34sbh2sbhL + 34sbh0.82 GB6.9%
Output logits for the loss10 B×sbV10\,\text{B} \times sbV2.63 GB22%
CUDA context and kernels1.3 GB1.3\,\text{GB}1.30 GB10.9%
Safety margin10 %10\,\%0.97 GB8.1%
Total VRAM11.92 GB100%

Ψ\Psi is the number of parameters, Ψlin\Psi_{\text{lin}} those in linear layers, Ψemb\Psi_{\text{emb}} the embedding and output-head parameters and ΨLoRA\Psi_{\text{LoRA}} the adapter parameters; ss is the sequence length, bb the micro-batch, hh the hidden size, LL the number of layers, VV the vocabulary and NN the number of GPUs. B is bytes; GB means a billion bytes.

Which GPUs fit

  • RTX 3060 12GB 12 GB (12.9 GB)1.0 GB free
  • RTX 4060 Ti 16GB 16 GB (17.2 GB)5.3 GB free
  • RTX 4080 16 GB (17.2 GB)5.3 GB free
  • Tesla P100 16GB 16 GB (17.2 GB)5.3 GB free
  • RTX 3090 24 GB (25.8 GB)13.9 GB free
  • RTX 4090 24 GB (25.8 GB)13.9 GB free
  • Tesla P40 24 GB (25.8 GB)13.9 GB free
  • MI50 32GB 32 GB (34.4 GB)22.5 GB free
  • RTX 5090 32 GB (34.4 GB)22.5 GB free
  • Tesla V100 32GB 32 GB (34.4 GB)22.5 GB free
  • A100 40GB 40 GB (42.9 GB)31.0 GB free
  • L40S 48 GB (51.5 GB)39.6 GB free
  • RTX 4090 48GB 48 GB (51.5 GB)39.6 GB free
  • RTX A6000 48 GB (51.5 GB)39.6 GB free
  • A100 80GB 80 GB (85.9 GB)74.0 GB free
  • H100 80GB 80 GB (85.9 GB)74.0 GB free
  • H100 NVL 94 GB (100.9 GB)89.0 GB free
  • RTX PRO 6000 96 GB (103.1 GB)91.2 GB free
  • DGX Spark GB10 128 GB (137.4 GB)125.5 GB free
  • H200 141 GB (151.4 GB)139.5 GB free
  • B200 180 GB (193.3 GB)181.4 GB free

Nominal memory from the vendors' spec pages. Vendors count binary gigabytes, so a 24 GB card holds 25.8 billion bytes (in brackets); the fit uses that figure, since the CUDA context is already part of the total. The MI50 has no CUDA build of bitsandbytes (QLoRA, 8-bit AdamW); the V100 and Pascal cards (P40, P100) have no bf16, and in fp16 the memory is the same.

About this tool

Fine-tuning a language model needs far more GPU memory than running it. Besides the weights, the GPU holds gradients, optimizer states and the activations of every layer until the backward pass. This calculator adds those parts up for full fine-tuning, LoRA and QLoRA, for a model from the list or one you describe, and shows which common GPUs have room for the result.

Every part is listed with the rule behind it and its source, because fine-tuning calculators often disagree by a factor of two: one counts 18 bytes per parameter, another 12, one leaves out the logits, another assumes checkpointing. Here each assumption is a setting you can see and change.

How the memory is estimated

Full fine-tuning in mixed precision keeps, for every parameter, 2 bytes of bf16 weights, 2 bytes of gradients, a 4-byte fp32 master copy and 8 bytes of AdamW momentum and variance: 16 bytes, the accounting of the ZeRO paper (Rajbhandari et al., 2020). 8-bit AdamW stores 2 bytes of optimizer state instead of 8, and SGD with momentum 4. ZeRO stages 1, 2 and 3 divide the optimizer states, then the gradients, then the weights by the number of GPUs.

LoRA freezes the base model at 2 bytes per parameter and trains two small matrices beside each adapted weight: a d×kd \times k weight gets r(d+k)r(d + k) adapter parameters (Hu et al., 2021). Only these have gradients and optimizer states: PEFT creates them in fp32 (16 bytes each with AdamW), and TRL's SFTTrainer casts them to bf16 for quantized models (12 bytes). QLoRA stores the frozen linear layers in 4-bit NormalFloat with double quantization, 4+8/64+32/(64⋅256)≈4.1274 + 8/64 + 32/(64 \cdot 256) \approx 4.127 bits per parameter (Dettmers et al., 2023), while the embeddings and output head stay in bf16.

Activations depend on the data. Each layer stores about 34 sbh34\,sbh bytes for sequence length ss, micro-batch bb and hidden size hh (Korthikanti et al., 2022); FlashAttention removes their attention-matrix term. Gradient checkpointing keeps only each layer's input, 2 sbh2\,sbh, and recomputes one layer at a time. The loss adds about 10 bytes per token and vocabulary entry for the logits unless a fused kernel computes it in chunks. With WW the weights, GG the gradients, OO the optimizer states and master copy, AA the activations and Λ\Lambda the logits, plus 1.3 GB for the CUDA context and a 10% margin:

M=(W+G+O+A+Λ)×1.1+1.3 GBM = (W + G + O + A + \Lambda) \times 1.1 + 1.3\ \text{GB}

Why fine-tuning calculators disagree, and what is left out

Three assumptions explain most of the gap between calculators, and each is a setting here. Bytes per parameter: the Hugging Face training guide counts 18 for mixed-precision AdamW (fp32 gradients and a second copy of the weights), the ZeRO paper 16, and the QLoRA paper's "more than 780 GB" for 16-bit fine-tuning of LLaMA 65B matches 12, without a master copy; switch the master weights off to reproduce it. Logits: with Llama 3's vocabulary of 128,256 tokens, 2,048 tokens of logits take 2.6 GB, which fused cross-entropy kernels avoid. Checkpointing: for Llama 3.1 8B at 2,048 tokens, switching it off raises activations from 0.8 GB to 9.1 GB.

Not modelled: short-lived buffers inside one operation (the dequantized weight of one layer in QLoRA, the layer ZeRO-3 gathers), CPU offloading and paged optimizers, mixture-of-experts models, vision encoders, and preference training such as DPO, which keeps a second, reference model. Use the result to plan, then check the first training steps with torch.cuda.max_memory_allocated().

Example

QLoRA paper check. LLaMA 65B has 65.29 billion parameters, 80 layers and a hidden size of 8,192. With QLoRA, bf16 adapters of rank 64 on all linear layers as in the paper, and 512-token sequences: the 4-bit weights take 32.38 GB, the quantization constants 1.03 GB and the bf16 embeddings 1.05 GB; the 799.5 million adapter parameters take 1.60 GB, their gradients 1.60 GB and their AdamW states 6.40 GB; checkpointed activations 0.81 GB and logits 0.16 GB. That is 45.0 GB of tensors, and 50.8 GB with the 1.3 GB context and the 10% margin. A 48 GB card holds 51.5 billion bytes, so the paper's single 48 GB GPU fits with 0.7 GB to spare; the paper's paged optimizers add more slack by moving the 6.4 GB of optimizer states to CPU memory when the GPU runs short.

ZeRO check. For a 7.5 billion parameter model on 64 GPUs, the ZeRO paper gives 120 GB of model states per GPU without sharding, 31.4 GB at stage 1, 16.6 GB at stage 2 and 1.88 GB at stage 3. The rules here give 120, 31.41, 16.64 and 1.875 GB.

Llama 3.1 8B. LoRA of rank 16 on all linear layers (41.9 million adapter parameters, bf16), 2,048 tokens, a micro-batch of 1 and checkpointing: 16.06 GB of frozen weights, 0.50 GB for the adapters with their gradients and optimizer states, 0.82 GB of activations and 2.63 GB of logits, 23.3 GB in all. Unsloth lists 22 GB as the minimum for 16-bit LoRA on an 8B model. The same run with QLoRA needs 11.9 GB, or 9.0 GB with a fused loss.

Frequently asked questions

How much VRAM do I need to fine-tune a 7B or 8B model?

With QLoRA, about 9 to 12 GB at 2,048 tokens and a micro-batch of 1: 11.9 GB with the default settings and 9.0 GB with a fused loss. A 12 GB card (12.9 billion bytes) just holds it; 16 GB leaves room for longer sequences. 16-bit LoRA needs about 23 GB, so a 24 GB card. Full fine-tuning with AdamW needs about 146 GB, which means several GPUs with ZeRO or FSDP. Longer sequences and larger batches add activation memory; the calculator shows how much.

What is the difference between LoRA and QLoRA memory?

Both train the same small adapters; the difference is the frozen base model. LoRA keeps it in bf16 at 2 bytes per parameter, QLoRA in 4-bit NF4 at about 0.52 bytes. For Llama 3.1 8B that is 16.1 GB against 5.7 GB, the bf16 embeddings included. QLoRA trains somewhat slower because each layer is dequantized on the fly.

Why does full fine-tuning need 16 bytes per parameter?

Mixed-precision training computes in bf16 (2 bytes of weights and 2 of gradients per parameter) but updates an fp32 master copy (4 bytes), and AdamW keeps two fp32 statistics per parameter, momentum and variance (8 bytes). An 8B model therefore needs about 128 GB before activations. 8-bit AdamW brings it to 10 bytes per parameter, and ZeRO or FSDP splits the total across GPUs.

Does gradient checkpointing reduce memory?

Yes, it reduces activation memory. For Llama 3.1 8B at 2,048 tokens, activations fall from 9.1 GB to 0.8 GB, because only each layer's input is kept and the rest is recomputed during the backward pass. The cost is an extra forward pass: Korthikanti et al. measured as much as 30 to 40% more training time.

How do I split fine-tuning across several GPUs?

With DeepSpeed ZeRO or PyTorch FSDP. Stage 1 splits the optimizer states, stage 2 also the gradients, and stage 3 (FSDP full sharding) also the weights. Full fine-tuning of Llama 3.1 8B on eight GPUs at stage 3 needs about 22.8 GB per GPU. Activations are not split: each GPU processes its own micro-batch. Enter the number of GPUs to see the memory each one needs.

Sources

Link with your values

Add these parameters to this page’s address to open it with the values filled in, for example to share a calculation or to link to it from an AI assistant. The result is calculated in the browser: the page HTML shows an example for the default values, and the tool sets the data-url-params attribute once the link’s values are applied.

ParameterValueIf omitted
methodMethodone of: full, lora, qloradefault qlora
architectureCustom modelone of: preset, customdefault preset
modelModelone of: llama3.2-1b, llama3.2-3b, qwen3-4b, mistral-7b-v0.3, qwen2.5-7b, llama3.1-8b, qwen3-8b, phi-4, qwen3-14b, qwen3-32b, llama3.3-70b, llama-2-7b, llama-7b, llama-65b; used with architecture = presetdefault llama3.1-8b
parameters_billionsParametersnumber; allowed 0.01–2000; used with architecture = customrequired
layersLayerswhole number; allowed 1–512; used with architecture = customrequired
hidden_sizeHidden sizewhole number; allowed 64–65536; used with architecture = customrequired
vocabulary_sizeVocabulary sizewhole number; allowed 1–2000000; used with architecture = customrequired
tied_embeddingsTied embeddings (tie_word_embeddings)true or false; used with architecture = customdefault false
optimizerOptimizerone of: adamw, adamw_8bit, sgddefault adamw
master_weightsKeep an fp32 master copy of the weightstrue or false; used with method = fulldefault true
lora_rankLoRA rank (r)whole number; allowed 1–1024; used with method = lora, qloradefault 16
lora_targetsLoRA target modulesone of: qv, attention, all; used with method = lora, qloradefault all
adapter_precisionAdapter weightsone of: fp32, bf16; used with method = lora, qloradefault bf16
sequence_lengthSequence length (tokens)whole number; allowed 1–1048576default 2048
batch_sizeMicro-batch per GPUwhole number; allowed 1–1024default 1
gradient_checkpointingGradient checkpointingtrue or falsedefault true
fused_lossFused cross-entropy kernel (Liger, Unsloth)true or falsedefault false
gpu_countNumber of GPUswhole number; allowed 1–1024default 1
zero_stageSharding (ZeRO stage)one of: 0, 1, 2, 3default 0

Examples

  • ?method=qlora&model=llama-65b&lora_rank=64&adapter_precision=bf16&sequence_length=512
  • ?method=lora&model=llama3.1-8b&lora_rank=16&lora_targets=all&sequence_length=2048&batch_size=1
  • ?method=full&model=llama3.1-8b&gpu_count=8&zero_stage=3
  • ?method=qlora&architecture=custom&parameters_billions=12&layers=48&hidden_size=3840&vocabulary_size=262144&tied_embeddings=true

Write decimals with a point. Encode a space as %20, + as %2B, # as %23 and & as %26.

Result elements: [data-result-block="fine-tuning-vram"]