Show the math

The calculation chain behind the current config — every formula symbolically, then with your numbers, then the result. Tied to the shared config, so it matches Modelling / Workload / Training exactly.

How to use this tab

Pick the view. Inference is the Modelling / Workload roofline (memory fit, step time, throughput, TTFT). Training is the fine-tune memory + step-time story. Change the config below (or on any other tab) and the math updates live. Results are sourced from the same functions the tool computes with — this tab can't drift from them.

View

Model

GPU

GPUs per node

Instances (nodes)

= 8 GPUs total (1 × 8)

Network fabric (per GPU)

Tensor parallel (TP)

Pipeline parallel (PP) split layers into stages

Phase

Batch size 32

Input tokens 4k tok

Output tokens 4k tok

Total context = input + output = 8k tok. For multimodal models, count image/video patch tokens in input: a token is a token to the transformer.

Cached prefix 0%

Speculative decoding draft proposes, target verifies

Weight format

KV cache precision

KV allocation

Setup

Model parameters

P = 70.6 B

70.6B

GPU peak (compute · bandwidth)

π_{flops}, π_{bw} = 99 TFLOP/s, 3.35 TB/s

99 TFLOP/s (fp16), 3.35 TB/s

Parallelism

t_{p}, p_{p}, e_{p}, d_{p} = 8, 1, 1, 1

TP 8 · PP 1 · EP 1 · DP 1 (of 8 GPUs)

Sequence

L = s_{in} + s_{out} = 4096 + 4096 = 8192

8192 tokens

Memory (per GPU)

What one GPU must hold for this replica.

Weights

W = \frac{P \cdot b _{w}}{t _{p} \cdot p _{p}} = \frac{70.6 B \cdot 2}{8 \cdot 1} = 17.6 GB

b_w = 2 bytes/param (BF16 / FP16)

17.6 GB

KV per token · layer

k_{tl} = 2 n_{k v} d_{h} b_{k v} = 2 \cdot 8 \cdot 128 \cdot 2 = 4096 B

GQA: 8 KV heads

4096 B

KV cache

KV = \frac{n _{kvL} L k _{tl}}{s _{k v}} \cdot w_{pad} \cdot B = \frac{80 \cdot 8192 \cdot 4096}{8} \cdot w_{pad} \cdot 32 = 11 GB

paged allocation, KV on 80/80 layers, sharded 8×

11 GB

Activations

A = \frac{2 B L h b _{a}}{t _{p} p _{p}} = \frac{2 \cdot 32 \cdot 8192 \cdot 8192 \cdot 2}{8} = 1.07 GB

1.07 GB

Overhead

O = 1.5 GiB

CUDA context + framework + fragmentation

1.61 GB

Total vs capacity

U = W + KV + A + O ≶ C = 31.3 GB \leq 85.9 GB

31.3 GB / 85.9 GB — fits

Throughput (roofline)

A step is bounded by the slower of memory traffic and compute, plus in-stage collectives.

Bytes read per step (decode: weights + KV)

b_{step} = W + KV = 17.6 GB + 11 GB = 28.6 GB

28.6 GB

Memory time

t_{mem} = \frac{b _{step}}{π _{bw} \cdot MBU} = \frac{28.6 GB}{3.35 TB/s \cdot 0.8} = 10.7 ms

MBU = 0.8

10.7 ms

FLOPs per step

f_{step} = 2 n_{tok} (\frac{P _{ne}}{t _{p} p _{p}} + \frac{P _{exp,act}}{e _{p} p _{p}}) = 2 \cdot 32 \cdot (\dots) = 565 GFLOP

0.565 TFLOP (32 tokens/step)

Compute time

t_{cmp} = \frac{f _{step}}{π _{flops} \cdot MFU \cdot ϕ} = \frac{0.565 T}{99 \cdot 0.74 \cdot 1} = 0.771 ms

MFU = 0.74, per-GPU factor φ = 1

0.771 ms

Step time (roofline + comm)

t_{step} = max (t_{mem}, t_{cmp}) + t_{comm} = max (10.7 ms, 0.771 ms) + t_{comm} = 11.8 ms

bottleneck: memory

11.8 ms

Aggregate throughput

T = \frac{n _{tok}}{t _{step}} \cdot d_{p} = \frac{32}{0.0118 s} \cdot 1 = 2706 tok/s

2706 tok/s

Per-user throughput

T_{u} = \frac{s _{tok}}{p _{p} t _{step} + ( p _{p} - 1 ) t _{pp}} = \approx 84.6 tok/s

84.6 tok/s per user

Latency (TTFT)

Prefill FLOPs (uncached input)

f_{pre} = 2 B s_{pre} (\frac{P _{ne}}{t _{p} p _{p}} + \frac{P _{exp,act}}{e _{p} p _{p}}) = 2 \cdot 32 \cdot 4096 \cdot (\dots) = 2313 TFLOP

2313 TFLOP

Time to first token

TTFT = \frac{f _{pre}}{π _{flops} \cdot MFU \cdot ϕ} = \frac{2313 T}{99 \cdot 0.74 \cdot 1} = 3159 ms

3159 ms