Skip to content

B200 vs B300 for Inference: Same Speed, More Room

The B300 keeps the B200's bandwidth and FP8 compute and adds memory, FP4 and attention speed. This guide prices a full 8-GPU node per concurrent user for a 671B mixture-of-experts model and shows when each GPU is the better buy.

Jim Cai6 min read
B200 vs B300 for Inference: Same Speed, More Room

Per GPU, the B300 is not faster than the B200 at FP8 or BF16; it has more memory, 1.5 times the FP4 compute and up to 2 times faster attention. On fal it costs 1.63 times a B200 at list price and 1.09 times at "as low as" rates. For a 671B MoE model on an 8-GPU node, the B300 holds about twice the 32K sessions, so it costs less per concurrent user even at list price, at some cost in per-user speed.

Like the H200 before it, Blackwell Ultra does not get faster at FP8 or BF16. The B300 keeps the B200's 8 TB/s of bandwidth and its FP8 and BF16 compute. It adds memory, FP4 compute and faster attention, and it gives up almost all of its INT8 and FP64.12 So per GPU, a B300 is not a faster B200. It is a bigger one. For inference that means the useful comparison is not one GPU against one GPU. It is how many users a whole 8-GPU node can serve, and what each of them costs.

Same speed, more room

B200B300B300 ÷ B200
Usable memory, 8-GPU system1,440 GB2.1 TB~1.46×
Memory bandwidth per GPU8 TB/s8 TB/s1.00×
FP8 and BF16 dense computeSameSame1.00×
FP4 dense, per 8 GPUs72 PFLOPS108 PFLOPS1.50×
Attention-layer computeBaselineUp to 2×up to 2×
INT8, per 8 GPUs (sparse)72 POPS3 POPS0.04×
fal list price, billed per second$7.99/h ($0.002219/s)$12.99/h ($0.003608/s)1.63×
fal "as low as"$5.49/h$5.99/h1.09×

Figures are from NVIDIA's HGX and DGX specifications and the Blackwell Ultra technical blog.1234 Prices are per GPU-hour from fal.ai/pricing.5 The B200 price guide and the B300 price guide cover specs, billing and reservations for each GPU on its own.

Two numbers drive everything below. At list price, the B300 must do 1.63 times the work per hour to break even. At "as low as" rates, only 1.09 times.

Worked example: a 671B mixture-of-experts model on one node

Many of the largest open models are now mixture-of-experts (MoE) models that need a full node. DeepSeek-V3 has 671B parameters, of which 37B are active per token, and ships with FP8 weights.6 Its attention stores a compressed KV cache: 61 layers × (512 + 64) values × 2 bytes, about 69 KiB per token.7

Load the weights on an 8-GPU node, keep 10% of memory for the runtime, and give each session 32,768 tokens of context:

8× B2008× B300
Memory after weights and runtime~625 GB~1,220 GB
Concurrent 32K sessions~270~530
Node price, list$63.92/h$103.92/h
List price per session-hour~$0.24~$0.20
Node price, "as low as"$43.92/h$47.92/h
"As low as" per session-hour~$0.16~$0.09

The weights take the same 671 GB on both nodes, so the B300's extra memory, about 600 GB after the runtime reserve, all goes to KV cache. That nearly doubles the sessions for 1.63 times the price. Per concurrent user, the B300 node is cheaper even at list price. At "as low as" rates it costs a little over half as much.

The trade: per-user speed

More sessions means more KV cache to read on every decode step, and bandwidth per GPU is the same. With both nodes full, each B300 step reads about 1.5 times as many bytes, so each user sees roughly two-thirds of the B200's token rate. This treats every weight as read on each step. An MoE model reads fewer weights per step, so the per-user gap can be larger. Total throughput per node still goes up.

That gives a simple rule:

  • Cost per user is the target (chat at scale, agents with long histories, batch reasoning): fill the B300.
  • Per-user latency is the target (interactive coding, voice): run the B300 below full load, or stay on the B200.

Where the B300's other gains show up

  • Long-context prefill. Up to 2 times faster attention helps most when prompts are long, because attention cost grows with the square of the prompt length.3 Short prompts gain little.
  • FP4 models. 1.5 times the FP4 compute speeds up prefill and diffusion for models that keep their quality at 4 bits. Decode stays bandwidth-bound and gains nothing from extra FP4 compute.
  • Video and multi-model pipelines. 288 GB per GPU holds a large video model plus its text encoder and VAE, or several chained models, without swapping weights.

Where the B200 stays ahead

  • Models that already fit with room to spare. If a model and its peak KV cache fit on a B200, the B300 adds cost but not speed at FP8 or BF16.
  • INT8 models. INT8 throughput on the B300 is about 4% of the B200's.1 Re-quantize to FP8 or FP4 before moving, or stay put.
  • Self-serve capacity. On fal, the B200 is a standard Serverless machine type, GPU-B200, that you set in code and that scales from zero.8 fal's pricing page lists the B300 at $12.99 per GPU-hour, billed per second ($0.003608 a second), but the B300 is not a standard machine type in the docs.58 B300 capacity is set up with fal's team, as single GPUs or full 8-GPU nodes, so it suits a planned baseline more than quick experiments.

A sizing checklist

  1. Does the model fit on one B200 node with the KV cache your peak needs? If yes and you buy at list price, stay on the B200.
  2. Is concurrency limited by memory? If requests queue because KV cache runs out, price the B300 per session, as above.
  3. Are your weights FP8, BF16 or FP4? INT8 needs re-quantizing first.
  4. Will the node stay busy? The B300's best rate comes with committed capacity. The break-even guide shows how busy a GPU must be before a commitment beats per-second billing.

FAQ

Is the B300 faster than the B200?

Only for some work. It has the same memory bandwidth and the same FP8 and BF16 compute. It is faster at FP4 (1.5 times) and at attention (up to 2 times), and it has about 46% more usable memory per node.13 Most of its inference gain comes from serving more users per node.

Is the B300 worth the price over the B200?

On fal the B300 lists at $12.99 per GPU-hour and the B200 at $7.99, or 1.63 times. At "as low as" rates they are $5.99 and $5.49, or 1.09 times. For large models where memory limits concurrency, the B300 can cost less per user even at list price. For models that fit easily on a B200, it rarely pays at list price.

Can a B300 run INT8 models?

It can, but slowly. NVIDIA lists 3 POPS of INT8 for an 8-GPU HGX B300, against 72 for an HGX B200.1 Use FP8 or FP4 on a B300.

How do I get B300 capacity on fal?

The B300 lists at $12.99 per GPU-hour, billed per second ($0.003608 a second), and goes as low as $5.99 with committed capacity.5 To set up B300 capacity, contact the fal team at fal.ai/enterprise with your model and expected traffic. The B300 price guide has more detail.

References

  1. NVIDIA HGX Platform ↩ ↩2 ↩3 ↩4 ↩5

  2. NVIDIA DGX B300 ↩ ↩2

  3. Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era, NVIDIA Developer Blog ↩ ↩2 ↩3

  4. NVIDIA DGX B200 ↩

  5. fal Pricing ↩ ↩2 ↩3

  6. DeepSeek-V3 model card, Hugging Face ↩

  7. DeepSeek-V3 configuration, Hugging Face ↩

  8. fal Docs: Machine Types ↩ ↩2

Run your own models on fal Serverless

Run any AI workload on managed, autoscaling GPUs.