Dimensioning the Bank's LLMs — v2

Interactive calculator for the AI Architecture note. Move the load inputs and the sizing follows the method of Appendix A of AI Architecture for Secure Client Interactions and Fraud Defense: model size is set by the task, and load sets replica counts. LLM 1 is Sarvam — it handles the open conversation with the customer; LLM 2 and LLM 3 are the bank's own models for guided interaction and document scrutiny.

Modeled planning estimates — not measurements. To be fixed by the pilot.

The RL NN (DRL defender) is NOT included in this dimensioning — it is a small CPU-only network, not a language model. Only the three language models below are sized here.

Load inputs

sess.
%

Model & deployment choices

%

Assumptions — the arithmetic behind the numbers

  • Weights memory: FP16 ≈ 2 GB per billion parameters; INT8 halves it (1 GB/B); 4-bit quarters it (0.5 GB/B). Headroom for the KV cache comes from the 80 GB card, not from extra cards.
  • Session token rate: peak tokens/sec = concurrent sessions × turns × tokens/turn ÷ session length. A session's turns are spread over its average length, so this is the sustained generation rate the replicas must carry.
  • Interaction replicas: one replica (7–14B, INT8, one 80 GB GPU) sustains ≈ 2,000 generated tokens/sec. replicas = ceil(peak tokens/sec × (1 + headroom) ÷ replica rate), spread across the regional nodes.
  • Front layer — Sarvam open conversation: Sarvam (LLM 1) handles the open, free-form conversation with the customer. 30B mixture-of-experts at FP16 ≈ 60 GB of weights, served on 2 × 80 GB GPUs per regional node (Appendix A default). Speech (ASR/TTS) runs alongside it on the same node.
  • Scrutiny batch: case files are processed in a nightly window at ≈ 1,500 tokens/sec per GPU on long documents. GPUs = ceil(daily tokens ÷ (window × rate) × (1 + headroom)), held inside the Appendix A band of 2–4 GPUs.
  • Cost anchors (from §12 of the note): one 8 × 80 GB GPU server ≈ $150K–$200K; the $175K midpoint gives a unit GPU price of ≈ $21.9K per GPU, applied to each component's computed GPU count in the cost table below. CPU/host servers (gateway, guardrails, defender host, orchestration) are priced per server. Subtotal plus 20% for storage, network and backup. Excludes manpower, installation, facilities and licence fees. ₹83 per $1. All costs are indicative planning estimates — not vendor quotations.
Verdict at this load —
Total GPUs –
Regional serving – Sarvam front layer (open conversation) + interaction replicas
National HQ – document scrutiny pool

LLM 1 — Sarvam Multilingual Front Layer Open conversation — Sarvam handles the free, open-ended conversation with the customer, in the customer's language, Hindi–English code-mixed

Regional nodes

Sarvam — open conversation. This is the Sarvam model (licensed, locally deployed): it carries the open conversation — the customer speaks freely, not in fixed menus — in Hindi / English / Hinglish. 30B mixture-of-experts · 32K context · real-time conversation. Speech (ASR/TTS) runs on the same node — no separate sizing. Guided, transactional flows then pass to LLM 2.

Nodes–
GPUs–
Weights / node–
Voice sessions–

2 × 80 GB GPUs per node per Appendix A — the 60 GB of FP16 weights plus KV cache and the speech pair fit the pair of cards.

Representative architecture for this size class (not a specific checkpoint)
Layers (transformer blocks)~48
Hidden size (nodes / layer)~3,072
Experts per layer8
Total weights~30B
Active weights / token~5B
Vocabulary~128K–256K tokens

Input: token embeddings — embedding dimension = hidden size (~3,072); vocab ~128K–256K tokens for an Indic tokenizer.

LLM 2 — Client Interaction LLM (bank's own) — guided flows and retrieval-grounded answers

Regional nodes

Dense, fine-tuned on bank documents · 8–32K context · 1 × 80 GB GPU per replica. Load sets the replica count; model size does not grow with customers.

Peak generation–
Replicas–
Per node–
Weights / replica–

Representative architecture for this size class (not a specific checkpoint)
Layers (transformer blocks)–
Hidden size (nodes / layer)–
Total weights–
Vocabulary~128K–256K tokens

Input: token embeddings — embedding dimension = hidden size (–); vocab ~128K–256K tokens for an Indic tokenizer.

LLM 3 — Document Scrutiny LLM (bank's own) — reads case files and the audit log; drafts summaries and STR narratives

National HQ

64–128K context for case files and audit logs · batch / offline, reads records and touches nothing · shares the HQ pool off-peak.

Daily tokens–
GPUs–
Weights total–
Window–

Representative architecture for this size class (not a specific checkpoint)
Layers (transformer blocks)–
Hidden size (nodes / layer)–
Total weights–
Vocabulary~128K–256K tokens

Input: token embeddings — embedding dimension = hidden size (–); vocab ~128K–256K tokens for an Indic tokenizer.

Branches

Endpoints only

No models, no GPUs, no local data store. Branch staff and customer devices are endpoints of the regional node above them; branch sizing scales with branch count, not with compute.

Indicative cost (US$)

Planning estimate

Each line picks up the capacity computed above — GPU capacity priced per GPU, CPU/host capacity per server. Indicative planning estimate — not a vendor quotation. Excludes manpower, installation, facilities and licence fees.

ComponentIndicative US$
LLM 1 — Sarvam front layer (open conversation)– –
LLM 2 — Client interaction– –
LLM 3 — Document scrutiny– –
CPU / host serversgateway, guardrails, defender host, orchestration –
Subtotal –
Storage, network & backup+20% of subtotal –
Total indicative cost– –