Self-Hosting LLM Inference After DeepInfer: What An Engine Does, When It Pays For A Trading Firm, And The AI Bill Of Materials
On 20 October 2026 DeepSeek open-sourced DeepInfer, the inference engine it says runs its own production services, three days after releasing DeepSeek V4.1 Flash as open weights; a day earlier, CyberScoop reported a US Department of Homeland Security order barring DeepSeek products and services from federal civilian agencies. This guide explains what an inference engine does in terms a CTO can act on, sets out when self-hosting LLM inference pays for a trading firm and when it does not, and builds three Python pieces: an overnight structured-output batch job for trade confirmations, a cost and batch-window calculator with every price as an input, and an AI bill of materials with a CI gate that routes each client to a model origin their contract allows.
AlchmAI Engineering18 min read
17.5k
GitHub stars on DeepSeek's DeepInfer repository when checked on 20 October 2026, the day of its first commit, with 1.8k forks
2,290
Points, with 878 comments, on the top Hacker News story in the week of DeepSeek's October releases
4 and 64
SM100 GPUs behind DeepInfer's published prefill and decode benchmarks respectively, per its README
19 Oct
2026: CyberScoop reports a DHS order barring all IT products and services from Hangzhou DeepSeek Artificial Intelligence from US federal civilian agencies
On 20 October 2026 DeepSeek published DeepInfer on GitHub under the Apache-2.0 licence, describing it as 'the high-performance inference framework that powers DeepSeek's large-scale production services'. Three days earlier, on 17 October, it released DeepSeek V4.1 Flash as open weights on Hugging Face and ModelScope. Labs usually publish weights and leave serving to vLLM, SGLang or a cloud provider; this time DeepSeek also published its serving layer. For a trading firm weighing self-hosting LLM inference, that moves the question from 'can we get a capable model?' to 'can we run the serving layer ourselves, at what cost, and can we prove to a client where every model we use came from?'
Engineers noticed. The top Hacker News story in the week of the releases reached 2,290 points and 878 comments, a thread on the model's Hugging Face page reached 959 points and 721 comments, and the repository had 17.5k stars and 1.8k forks when we checked on 20 October 2026. The politics arrived at the same time. On 19 October 2026 CyberScoop reported that the US Department of Homeland Security had barred all IT products and services from Hangzhou DeepSeek Artificial Intelligence from federal civilian agencies under the Federal Acquisition Supply Chain Security Act: agencies must exclude them from procurement and remove existing deployments, and exceptions need a 'compelling justification'. This guide does not tell you to adopt or avoid DeepSeek. It explains what an inference engine does, gives a cost test for self-hosting, and builds the record that answers a client's model-origin question in minutes.
1. What Does An LLM Inference Engine Actually Do?
The weights are a file. Turning that file into answers at a predictable cost is the engine's job, and five mechanisms decide most of the economics.
- Prefill versus decode. Prefill reads the whole prompt in one parallel pass and is limited by compute. Decode then produces the answer one token at a time and is limited by memory bandwidth. Classifying a long trade confirmation into a short JSON label is prefill-heavy; drafting a client note is decode-heavy. The two phases load a GPU differently, which is why they are benchmarked separately and, at DeepSeek's scale, run on separate machines.
- Continuous batching. Instead of waiting for a whole batch to finish, the scheduler admits new requests and retires finished ones at every decode step, so a GPU is not left idle while one long answer completes. This is why a self-hosted GPU's cost per job depends so heavily on how full you keep it.
- KV cache. For every token already processed, attention stores key and value tensors so they are not recomputed. That memory limits how many requests fit on a GPU at once, and reusing it across requests that share a prefix, such as the same instructions and schema, is a large part of what makes production serving cheap.
- Scheduling. The scheduler decides which requests run in each step, how prefill work is split between decode steps, and what waits. DeepInfer lists asynchronous scheduling, a 'zero-overhead batch scheduler' that overlaps CPU batch scheduling with device execution.
- Structured output. The engine masks any token that would break a regex, JSON schema or grammar, so the output parses by construction. DeepInfer supports all three through an so_args field on SamplingParams; vLLM does it through StructuredOutputsParams.
Why Does Open-Sourcing The Engine Matter More Than The Weights?
Weights tell you what a model can do; the engine decides what it costs to do it at scale. DeepInfer's feature list reads like a description of DeepSeek's own serving problems: expert parallelism with load balancing for its mixture-of-experts models, draft-token decoding with 'confidence-scheduled verification', and custom kernels. Two features matter directly to regulated users. Batch-invariant inference keeps 'model results identical across different batch compositions', and deterministic sampling produces 'identical outputs for identical input tokens and sampling parameters', both on NVIDIA only. A firm that must show why a confirmation was labelled one way last Tuesday wants an answer that does not change with whatever else shared the batch. DeepInfer credits vLLM as the base it was 'originally built primarily on', so these ideas can travel back into engines a small team already runs.
Read the limits before the star count. This release is offline inference only and supports one model, DeepSeek V4.1 Flash, with the default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017. It builds from source with ./build.sh on Linux with Python 3.12 and CUDA 13, and runs on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs. The README lists three things 'required for a complete inference service' that are 'not included in this release': a KV cache system, prefill and decode disaggregation, and load balancing across data-parallel replicas. Its benchmark charts were measured on 4 SM100 GPUs for prefill and 64 for decode, and it gives no numeric comparison with vLLM or SGLang. It is a reference design, not a drop-in replacement for an API.
The workload where an offline engine fits a trading firm today is the overnight batch. The job below labels trade-confirmation text against a fixed JSON schema. The engine guarantees the JSON parses; code then checks what a schema cannot, such as whether the counterparty LEI the model returned appears in the source text and whether settlement falls on or after the trade date. Anything that fails goes to a person. It runs on vLLM's documented offline API and includes an adapter for DeepInfer that follows the SamplingParams and so_args call in DeepInfer's own structured-output example. DeepInfer builds its LLM object through the repository's example launcher, whose parallelism arguments depend on your cluster, so the adapter takes that object as given rather than guessing a constructor.
# Python 3.12, pydantic 2.x, and one engine:
# vLLM 0.11 or later (StructuredOutputsParams), or
# DeepInfer's first open-source release (20 October 2026), built from source with ./build.sh
"""Overnight batch: label trade-confirmation text against a fixed JSON schema.
The engine guarantees the JSON parses. Code checks what a schema cannot.
Anything that fails goes to review.jsonl for a person, never downstream.
"""
from __future__ import annotations
import hashlib
import json
import os
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Literal, Protocol
from pydantic import BaseModel, Field, ValidationError
ISO_DATE = r"^\d{4}-\d{2}-\d{2}$"
LEI = re.compile(r"\b[A-Z0-9]{18}[0-9]{2}\b")
class ConfirmationLabel(BaseModel):
product: Literal["fx_spot", "fx_forward", "fx_swap", "equity", "cfd", "future", "option", "bond", "other"]
event: Literal["new", "amendment", "cancellation", "allocation", "unknown"]
trade_date: str = Field(pattern=ISO_DATE)
settlement_date: str | None = Field(pattern=ISO_DATE)
counterparty_lei: str | None = Field(pattern=r"^[A-Z0-9]{18}[0-9]{2}$")
needs_review: bool
evidence: str = Field(max_length=200, description="Shortest exact quote supporting product and event")
PROMPT = (
"You label one trade confirmation for a back office. Reply with JSON only, matching the schema. "
"Copy dates as YYYY-MM-DD. If a field is not stated, use null or unknown and set needs_review to true.\n\n"
"Confirmation:\n"
)
class Engine(Protocol):
name: str
def label(self, prompts: list[str], schema: dict[str, Any]) -> list[tuple[str, str]]: ...
class VllmEngine:
name = "vllm"
def __init__(self, model: str) -> None:
from vllm import LLM
self.llm = LLM(model=model)
def label(self, prompts: list[str], schema: dict[str, Any]) -> list[tuple[str, str]]:
from vllm import SamplingParams
from vllm.sampling_params import StructuredOutputsParams
params = SamplingParams(temperature=0, max_tokens=400,
structured_outputs=StructuredOutputsParams(json=schema))
outs = self.llm.generate(prompts, params) # the engine batches these continuously
return [(o.outputs[0].text, o.outputs[0].finish_reason or "") for o in outs]
class DeepInferEngine:
"""Wraps the LLM object built by DeepInfer's example launcher (examples/_config.py), whose
parallelism arguments depend on your cluster. The call follows examples/_structured_output.py."""
name = "deepinfer"
def __init__(self, llm: Any) -> None:
self.llm = llm
def label(self, prompts: list[str], schema: dict[str, Any]) -> list[tuple[str, str]]:
from deepinfer.sampling_params import SamplingParams
so_args = {"schema_type": "json", "schema": json.dumps(schema)}
outs = self.llm.generate(
prompts=prompts,
sampling_params=[SamplingParams(temperature=0, max_tokens=400, so_args=so_args) for _ in prompts],
use_tqdm=False,
)
tok = self.llm.get_tokenizer()
results: list[tuple[str, str]] = []
for out in outs: # one list of per-token outputs per prompt; the last one carries the finish state
last = out[-1]
text = tok.decode([t.token_id for t in out[:-1]], skip_special_tokens=False,
clean_up_tokenization_spaces=False)
results.append((text, last.finish_reason if last.finished else "unfinished"))
return results
def check(source: str, raw: str, finish: str) -> tuple[ConfirmationLabel | None, list[str]]:
if finish != "stop":
return None, [f"finish_reason={finish}"]
try:
label = ConfirmationLabel.model_validate_json(raw)
except ValidationError as err:
return None, [f"schema: {err.error_count()} errors"]
problems: list[str] = []
if label.counterparty_lei and label.counterparty_lei not in LEI.findall(source):
problems.append("LEI not found in the source text")
if label.settlement_date and label.settlement_date < label.trade_date:
problems.append("settlement date before trade date")
if label.needs_review:
problems.append("model asked for review")
return label, problems
def run(engine: Engine, model_id: str, src: Path, out_dir: Path) -> None:
rows = [json.loads(line) for line in src.read_text(encoding="utf-8").splitlines() if line.strip()]
results = engine.label([PROMPT + r["text"] for r in rows], ConfirmationLabel.model_json_schema())
out_dir.mkdir(parents=True, exist_ok=True)
stamp = datetime.now(timezone.utc).isoformat()
with (
(out_dir / "labelled.jsonl").open("w", encoding="utf-8") as ok,
(out_dir / "review.jsonl").open("w", encoding="utf-8") as review,
(out_dir / "audit.jsonl").open("a", encoding="utf-8") as audit,
):
for row, (raw, finish) in zip(rows, results, strict=True):
label, problems = check(row["text"], raw, finish)
routed = "review" if problems or label is None else "labelled"
record = {"id": row["id"], "label": label.model_dump() if label else None, "problems": problems}
(review if routed == "review" else ok).write(json.dumps(record) + "\n")
audit.write(json.dumps({
"at": stamp, "id": row["id"], "engine": engine.name, "model": model_id, "routed": routed,
"input_sha256": hashlib.sha256(row["text"].encode()).hexdigest(),
"output_sha256": hashlib.sha256(raw.encode()).hexdigest(),
}) + "\n")
if __name__ == "__main__":
# LABEL_MODEL: a local path or Hugging Face ID pinned to the revision recorded in your AI-BOM
model = os.environ["LABEL_MODEL"]
run(VllmEngine(model), model, Path(sys.argv[1]), Path(sys.argv[2]))Temperature zero and a fixed schema make the label repeatable on one pinned engine and model version, and the audit line records both so any label can be traced. The review file is the real output: if a large share of rows lands there each night, fix the prompt or the schema before buying GPUs.
2. When Does Self-Hosting LLM Inference Make Sense For A Trading Firm?
Self-hosting swaps a variable bill for a fixed one plus a team. It pays when the fixed cost is spread across enough steady work, or when something other than cost, usually where the data may go, rules out a hosted API.
- It tends to make sense when data residency or client terms require that confirmations, client communications or research stay on infrastructure you control.
- It tends to make sense when the work is batch-shaped: classifying the day's trade confirmations, tagging the research inbox, extracting fields from broker statements, all of which can run between the US close and the London open.
- It tends to make sense when volume is predictable enough to keep the GPUs busy, and when you need a pinned model and engine version whose outputs can be reproduced months later.
- It tends not to when load is spiky, around session opens or on news days, because you size for the peak and pay for the trough.
- It tends not to when volumes are small, so GPU hours and engineer time are spread across too few jobs, or when nobody on the team has run GPU infrastructure. Drivers, CUDA versions, weight updates, evals and on-call are real work, and DeepInfer alone needs GCC 13 or later, Python 3.12 and CUDA 13.
- It tends not to when the feature needs a model you cannot run yourself, or an interactive latency target your cluster cannot hold at peak.
Vendor relationships also rarely stay single-product. DeepSeek's 17 October release launched a hosted Search & Fetch API alongside the open weights, so the same vendor can be a weights supplier to one team and an API supplier to another. The calculator below makes the cost trade explicit. It contains no prices: GPU hourly cost, ops time and API token rates are inputs from your own quotes and contract, and throughput comes from a benchmark of your prompts on your hardware, with prefill and decode measured separately because the two phases load the GPU differently. It reports cost per 1,000 jobs for each route, the volume at which self-hosting breaks even, and whether each nightly run fits its window.
# Python 3.12, standard library only.
"""Self-host versus hosted API for one workload: monthly cost, cost per 1,000 jobs,
break-even volume and whether each run fits its batch window.
No prices live in this file. Take GPU and API rates from your own quotes and contract,
and throughput from a benchmark of your prompts on your hardware, with prefill and
decode measured separately.
python selfhost_vs_api.py assumptions.json
"""
from __future__ import annotations
import json
import math
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any
@dataclass(frozen=True)
class Workload:
jobs_per_month: int
runs_per_month: int # e.g. one overnight run per business day
input_tokens_per_job: int # instructions + schema + document, counted with the model's tokenizer
output_tokens_per_job: int
batch_window_hours: float # time available for each run
@dataclass(frozen=True)
class SelfHost:
gpu_hourly_cost: float # per GPU, from your cloud or colocation quote
gpus_per_replica: int # what the model needs, with room left for the KV cache
replicas: int
billed_hours_per_month: float # 730 if always on; fewer if you start and stop for each run
prefill_tokens_per_sec: float # per replica, measured on your prompts
decode_tokens_per_sec: float # per replica, summed across the batch, measured
utilisation: float # share of measured throughput you sustain in practice, 0 to 1
ops_hours_per_month: float # upgrades, evals, on-call
ops_hourly_cost: float
@dataclass(frozen=True)
class HostedApi:
input_cost_per_mtok: float # from your contract, per million input tokens
output_cost_per_mtok: float
requests_per_minute: int # your rate limit; concurrency limits may bind first
p95_seconds_per_job: float # measured on your prompts
def self_host(w: Workload, s: SelfHost) -> dict[str, Any]:
# Conservative: treats prefill and decode as sequential. Continuous batching overlaps them.
replica_seconds_per_job = (
w.input_tokens_per_job / s.prefill_tokens_per_sec
+ w.output_tokens_per_job / s.decode_tokens_per_sec
) / s.utilisation
monthly = (s.gpu_hourly_cost * s.gpus_per_replica * s.replicas * s.billed_hours_per_month
+ s.ops_hours_per_month * s.ops_hourly_cost)
busy_hours = w.jobs_per_month * replica_seconds_per_job / 3600 / s.replicas
hours_per_run = busy_hours / w.runs_per_month
return {
"monthly_cost": round(monthly, 2),
"cost_per_1000_jobs": round(monthly / w.jobs_per_month * 1000, 4),
"busy_share_of_billed_hours": round(busy_hours / s.billed_hours_per_month, 3),
"hours_per_run": round(hours_per_run, 2),
"fits_window": hours_per_run <= w.batch_window_hours,
"max_jobs_per_month": math.floor(s.replicas * s.billed_hours_per_month * 3600 / replica_seconds_per_job),
}
def hosted(w: Workload, h: HostedApi) -> tuple[dict[str, Any], float]:
per_job = (w.input_tokens_per_job * h.input_cost_per_mtok
+ w.output_tokens_per_job * h.output_cost_per_mtok) / 1_000_000
jobs_per_run = w.jobs_per_month / w.runs_per_month
hours_per_run = jobs_per_run / (h.requests_per_minute * 60) + h.p95_seconds_per_job / 3600
return {
"monthly_cost": round(per_job * w.jobs_per_month, 2),
"cost_per_1000_jobs": round(per_job * 1000, 4),
"hours_per_run": round(hours_per_run, 2),
"fits_window": hours_per_run <= w.batch_window_hours,
}, per_job
def compare(cfg: dict[str, Any]) -> dict[str, Any]:
w, s, h = Workload(**cfg["workload"]), SelfHost(**cfg["self_host"]), HostedApi(**cfg["hosted_api"])
if not 0 < s.utilisation <= 1:
raise ValueError("utilisation must be greater than 0 and at most 1")
sh = self_host(w, s)
api, per_job = hosted(w, h)
break_even = math.ceil(sh["monthly_cost"] / per_job) if per_job > 0 else None
return {
"self_host": sh,
"hosted_api": api,
"break_even_jobs_per_month": break_even,
"break_even_within_capacity": break_even is not None and break_even <= sh["max_jobs_per_month"],
"lower_cost_at_this_volume": "self_host" if sh["monthly_cost"] < api["monthly_cost"] else "hosted_api",
"assumptions": cfg, # keep the inputs next to the answer
}
if __name__ == "__main__":
print(json.dumps(compare(json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))), indent=2))Read busy_share_of_billed_hours before the cost line. Reserved GPUs are billed whether or not they are working, so a cluster that is busy for a small share of its billed hours has a cost per job set mostly by idle time. If break-even sits well above your forecast volume, the honest answer is a hosted API under a data-processing agreement that meets your residency terms, revisited when volume grows.
3. How Do You Prove Where Every Model In Production Comes From?
The DHS order makes a vendor-risk question concrete. As CyberScoop reported it, the order applies to US federal civilian agencies and to IT products and services from Hangzhou DeepSeek Artificial Intelligence. The reporting does not say whether it reaches open weights run by a third party or tools built on them, and a UK trading firm is not a federal agency. The question is more likely to reach you from a client: a US bank's supplier questionnaire, a contract clause on model provenance, or your own third-party risk review. A Hacker News thread on Reuters' report of the order reached 575 points and 535 comments, so expect to be asked.
The answer is an AI bill of materials (AI-BOM): one record per service listing each model, adapter, engine and hosted API it depends on, with the publishing entity, country, licence, exact version and who hosts it. Record the engine as well as the weights. A service running another lab's model on DeepInfer, or a DeepSeek model on vLLM, has different origins for different components, and a contract clause may be written around the supplier rather than the weights. Which components a clause reaches is a question for counsel; the record should let you answer either way.
The gate below runs in CI before every deploy. For each service and each client it serves, it picks the first approved option, primary first, whose components that client's contract allows, and writes the routing file the runtime reads. An option counts as approved only when a named person has signed off its evaluations, so an untested fallback is never used automatically. A service missing from the AI-BOM, a client with no recorded policy, or a component with no recorded origin or version fails the deploy. The same file answers the urgent question with one command.
# Python 3.12, standard library only. Runs in CI before every deploy.
"""AI bill of materials (AI-BOM) and deploy gate.
For each service and each client it serves, choose the first approved option, primary first,
whose components that client's contract allows. Fail the deploy when none exists.
python aibom_gate.py gate --bom ai-bom.json --policies client-policies.json --deploying confirmations-batch,research-tagger
python aibom_gate.py who --bom ai-bom.json --origin "Hangzhou DeepSeek Artificial Intelligence"
"""
from __future__ import annotations
import argparse
import json
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any
@dataclass(frozen=True)
class Component:
name: str # e.g. "deepseek-ai/DeepSeek-V4.1-Flash-1017", "vllm", "DeepInfer"
kind: str # "model" | "adapter" | "engine" | "hosted_api"
origin: str # publishing legal entity, as named in the licence or terms
origin_country: str # ISO 3166-1 alpha-2
licence: str
version: str # commit, revision hash or API model ID; never "latest"
hosted_by: str # "self" or the provider's legal entity
@dataclass(frozen=True)
class Option:
components: tuple[Component, ...]
approved_by: str | None # named person who signed off this option's evals
approved_on: str | None
@dataclass(frozen=True)
class Service:
name: str
owner: str
clients: tuple[str, ...]
options: tuple[Option, ...] # primary first, then fallbacks in order of preference
@dataclass(frozen=True)
class ClientPolicy:
client: str
clause: str # contract reference, e.g. "MSA s.14.3"
disallowed_origins: frozenset[str]
disallowed_countries: frozenset[str]
self_hosted_only: bool
def load(bom_path: Path, policy_path: Path | None) -> tuple[dict[str, Service], dict[str, ClientPolicy]]:
def option(d: dict[str, Any]) -> Option:
return Option(tuple(Component(**c) for c in d["components"]), d.get("approved_by"), d.get("approved_on"))
bom = json.loads(bom_path.read_text(encoding="utf-8"))
services = {
s["name"]: Service(s["name"], s["owner"], tuple(s["clients"]), tuple(option(o) for o in s["options"]))
for s in bom["services"]
}
policies: dict[str, ClientPolicy] = {}
if policy_path is not None:
for p in json.loads(policy_path.read_text(encoding="utf-8"))["clients"]:
policies[p["client"]] = ClientPolicy(
p["client"], p["clause"], frozenset(p.get("disallowed_origins", [])),
frozenset(p.get("disallowed_countries", [])), bool(p.get("self_hosted_only", False)),
)
return services, policies
def breaches(opt: Option, p: ClientPolicy) -> list[str]:
found: list[str] = [] if opt.approved_by else ["option not approved"]
for c in opt.components:
if not (c.origin and c.origin_country and c.version):
found.append(f"{c.name}: origin or version not recorded")
if c.origin in p.disallowed_origins:
found.append(f"{c.name}: origin '{c.origin}' disallowed by {p.clause}")
if c.origin_country in p.disallowed_countries:
found.append(f"{c.name}: country {c.origin_country} disallowed by {p.clause}")
if p.self_hosted_only and c.hosted_by != "self":
found.append(f"{c.name}: hosted by {c.hosted_by}, but {p.clause} requires self-hosting")
return found
def gate(services: dict[str, Service], policies: dict[str, ClientPolicy],
deploying: list[str]) -> tuple[dict[str, Any], list[str]]:
routing: dict[str, Any] = {}
errors: list[str] = []
for name in deploying:
svc = services.get(name)
if svc is None:
errors.append(f"{name}: not in the AI-BOM. No record, no deploy.")
continue
for client in svc.clients:
policy = policies.get(client)
if policy is None:
errors.append(f"{name}/{client}: no contract policy recorded")
continue
skipped: list[str] = []
for rank, opt in enumerate(svc.options):
found = breaches(opt, policy)
if not found:
routing[f"{name}/{client}"] = {
"option": rank, "fallback": rank > 0, "skipped_because": skipped,
"components": [f"{c.kind}:{c.name}@{c.version}" for c in opt.components],
"approved_by": opt.approved_by, "approved_on": opt.approved_on,
}
break
skipped.extend(f"option {rank}: {b}" for b in found)
else:
errors.append(f"{name}/{client}: no allowed, approved option. " + "; ".join(skipped))
return routing, errors
def who_uses(services: dict[str, Service], origin: str) -> list[str]:
return sorted(
f"{s.name} (owner {s.owner}, option {rank}): {c.kind} {c.name}@{c.version}, "
f"hosted by {c.hosted_by}, clients {', '.join(s.clients)}"
for s in services.values()
for rank, opt in enumerate(s.options)
for c in opt.components
if c.origin == origin
)
def main() -> int:
ap = argparse.ArgumentParser(description="AI-BOM deploy gate")
sub = ap.add_subparsers(dest="cmd", required=True)
g = sub.add_parser("gate")
g.add_argument("--bom", type=Path, required=True)
g.add_argument("--policies", type=Path, required=True)
g.add_argument("--deploying", required=True, help="comma-separated service names from the deploy manifest")
g.add_argument("--out", type=Path, default=Path("build/model-routing.json"))
w = sub.add_parser("who")
w.add_argument("--bom", type=Path, required=True)
w.add_argument("--origin", required=True)
args = ap.parse_args()
if args.cmd == "who":
services, _ = load(args.bom, None)
print("\n".join(who_uses(services, args.origin)) or f"No component from {args.origin}")
return 0
services, policies = load(args.bom, args.policies)
routing, errors = gate(services, policies, [s.strip() for s in args.deploying.split(",") if s.strip()])
for e in errors:
print(f"AI-BOM GATE FAIL {e}", file=sys.stderr)
if errors:
return 1
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(json.dumps(routing, indent=2), encoding="utf-8") # the runtime reads this, not the BOM
for key, r in routing.items():
if r["fallback"]:
print(f"AI-BOM GATE FALLBACK {key}: option {r['option']} because {'; '.join(r['skipped_because'])}")
return 0
if __name__ == "__main__":
sys.exit(main())- Keep ai-bom.json in the repository that holds the deploy manifests, so a model change and its record change in one reviewed pull request.
- Fill the origin field from the licence or terms of service, not from the model's name. A fine-tune has two origins, the base model's publisher and yours, so record the base model and the adapter as separate components.
- When a questionnaire arrives, run the who command and attach its output with the commit hash. It is a better answer than a meeting, and it takes minutes.
“The weights tell you what a model can do. The engine decides what it costs, and the bill of materials decides whether you are allowed to use it.”
Where This Belongs: Brokers, Prop Firms, Startups And Quant Teams
- 01Brokers and CFD providers: start the AI-BOM now and include the models inside vendor tools, such as support chat and CRM assistants, because white-label partners and institutional counterparties ask about suppliers in due diligence.
- 02Prop trading firms: most AI work is interactive and peaks around session opens, which suits a hosted API. Self-host only the steady batch work, such as classifying rule-breach tickets or tagging trader journals overnight.
- 03Fintech and trading startups: run the calculator before the architecture decision, not after, and put the CI gate in place before the first enterprise client sends a supplier questionnaire.
- 04Research and quant teams: research tagging and document extraction are the natural first self-hosted workload. Pin the model and engine versions and turn on batch-invariant or deterministic settings where the engine supports them, so last month's labels can be reproduced.
The Bottom Line
Self-hosting LLM inference became more concrete on 20 October 2026, when DeepSeek published DeepInfer, the engine behind its own production services, three days after releasing DeepSeek V4.1 Flash as open weights. It is not yet a complete service: offline only, one model, two hardware targets and no bundled KV cache system. For a trading firm the decision rests on three tests that do not depend on any one vendor: whether the work is batch-shaped and steady enough to keep GPUs busy, whether data residency rules out a hosted API, and whether you can show a client the origin of every model and engine in production within minutes. AlchmAI, a London trading technology and AI studio, builds that trading AI architecture for brokers, prop firms and fintech teams: the batch pipeline, the cost model and the AI-BOM gate, with a person reviewing anything that reaches a client.
References & Further Reading
- DeepSeek - DeepInfer repository and README (20 October 2026). github.com/deepseek-ai/DeepInfer
- DeepSeek - DeepInfer commit history. github.com/deepseek-ai/DeepInfer/commits/main
- DeepSeek - DeepInfer structured-output example. github.com/deepseek-ai/DeepInfer/blob/main/examples/_structured_output.py
- DeepSeek - DeepSeek V4.1 Flash release news (17 October 2026). api-docs.deepseek.com/news/news261017
- Hugging Face - deepseek-ai/DeepSeek-V4.1-Flash-1017 model card. huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
- CyberScoop - Report on the DHS order barring DeepSeek products and services from federal civilian agencies (19 October 2026). cyberscoop.com/dhs-bans-deepseek-federal-agencies
- Hacker News - Search of discussions on DeepSeek, DeepInfer and the DHS order. hn.algolia.com/?q=deepseek
- vLLM - Structured Outputs documentation. github.com/vllm-project/vllm/blob/main/docs/features/structured_outputs.md
- AlchmAI - Market data infrastructure for research and quant teams. alchmai.com/solutions/market-data-infrastructure
- AlchmAI - AI agents for trading teams. alchmai.com/solutions/ai-agents-for-trading-teams
- AlchmAI - Controls for AI features. alchmai.com/solutions/ai-controls
AlchmAI Engineering
Engineering, London
Written by the AlchmAI engineering team in Mayfair, London. We build trading platforms, real-time charts, market data pipelines and AI features for brokers, prop firms and fintech teams. The Playbook is where we explain how we approach these systems, with code you can run and sources you can check.
Code in this guide is illustrative and supplied without warranty. Review and test it before production use. Nothing here is investment advice. Important information
