80.8% Of Trackers Survived 'Reject All' And Chat Titles Went To Meta And TikTok: What The Conversational-AI Privacy Study Means For Any Bank Shipping An LLM Assistant

The paper's title - 'Prompt like a butterfly, sting like a tracker' - undersells how ordinary its findings are. The researchers instrumented nine popular conversational AI clients on the web and on Android and watched the network. What they saw was the standard adtech stack, transplanted into the most sensitive interface most people now use: trackers that survive 'reject all' at a rate of 80.8%; conversation-derived artefacts - titles, prompts, screenshots - disclosed to third parties alongside persistent identifiers that let those parties attribute them to a person; conversation permalinks, or the identifiers needed to reconstruct them, sent from five web clients to nine tracking organisations; conversation titles from three clients going to Meta, TikTok and DoubleClick; and some providers exposing public conversation links with no access control, so a tracker holding the link could read the whole exchange. Paid tiers behaved almost identically to free ones.

None of the audited products is a bank's. But the mechanism that produced the leaks is precisely the one a bank inherits when it drops an LLM assistant into an existing web app: the tag manager, the analytics SDKs, the session-replay tool, the marketing pixels, the auto-generated conversation title in the page heading and URL. A customer asking 'can I afford to take my pension early' generates a title, a URL, and a screen full of text that every script on the page can read. That is a UK GDPR special-category-adjacent data flow, a Consumer Duty problem and, if the assistant is inside authenticated banking, a security incident waiting for a disclosure.

1. A Content Security Policy That Ends The Question

The assistant should live on its own origin or route with a strict CSP that allows scripts only from your own hosts and connections only to your API. No tag manager, no analytics SDK, no session replay. If the marketing team needs metrics, they get them server-side from your telemetry, not from a pixel in the chat.

// Served on the assistant route only. Nonce per response; no third-party origins at all.
export function assistantCsp(nonce: string): string {
  return [
    "default-src 'none'",
    "script-src 'self' 'nonce-" + nonce + "' 'strict-dynamic'",
    "connect-src 'self' https://assistant-api.yourbank.co.uk",
    "img-src 'self' data:",
    "style-src 'self' 'nonce-" + nonce + "'",
    "font-src 'self'",
    "frame-ancestors 'none'",
    "base-uri 'none'",
    "form-action 'self'",
    "report-uri /csp-report",           // violations = someone added a tag; page the owner
  ].join("; ");
}

// Also: Referrer-Policy: no-referrer  (a conversation URL must never travel in a Referer header)
//       Permissions-Policy: interest-cohort=(), browsing-topics=()

2. Telemetry That Can Only Say What You Allowed

You still need product analytics: how many conversations, how often the assistant abstained, how long answers took. Build the event schema as an allow-list and reject anything else at the collector. The event must be able to carry counts and categories; it must be unable to carry text.

const ALLOWED_EVENTS = {
  conversation_started: { channel: ["web", "ios", "android"] },
  answer_rendered: { grounded: ["true", "false"], abstained: ["true", "false"], latencyBucketMs: ["<500", "<1000", "<2000", ">=2000"] },
  feedback: { rating: ["up", "down"] },
  handoff_to_human: { reason: ["abstained", "requested", "policy"] },
} as const;

type EventName = keyof typeof ALLOWED_EVENTS;

export function validateEvent(name: string, props: Record<string, string>): { ok: boolean; reason?: string } {
  const schema = ALLOWED_EVENTS[name as EventName];
  if (!schema) return { ok: false, reason: "unknown event " + name };
  for (const [k, v] of Object.entries(props)) {
    const allowed = (schema as Record<string, readonly string[]>)[k];
    if (!allowed) return { ok: false, reason: "field not allowed: " + k };       // no free-text fields exist
    if (!allowed.includes(v)) return { ok: false, reason: "value not allowed for " + k };
  }
  return { ok: true };
}

// Collector: drop and alert on any invalid event. Identifier is a per-session random id,
// never the customer id and never a device/advertising identifier.

3. Nothing Conversational In URLs, Titles Or Logs

• Conversation IDs are opaque random tokens; the URL is /assistant/c/<id>, never a slug derived from the question.
• The document title is static ('Assistant') on the client. Auto-generated conversation names live in your database and render inside the DOM only.
• Server logs record the conversation id and event, never the prompt. Prompts are stored encrypted under a separate key with a retention policy the DPIA actually states.
• Redact before anything leaves the trust boundary - including the model provider if your contract does not cover the data class.

import re

PATTERNS = {
    "uk_sort_account": re.compile(r"[0-9]{2}-?[0-9]{2}-?[0-9]{2}[ ]?[0-9]{8}"),
    "card_pan":        re.compile(r"(?:[0-9][ -]?){13,19}"),
    "ni_number":       re.compile(r"[A-CEGHJ-PR-TW-Z]{2}[0-9]{6}[A-D]", re.I),
    "email":           re.compile(r"[A-Z0-9._%+-]+@[A-Z0-9.-]+[.][A-Z]{2,}", re.I),
    "uk_phone":        re.compile(r"(?:[+]44|0)7[0-9]{3}[ ]?[0-9]{6}"),
    "postcode":        re.compile(r"[A-Z]{1,2}[0-9][A-Z0-9]?[ ]?[0-9][A-Z]{2}", re.I),
}

def redact(text: str) -> tuple:
    """Return redacted text and the set of categories found. Deterministic, before storage or model call."""
    found = set()
    for name, rx in PATTERNS.items():
        if rx.search(text):
            found.add(name)
            text = rx.sub("[" + name.upper() + "]", text)
    return text, found

4. Share Links That Require A Login

The study's most alarming finding was public conversation permalinks readable by anyone holding the URL. A financial assistant should not have public share links at all. If a customer needs to share a conversation - with an adviser, say - the link resolves only for authenticated users the owner named, expires, and is logged.

export async function createShare(convId: string, owner: User, recipients: string[], ttlHours = 72) {
  const token = crypto.randomUUID();
  await db.shares.insert({ token, convId, ownerId: owner.id, recipients, expiresAt: Date.now() + ttlHours * 3600_000 });
  audit.log(owner.id, "share_created", { convId, recipients: recipients.length });
  return "/assistant/shared/" + token;      // meaningless without login + being a named recipient
}

export async function resolveShare(token: string, viewer: User | null) {
  if (!viewer) throw new Unauthorized();     // never render for anonymous viewers
  const s = await db.shares.get(token);
  if (!s || s.expiresAt < Date.now()) throw new NotFound();
  if (s.ownerId !== viewer.id && !s.recipients.includes(viewer.id)) throw new Forbidden();
  audit.log(viewer.id, "share_viewed", { convId: s.convId });
  return db.conversations.get(s.convId);
}

5. Prove It: The No-Third-Party Test

Finally, the test the study effectively ran against nine products, run against yours in CI: load the assistant, hold a conversation containing a canary string, and assert that no request left the page to any host outside your allow-list and that the canary never appeared in any URL, title or outbound request body.

import { test, expect } from "@playwright/test";

const ALLOWED_HOSTS = new Set(["assistant.yourbank.co.uk", "assistant-api.yourbank.co.uk"]);
const CANARY = "canary-pension-4471";

test("assistant sends nothing to third parties and leaks nothing conversational", async ({ page }) => {
  const outbound: { host: string; url: string; body: string }[] = [];
  page.on("request", (r) => outbound.push({ host: new URL(r.url()).host, url: r.url(), body: r.postData() ?? "" }));

  await page.goto("https://assistant.yourbank.co.uk/assistant");
  await page.getByRole("textbox").fill("Can I take my pension early? " + CANARY);
  await page.keyboard.press("Enter");
  await page.getByText(/./).first().waitFor();

  const thirdParty = outbound.filter((r) => !ALLOWED_HOSTS.has(r.host));
  expect(thirdParty, "requests to non-allow-listed hosts").toEqual([]);
  expect(outbound.filter((r) => r.url.includes(CANARY)), "canary in a URL").toEqual([]);
  expect(await page.title()).toBe("Assistant");
  expect(page.url()).not.toMatch(/pension/i);
});

“Nobody at those nine companies decided to send pension questions to an ad network. They just built the chat on the same page as the pixels. Do not build the chat on the same page as the pixels.”

The Bottom Line

The IMDEA study documents what happens when conversational AI is built on the ordinary web: 80.8% of trackers surviving 'reject all', conversation links sent to nine tracking organisations, titles to Meta, TikTok and DoubleClick, public permalinks with no access control, and no privacy premium for paying. A financial firm shipping an LLM assistant inherits the same mechanism unless it designs against it: a strict CSP on a dedicated route with no third-party scripts, a telemetry schema that cannot carry text, opaque IDs and static titles, redaction before storage or model calls, share links that require login and expire, and a CI test that fails if a single byte leaves to a host you did not name. That is how we build customer-facing AI for regulated firms in London, and this week's study is the clearest argument yet that the default web stack is not fit for the conversation.

References & Further Reading

• IMDEA Networks et al. - Prompt like a butterfly, sting like a tracker: a privacy analysis of web and mobile conversational AI agents (PDF). jorgegarciaherrero.com/wp-content/interactivos/20260916-Prompt-like-a-butterfly-sting-like-a-tracker-(clean).pdf
• Hacker News - A privacy analysis of web and mobile conversational AI agents (discussion). news.ycombinator.com/item?id=49890226
• DEV Community - Your AI chatbot is talking to advertisers behind your back: what a new tracking study found. dev.to/jamilxt/your-ai-chatbot-is-talking-to-advertisers-behind-your-back-what-a-new-tracking-study-found-1e2d
• MDN - Content Security Policy. developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP
• ICO - Guidance on AI and data protection. ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection
• FCA - Consumer Duty. fca.org.uk/firms/consumer-duty
• Playwright - network interception. playwright.dev/docs/network

Code in this guide is illustrative and supplied without warranty. Review and test it before production use. Nothing here is investment advice. Important information