• CRYPTO-GRAM, November 15, 2025 Part2

    From Sean Rima@21:1/229 to All on Tue Nov 18 14:29:34 2025
    what makes it vulnerable. The security challenges we face today are structural consequences of using AI for everything.

    Insecurities can have far-reaching effects. A single poisoned piece of training data can affect millions of downstream applications. In this environment, security debt accrues like technical debt. AI security has a temporal asymmetry. The temporal disconnect between training and deployment creates unauditable vulnerabilities. Attackers can poison a model?s training data and then deploy an exploit years later. Integrity violations are frozen in the model. Models aren?t aware of previous compromises since each inference starts fresh and is equally vulnerable. AI increasingly maintains state -- in the form of chat history and key-value caches. These states accumulate compromises. Every iteration is potentially malicious, and cache poisoning persists across interactions. Agents compound the risks. Pretrained OODA loops running in one or a dozen AI agents inherit all of these upstream compromises. Model Context Protocol (MCP) and similar systems that allow AI to use tools create their own vulnerabilities that interact with each other. Each tool has its own OODA loop, which nests, interleaves, and races. Tool descriptions become injection vectors. Models can?t verify tool semantics, only syntax. ?Submit SQL query? might mean
    ?exfiltrate database? because an agent can be corrupted in prompts, training data, or tool definitions to do what the attacker wants. The abstraction layer itself can be adversarial.
    For example, an attacker might want AI agents to leak all the secret keys that the AI knows to the attacker, who might have a collector running in bulletproof hosting in a poorly regulated jurisdiction. They could plant coded instructions in easily scraped web content, waiting for the next AI training set to include it. Once that happens, they can activate the behavior through the front door: tricking AI agents (think a lowly chatbot or an analytics engine or a coding bot or anything in between) that are increasingly taking their own actions, in an OODA loop, using untrustworthy input from a third-party user. This compromise persists in the conversation history and cached responses, spreading to multiple future interactions and even to other AI agents. All this requires us to reconsider risks to the agentic AI OODA loop, from top to bottom.

    Observe: The risks include adversarial examples, prompt injection, and sensor spoofing. A sticker fools computer vision, a string fools an LLM. The observation layer lacks authentication and integrity. Orient: The risks include training data poisoning, context manipulation, and semantic backdoors. The model?s worldview -- its orientation -- can be influenced by attackers months before deployment. Encoded behavior activates on trigger phrases.
    Decide: The risks include logic corruption via fine-tuning attacks, reward hacking, and objective misalignment. The decision process itself becomes the payload. Models can be manipulated to trust malicious sources preferentially. Act: The risks include output manipulation, tool confusion, and action hijacking. MCP and similar protocols multiply attack surfaces. Each tool call trusts prior stages implicitly.
    AI gives the old phrase ?inside your adversary?s OODA loop? new meaning. For Boyd?s fighter pilots, it meant that you were operating faster than your adversary, able to act on current data while they were still on the previous iteration. With agentic AI, adversaries aren?t just metaphorically inside; they?re literally providing the observations and manipulating the output. We want adversaries inside our loop because that?s where the data are. AI?s OODA loops must observe untrusted sources to be useful. The competitive advantage, accessing web-scale information, is identical to the attack surface. The speed of your OODA loop is irrelevant when the adversary controls your sensors and actuators.

    Worse, speed can itself be a vulnerability. The faster the loop, the less time for verification. Millisecond decisions result in millisecond compromises.

    The Source of the Problem

    The fundamental problem is that AI must compress reality into model-legible forms. In this setting, adversaries can exploit the compression. They don?t have to attack the territory; they can attack the map. Models lack local contextual knowledge. They process symbols, not meaning. A human sees a suspicious URL; an AI sees valid syntax. And that semantic gap becomes a security gap.

    Prompt injection might be unsolvable in today?s LLMs. LLMs process token sequences, but no mechanism exists to mark token privileges. Every solution proposed introduces new injection vectors: Delimiter? Attackers include delimiters. Instruction hierarchy? Attackers claim priority. Separate models? Double the attack surface. Security requires boundaries, but LLMs dissolve boundaries. More generally, existing mechanisms to improve models won?t help protect against attack. Fine-tuning preserves backdoors. Reinforcement learning with human feedback adds human preferences without removing model biases. Each training phase compounds prior compromises.

    This is Ken Thompson?s ?trusting trust? attack all over again.3 Poisoned states generate poisoned outputs, which poison future states. Try to summarize the conversation history? The summary includes the injection. Clear the cache to remove the poison? Lose all context. Keep the cache for continuity? Keep the contamination. Stateful systems can?t forget attacks, and so memory becomes a liability. Adversaries can craft inputs that corrupt future outputs.

    This is the agentic AI security trilemma. Fast, smart, secure; pick any two. Fast and smart -- you can?t verify your inputs. Smart and secure -- you check everything, slowly, because AI itself can?t be used for this. Secure and fast
    -- you?re stuck with models with intentionally limited capabilities.

    This trilemma isn?t unique to AI. Some autoimmune disorders are examples of molecular mimicry -- when biological recognition systems fail to distinguish self from nonself. The mechanism designed for protection becomes the pathology as T cells attack healthy tissue or fail to attack pathogens and bad cells. AI exhibits the same kind of recognition failure. No digital immunological markers separate trusted instructions from hostile input. The model?s core capability, following instructions in natural language, is inseparable from its vulnerability. Or like oncogenes, the normal function and the malignant behavior share identical machinery.

    Prompt injection is semantic mimicry: adversarial instructions that resemble legitimate prompts, which trigger self-compromise. The immune system can?t add better recognition without rejecting legitimate cells. AI can?t filter malicious prompts without rejecting legitimate instructions. Immune systems can?t verify their own recognition mechanisms, and AI systems can?t verify their own integrity because the verification system uses the same corrupted mechanisms.

    In security, we often assume that foreign/hostile

    --- BBBS/LiR v4.10 Toy-7
    * Origin: TCOB1: https/binkd/telnet binkd.rima.ie (21:1/229)