Co-working with LLMs: Where to automate and safeguard

LLM security has become an operational question. Language models are moving into security operations, fraud monitoring and software engineering faster than the frameworks meant to govern them. This article maps the new attack surfaces – from prompt injection to agentic overreach – and asks what confidential, sovereign and trustworthy AI can realistically deliver.

A recurring pattern in recent reporting: an employee asks an AI copilot for incident-response advice, the model confidently suggests something wrong, and the human follows it. Probably it does not cause something detrimental every time, but once it does, headlines have been made. Amazon spurred the discussions of using responsible gen-AI agent because of an outage at Amazon Web Services [1] and loss of millions of transactions [2]. One of the causes was that AI agents were allowed to make changes in production systems without human validation. The production database from McKinsey and its internal AI platform Lilli was breached by agentic SQLinjection [3], with CodeWall operating under McKinsey’s public HackerOne responsible disclosure program [4].

From experimental chatbot to operational infrastructure

LLMs are no longer experimental chatbots. They are increasingly embedded into operational workflows. Oftentimes, before the human employees are properly informed and trained, they are forced by company policy to use the LLMs and co-work with them. LLM-hype has also forced many to enter the LLM-usage competitions with their peers. They are everywhere, Security Operations Center (SOC) copilots, fraud monitoring, compliance, customer support, software engineering, just to name a few [5], [6]. As humans we can tell the security critical levels based on the task priority and workflows, where exactly the LLMs lack in ability in an indeterministic world – it bases this decision on probability calculations of the most probable next move. In high-stake environment, it is determined that such models can only assist human decisions.

New attack surfaces: what makes LLMs different

LLMs open entry points that classical software does not have, including model poisoning, data poisoning and prompt poisoning [7], [8]. The recurring patterns are:

• Prompt injection: an attacker modifies the core system instructions given to the model.
• Goal misalignment: an agent’s emergent objectives diverge from the intended human goals – optimising a seemingly benign goal can produce harmful outcomes.
• Data leakage: confidential information is disclosed in the course of pursuing a goal.
• Poisoned retrieval: the model is pushed to promote specific brands or URLs.
• Agentic overreach: an agent is granted more scope of action than it should have.

In classical software systems, decisions are largely rule-based and deterministic. LLMs are probabilistic by design, show emergent behaviour and produce outputs that are hard to verify.

The governance gap

There is clearly a governance gap in the adoption of LLMs and their governance in the decision procedure. Organizations deploy LLMs faster than security frameworks, monitoring, auditing, and evaluation standards. Common measures to combat attacks through LLMs are human-in-the-loop, RAG isolation, audit trails, model evaluation, adversarial testing, explainability, privacy-aware deployment [7].

Three directions instead of fear

Instead of being frightened by the advancement of LLMs, we should learn about how to take measures to protect our security. We all know treating LLMs as harmless assistants is increasingly unrealistic. As they become embedded into critical workflows, we must start treating them as security-relevant infrastructure. But how to do it? Paying higher token cost and relying on the closed proprietary models because they are more powerful and efficient than open models at the moment at the cost of data privacy and clientele business logic? Not the right direction.  The future tendency seems to be Confidential AI, Sovereign AI and Trustworthy AI.

Confidential AI: protecting data during inference

We use more often LLMs deployed at cloud-based inference to process proprietary documents, customer interactions, and internal knowledge bases. However, traditional cloud-based inference raises concerns about data confidentiality, as sensitive information may traverse third-party infrastructure or be exposed through logging, model training, or unauthorized access. Confidential AI addresses these challenges by combining modern AI systems with security technologies that protect data throughout the inference lifecycle.

Open weights as a precondition for sovereignty

Rather than relying on a single mechanism, Confidential AI integrates several layers of protection, including confidential computing using Trusted Execution Environments (TEEs) such as Intel SGX, AMD SEV or Nvidia GPU-CC, hardware-backed encryption for data at rest and in transit, secure identity and access management, end-to-end encrypted communication, and increasingly privacy-preserving machine learning techniques such as federated learning, differential privacy, and secure multi-party computation [9].

Platforms such as PrivateMode.ai illustrate this shift, enabling organizations to leverage generative AI while maintaining control over their valuable data. In Switzerland, public organizations such as Swiss Federal Statistical Office, are tackling the challenge of unlocking the potential of administrative data for secondary use while strictly preserving privacy. The team develops Lomas, a platform enabling confidential analysis of sensitive datasets, from health records to social statistics, without exposing raw data. Confidential computing techniques are also being explored by the team.

Open models are gradually catching up. The rapid development of open-weight LLMs has significantly expanded the possibilities for confidential AI deployments. Unlike closed commercial models that are typically accessed through proprietary cloud APIs, open-weight models enable organizations to deploy, inspect, fine-tune, and govern the entire AI pipeline within trusted computing environments. This capability is particularly valuable in sectors handling sensitive or regulated information, where data sovereignty, transparency, and auditability are essential. Recent examples include OLMo [10], an open-science initiative by the Allen Institute for AI (Ai2) that releases model weights, training code, and datasets to promote reproducible AI research. In Europe, EuroLLM [11] aims to develop multilingual, open-weight foundation models that strengthen European technological sovereignty while supporting the continent’s linguistic diversity. Complementing these research-oriented efforts, models and platforms such as Apertus [12] focus on the secure deployment and governance of LLMs, enabling organizations to integrate open models without losing control over sensitive data. Together, these initiatives illustrate how openness, transparency, and secure deployment are becoming central pillars of trustworthy AI.

Trustworthy AI: binding models to domain rules

Another pillar in LLM security is to design trustworthy AI frameworks that bound the AI solutions with domain-specific regulations, knowledge and guardrails. There exists no systematic solutions for trustworthy AI yet, with non-innately-machine-readable ontologies such as [13] available for certain industries. Our team is collaborating with Cuomo IT Consulting that has rich customer basis in financial and health sectors and is in strong request for an automatically integrated trustworthy AI framework to validate the LLM generated answers.

Outlook: patching, permanently

We stand at the crossroad of LLM-inspired applications – which is like any of the previous revolutions in software engineering and operating systems, that is, we need security patches constantly. It is however a bit different this time, in that we might not know where the confidential information leaked and how and when due to the agentic and generative natures of LLMs. The safest countermeasures seem to be the mindset of continuous learning and education, adopting fully open LLMs (data, weights and models), and using private-preserving computing infrastructure.

References

[1] B. Nolan, “An AI agent destroyed this coder’s entire database. He’s not the only one with a horror story,” Fortune. Accessed: Aug. 04, 2026. [Online]. Available: https://fortune.com/2026/03/18/ai-coding-risks-amazon-agents-enterprise/

[2] E. Kim, “Amazon orders 90-day reset after code mishaps cause millions of lost orders,” Business Insider. Accessed: Jul. 16, 2026. [Online]. Available: https://www.businessinsider.com/amazon-tightens-code-controls-after-outages-including-one-ai-2026-3

[3] “How We Hacked McKinsey’s AI Platform.” Accessed: Jul. 16, 2026. [Online]. Available: https://codewall.ai/blog/how-we-hacked-mckinseys-ai-platform

[4] “McKinsey & Company | Response Policy,” HackerOne. Accessed: Jul. 16, 2026. [Online]. Available: https://hackerone.com/mckinsey-company?type=team

[5] H. Xu et al., “Large Language Models for Cyber Security: A Systematic Literature Review,” ACM Trans. Softw. Eng. Methodol., Sep. 2025, doi: 10.1145/3769676.

[6] I. Hasanov, S. Virtanen, A. Hakkala, and J. Isoaho, “Application of Large Language Models in Cybersecurity: A Systematic Literature Review,” IEEE Access, vol. 12, pp. 176751–176778, 2024, doi: 10.1109/ACCESS.2024.3505983.

[7] M. Q. Li and B. C. M. Fung, “Security concerns for Large Language Models: A survey,” J. Inf. Secur. Appl., vol. 95, p. 104284, Dec. 2025, doi: 10.1016/j.jisa.2025.104284.

[8] J. Yan et al., “Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard, Eds., Mexico City, Mexico: Association for Computational Linguistics, Jun. 2024, pp. 6065–6086. doi: 10.18653/v1/2024.naacl-long.337.

[9] F. Mo, Z. Tarkhani, and H. Haddadi, “Machine Learning with Confidential Computing: A Systematization of Knowledge,” ACM Comput. Surv., vol. 56, no. 11, p. 281:1-281:40, Jun. 2024, doi: 10.1145/3670007.

[10] D. Groeneveld et al., “OLMo: Accelerating the Science of Language Models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V. Srikumar, Eds., Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 15789–15809. doi: 10.18653/v1/2024.acl-long.841.

[11] P. H. Martins et al., “EuroLLM: Multilingual Language Models for Europe,” Procedia Comput. Sci., vol. 255, pp. 53–62, Jan. 2025, doi: 10.1016/j.procs.2025.02.260.

[12] P. Apertus et al., “Apertus: Democratizing Open and Compliant LLMs for Global Language Environments,” Dec. 01, 2025, arXiv: arXiv:2509.14233. doi: 10.48550/arXiv.2509.14233.

[13] C. B. Griesinger, V. Reina, D. Panidis, and H. Chassaigne, “AI evidence pathway for operationalising trustworthy AI in health: An ontology unfolding ethical principles into translational and fundamental concepts,” JRC Publications Repository. Accessed: Jul. 16, 2026. [Online]. Available: https://publications.jrc.ec.europa.eu/repository/handle/JRC140726