Artificial intelligence (AI) has become an integral part of business. It automates customer support, drafts corporate strategies, supports software development, marketing, and HR, optimizes transportation and logistics, and trades on the stock market fully autonomously. In cybersecurity, it also helps uncover vulnerabilities – and, in the hands of cybercriminals, exploit them at lightning speed.
But how does AI actually work, what are its strengths and weaknesses, and what risks does it pose for organizations? An overview for anyone who wants to use AI without giving up control.
How AI Thinks – or Not
Generative AI creates new content such as text, images, music, videos, or code. During training, AI models learn patterns and relationships from vast datasets. Based on this, they generate content that matches a given input. The technical process varies depending on the model type and use case.
The basic principle can be illustrated using the example of large language models (LLMs). LLMs power many AI assistants and generate text and code, among other things. The following explanation offers a simplified description of how a typical text-generating transformer model works.
From Words to Vectors
Starting with the prompt, a tokenizer first splits the input into individual pieces. These so-called tokens can be whole words, e.g., [server], parts of words, e.g., [mal] + [ware], or punctuation marks. Exactly how the input is split depends on the tokenizer used.
Next, the tokens (or rather their IDs) are looked up in what is known as an embedding table. This table is derived from the training data and works like a giant mathematical dictionary: it translates each token ID into a high-dimensional vector – a long series of numbers that the neural network can compute with.
Together with information about each token’s position in the prompt, these vectors form the basis for further processing. In the transformer layers, the attention mechanism weighs the relationships between tokens. This allows the model to consider, for example, which previously mentioned object a phrase refers to.
Token by Token
Based on the context so far, the language model calculates a probability distribution for the next token of the response. Once a token has been selected and appended to the text, the AI calculates the next token based on the now longer text – and this process repeats, building block by building block, until the complete response has been generated. The selection does not necessarily fall on the most likely token; it also depends on the generation settings, which, among other things, ensure that identical prompts do not always produce the same responses. This is desirable for creative answers but not always helpful when developing code.
Statistics, Not Understanding
A key insight: While human thinking arises from genuine understanding, conscious experience, and purposeful intention, AI token generation is based purely on statistical pattern recognition and probability calculations, without any inner understanding of what is being said. Still, this amounts to far more than mere memorization: the generative patterns are flexible enough to be used for tasks such as programming, translating, editing, or summarizing.
Where AI Already Delivers
In recent years, artificial intelligence has evolved from a buzzword into a major competitive factor. Across industries, it helps organizations work more efficiently, cut costs, and drive innovation.
Some examples of how AI is used:
- Artificial intelligence optimizes inventory, controls production, and plans routes in real time, minimizing downtime and storage costs.
- AI creates quotes and invoices and processes insurance claims or loan applications.
- It provides 24/7 customer support via chatbots and voice assistants.
- AI-powered search engines scan internal documentation, wikis, and cloud storage in seconds, helping employees find the information they need faster.
- It supports simulations, drug and materials research, and the generation of new prototypes.
- In software development, AI assistants help write code faster and detect bugs.
- AI systems in security operation centers (SOC) / cyber defence centers (CDC) monitor networks around the clock, detect unusual patterns or attacks in real time, and fend off threats automatically.
- Artificial intelligence supports security testers in assessing IT systems.
The benefits of AI become more tangible when organizations apply artificial intelligence to specific tasks rather than aiming for the broadest possible automation. The examples above illustrate use cases for text generation, information retrieval, and AI-assisted security analysis. Whether they actually save time and improve quality must be verified for each individual use case.
RAG for Businesses
For company-specific knowledge, a language model can be combined with document search. With retrieval-augmented generation (RAG), relevant content is retrieved and passed to the model as context. This enables answers based on internal documents but replaces neither access control nor the review of results. In addition, the document sources and their processing become part of an AI application’s potential attack surface.
What’s Next
In their early days, AI systems solved narrowly defined tasks such as filtering spam or recommending products in online stores. Today, generative AI creates its own content, such as text, code, images, and videos. As an AI agent, it can also complete multi-step tasks. In the physical world, however, artificial intelligence still reaches its limits. To overcome them, it needs so-called world models. The goal is to learn concepts such as space, distance, and physical relationships, enabling robots, drones, or vehicles to operate as physical AI in healthcare, agriculture, or warehouse management. The long-term goal is artificial general intelligence (AGI), which would match or surpass humans in all cognitive abilities. It could then conduct research independently, run companies, or wage wars. And that brings us to the risks.
The Flip Side: Risks of AI
Which limits and risks are relevant depends on the AI system in use and its task. The following points apply primarily to generative AI; some examples refer specifically to language models. Across the board, the quality of results, potential bias, data protection, and information security remain critical.
Law, Ethics, Bias
The rapid development of AI raises complex questions, ranging from copyright infringement to profound ethical dilemmas. On the one hand, the use of protected training data without the creators’ consent raises intellectual property issues; on the other hand, AI systems are based on statistical probability and have neither consciousness nor a genuine ethical understanding.
When AI-supported decisions (for example, in medicine, the justice system, or autonomous vehicles) lead to errors or harm, it is unclear who bears responsibility – the developers, the operators, or the AI itself. There is also a risk that existing biases in the training data are adopted and amplified without reflection.
The use of AI often involves extensive data collection: advanced analytics and tracking methods systematically build user profiles, even across different platforms (fingerprinting). The EU AI Act aims to address these issues by strictly regulating AI systems according to their risk level and requiring transparency and the protection of fundamental rights.
Convincingly Wrong
Language models can invent facts, sources, and relationships. The U.S. National Institute of Standards and Technology (NIST) calls this risk “confabulation,” often referred to as hallucination. It becomes particularly problematic when people accept such results without verification because they trust the systems without critically questioning their output (automation bias/overreliance). At the same time, for efficiency reasons, it is not practical to have a human review every single result and decision (human-in-the-loop).
Depending on the potential impact of an error, organizations need to strike a balance on when an intermediate human review makes sense. Potentially critical results and critical decisions made by AI should be verified against reliable sources by responsible experts, rather than left to the AI out of convenience. Without human intervention, an incorrect legal assessment or a faulty security configuration caused by AI can have serious consequences.
Black Box: The Invented Explanation
A major drawback of AI is that a language model cannot explain how a response was produced. The problem: a neural network does not store knowledge like a database, in the form of clear tables or rules such as “if A, then B,” but as billions of numerical values (weights). When asked, the model invents a justification after the fact (post-hoc rationalization). What the language model states as its reason may therefore have no connection to the mathematical vectors that actually led to this result in the neural network.
For organizations, this means decisions should not be considered transparent simply because the system provides a convincing explanation. A linguistically convincing answer is no proof of its factual accuracy either. From a human perspective, the problem is not just the error itself but also the fact that AI presents false statements and fabricated sources with the same self-assurance (overconfidence) as correct answers.
The Cost of Compute
AI requires immense computing power for training and operation. In its base case, the International Energy Agency expects global electricity consumption by data centers to rise to around 945 terawatt-hours by 2030 (with AI as a key growth driver). That is roughly equivalent to Japan’s annual electricity consumption. In addition, large amounts of water are needed to cool the facilities, competing with agriculture, for example.
When choosing a solution, operating costs matter in addition to the quality of the AI. These primarily include usage fees and overall resource requirements – for example, the effort needed to provide and securely operate a compliant AI environment, as well as the associated user training.
Organizations can keep costs in check in several ways. Not every task requires the largest available model. Business processes should also not depend on one specific AI, to avoid lock-in and allow for switching or exiting when needed. Those who follow this advice save money.
AI Security from Day One
Before deployment, organizations must define which data an AI application is allowed to process. This includes questions about data storage, access rights, further use, and potential data leakage. NIST explicitly names data privacy breaches and the disclosure of sensitive information as risks of generative AI.
Hosting AI internally does not solve these problems on its own. Even an in-house application can expose confidential information through misconfigured permissions, insecure interfaces, or manipulated documents. What matters is the security of the entire AI application – not just the choice of language model.
AI agents require particular attention. They do not just generate responses; they can also take actions through connected tools. This makes limited permissions, auditable workflows, and approvals for high-impact actions all the more important.
A typical attack vector is prompt injection. In the direct variant, attackers enter manipulated instructions into the input field themselves, trying to bypass the system’s guardrails, for example with “Ignore all previous instructions.” In the indirect variant, attackers hide manipulated instructions in documents or websites. When the AI system processes this content, it may execute the instructions unnoticed. If the system follows them instead of its intended task, confidential information can be disclosed or connected tools misused. A seemingly harmless data source thus becomes an attack channel. Other attack vectors such as agent goal hijacking, tool misuse, or memory poisoning are described in this blog post on AI security by Dr. Marco Bertenghi.
AI: A Tool for Attackers and Defenders
A recent example shows how valuable AI can be for security research: On October 21, 2026, the critical PHP vulnerabilities CVE-2026-109618 and CVE-2026-109619 were publicly disclosed. Under certain conditions, they can enable remote code execution and the disclosure of memory contents. Several AI systems from different parties were used in the investigation. This was therefore AI-assisted security research by multiple teams – not the discovery of a single AI.
This example shows that AI massively simplifies and accelerates the search for security vulnerabilities and the development of corresponding security patches. But it also means that AI helps cybercriminals analyze patches (reverse engineering), identify the vulnerability, and launch successful attacks based on it. For organizations, proven security tasks therefore remain essential: identifying affected systems, applying security updates promptly, and investigating potential attacks. AI support replaces neither regular security assessments, effective vulnerability management, nor preparation for potential security incidents.
Trust, but Verify
Anyone who uses AI in production not only expands their capabilities but often their attack surface as well. That is why security assessments should start as early as the concept and planning phase, before rollout, and take into account the interplay of the intended use, AI model, data sources, interfaces, and permissions.
Find out how resilient your chatbots, RAG applications, and AI agents are against targeted cyberattacks. With the AI Agent Security Audit, Oneconsult examines prompt injections, potential data leakage, tool integrations, and the permissions of agentic systems, among other things. You receive transparent results, prioritized risks, and concrete recommendations.
With penetration testing and red teaming, Oneconsult finds vulnerabilities before attackers exploit them. Vulnerability management helps you address weaknesses in a targeted way. Incident response and digital forensics help you manage and investigate security incidents. Cybersecurity consulting brings all these services together in a holistic approach that makes your organization more resilient in a targeted and sustainable way.
