Defending Against AI-Enabled Phishing and Deepfakes

In February 2024, an employee at engineering firm Arup attended a video conference call with what appeared to be the company's CFO and several colleagues. The meeting felt routine. The CFO gave instructions. The employee transferred $25 million.

None of the people on that call were real. The CFO, the colleagues — all AI-generated deepfakes in a live video call. By the time Arup confirmed the fraud, the money was gone.

This is not a story about a gullible employee. It is a story about an attack vector that existing security controls were not designed to stop — and that is rapidly becoming affordable for criminal organizations at all scales.

The Attack Has Changed Fundamentally

Traditional phishing is detectable by its characteristics: unfamiliar sender addresses, generic salutations, urgency pressure, grammatical errors. Security awareness training has been calibrated to these signals for two decades.

AI-enabled attacks eliminate these signals:

Hyper-personalized spear phishing. LLMs can ingest a target's LinkedIn profile, public social media, company website, press releases, and any available leaked data — and produce a phishing email that references specific projects, uses accurate names and titles, and matches the writing style of the purported sender. There are no grammar errors. There is no generic salutation. The email reads exactly as a real colleague would write it.

Voice cloning. With a few minutes of audio training data — available from YouTube videos, podcasts, or voicemails — adversaries can clone an executive's voice and use it in phone calls. The voice is indistinguishable to the human ear. "Vishing" attacks using cloned executive voices are increasingly reported in financial fraud cases.

Live video deepfakes. The Arup attack demonstrated that deepfake technology is now good enough to sustain a live video conversation without detection by untrained observers. The visual artifacts that characterized early deepfakes — unnatural blinking, edge distortion, lighting inconsistencies — are substantially reduced in current generation tools.

Why Awareness Training Alone Is No Longer Sufficient

Security awareness training remains valuable. But it cannot be the primary defense against attacks that produce artifacts indistinguishable from legitimate communication.

You cannot train employees to spot deepfakes with current detection technology any more than you could train them to detect that a document is forged by visual inspection alone. The defense layer has to exist at the process and control level — not exclusively at the human judgment level.

This is a process redesign problem as much as a technology problem.

Process Controls That Stop AI Fraud

Out-of-Band Verification for High-Stakes Requests

Any financial instruction — wire transfer, vendor payment change, payroll modification — should require verification through a channel that is independent of the original communication.

If the request came via email: call a known number (not a number provided in the email). If the request came via phone: respond via email to a pre-established address. If the request came via video call: confirm via a documented back-channel before executing.

The Arup attack succeeded because the victim trusted the video call as sufficient verification. An out-of-band verification requirement — "I will confirm this with your assistant via the internal ticketing system" — would have broken the attack.

Dual-Approval Controls for Transactions Above Threshold

Wire transfers, large purchases, and payment vendor changes above a defined threshold should require two independent approvers, neither of whom can be the same person who received the original instruction. This is a standard internal control that many mid-market organizations skip.

The control does not need to be burdensome. For transactions below $10,000, standard authorization may be sufficient. Above that threshold, the second approver requirement adds friction that is easily absorbed into workflow — and that would have stopped the Arup fraud entirely.

Executive Communication Protocols

Establish documented protocols for how executives communicate requests that require action. Common elements:

• Wire transfer requests from executives always come through the ticketing or workflow system, never via email or phone alone
• Executives will never request that normal approval processes be bypassed due to urgency
• Any communication that claims to override standard process should be treated as a red flag requiring immediate escalation

Publish these protocols internally. Include them in onboarding. Reference them in phishing awareness training. When employees know what the real CFO's communication patterns look like, they can identify anomalies.

Technical Controls for AI-Enabled Phishing

Email Authentication at the Gateway

SPF, DKIM, and DMARC are table-stakes controls that block spoofed sender addresses. Organizations that have not fully deployed and enforced DMARC (reject mode) are still vulnerable to the simplest class of sender impersonation — the kind that does not even require AI.

Advanced email security platforms now include AI-powered behavioral analysis that scores inbound emails against sender behavioral baselines. An email from a domain that typically sends low-volume transactional messages should get additional scrutiny if it suddenly sends an urgent wire transfer instruction.

Deepfake Detection Tools

Commercial deepfake detection tools have improved, though they remain in an ongoing arms race with synthesis technology. Tools like Microsoft's Video Authenticator and several third-party vendors offer real-time analysis of video and audio streams for synthetic generation artifacts.

These tools are most useful in asynchronous contexts — analyzing a video recording or audio clip. Live deepfake detection in real time is a harder problem and is not yet reliable enough to serve as a primary control.

Endpoint and Email Security That Catches LLM-Generated Content

Modern email security platforms are beginning to integrate AI-generated content detection. This helps with phishing volume, though motivated attackers running targeted spear phishing campaigns against specific high-value targets will invest in evading automated detection.

For high-value targets — executives, finance staff, M&A teams — the process controls above are more reliable than technical detection alone.

Building a Realistic Defense Program

The organizations that are best positioned against AI-enabled fraud are not necessarily the ones with the most advanced technology. They are the ones with:

1. Clear, documented, enforced authorization procedures for financial transactions
2. Out-of-band verification requirements built into workflow — not just trained
3. A culture where employees are empowered to pause and verify without fear of being seen as obstructive
4. Regular tabletop exercises that include deepfake and voice clone scenarios

The $25M Arup loss is a preview. The attack methodology is now sufficiently affordable and accessible that mid-market organizations are viable targets. The question is not whether to build these defenses, but how quickly.

If you want to assess your organization's exposure to AI-enabled fraud and build practical defenses, Talk to JP Stratton.