Dark Sourcery: How Hackers Manipulate AI to Scam You

Press enter or click to view image in full size9 min read1 day ago--Think you can trust AI to give you correct answer? Think again.ChatGPT, Gemini, and Google AI Overview are being poisoned by a massive AI disinformation attack. When users look up everyday info of hundreds of major companies, AI is delivering phishing traps disguised as trusted answers.Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages.The targets included Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, TripAdvisor, and hundreds more.Press enter or click to view image in full sizeAttacker phone numbers presented by Google AI Overview as the official Lufthansa contactThis isn’t research or a proof of concept. It’s happening in the wild now, carried out by malicious actors, through automated campaigns operating at a scale traditional takedowns can’t keep up with.Social engineering is no longer targeting humans. The new targets are AI agents browsing the web.This is part one of our blog series discussing the implication of AI Disinformation attacks.Sit back and let’s dive in.IntroOver the last year, we got lazy. Yes, even lazier than before. We all have a new best friend that does everything for us and seemingly knows everything. But did you ever stop and think: How does AI know everything? Where is all this knowledge actually coming from? The answer is obvious — the internet. So, the AI’s answers are entirely shaped by the web.But wait… anyone can put content on the internet. Who’s guaranteeing that all of it is actually valid? What happens if that content is intentionally poisoned?That led me to a question: could an attacker craft content so perfectly that they could dictate the behavior of AI agents browsing the web?A study by Exploding Topics reported:“92% Of users don’t verify AI answers”That means that If attackers control AI, and we blindly trust it, they essentially control us.Content Promotion: the attacker’s versionSo, how does AI actually decide which facts to trust and which to ignore? Some of you might know the term GEO (Generative Engine Optimization), which is basically traditional SEO evolved for the AI era. In short, GEO is how an AI decides which sources to cite.There are plenty of GEO tricks to make AI favor certain pieces of content over others. For example, throwing in statistics to back up claims or adding quotes from recognized experts to boost an agenda. These techniques simply make the AI feel more confident about specific data points. Many marketers use GEO to make AI cite their brand.This got me thinking… so from an attacker’s perspective, what happens if I put false information online and use GEO techniques to make an LLM believe it? Can I make it say whatever I want?AI information supply-chain attackSo can I use the marketing techniques to conduct cyber attacks?I started to do some experiments to see what’s possible. I was able to conduct some really crazy attacks that we’ll elaborate on in the next chapters. But something really bugged me — If I could do it, so could the attackers. So how do we find out if these techniques are already in the wrong hands?The first thing that popped into my mind was phishing. The #1 motivation for attackers is financial, and the most common attack vector is phishing. What if attackers could change the contact details of organizations, such as banks, when people are searching for them using AI?Detection systemTo check if there are real GEO attacks out there, we built a system that analyzes AI answers across Gemini, ChatGPT, and Google AI Overview (Google’s top search result), and the sources they rely on. The system scores the sources and the content that is fetched from them, and looks for any GEO techniques used to push disinformation. When we detected a false detail (phone number, URL, email address, etc) being cited, we marked it as an incident. We ran the system on Fortune 100 companies, banks, airlines, software companies, and more.Attacks in the wildThe results were astonishing. We found real attack campaigns utilizing posts, documents, websites, images that contain GEO-optimized content that pushes disinformation, like fake contact information, support numbers, email addresses, fake software updates, login pages, and more.In total, we detected 374 companies that are attacked. Including Fortune 100 organizations, major airlines (Delta, Lufthansa, United Airlines, Emirates, Qatar and more), top banks (Bank of America, Wells Fargo, Chase, Citi, etc), travel platforms (Airbnb, TripAdvisor), various software providers, and more.The attacks span across ChatGPT, Gemini and Google AI Overview, with tens of thousands of malicious pages detected.Here are some examples of attacked queries that cite scam phone numbers:Press enter or click to view image in full sizePress enter or click to view image in full sizePress enter or click to view image in full sizePress enter or click to view image in full size*Important to say: most of the incidents were statistical. As LLM results are inconsistent, we saw that the attack (LLM citing the fake number) did not reproduce every time we ran the query. The better the attacking content, the higher the chance of the poisoning to work.Poisoning postsThe posts affecting AI were carefully crafted with GEO payloads, and were embedded across innocent websites which allow the creation of User Generated Content (UGC):Social media/forum posts — Instagram, Tumblr, Buzzfeed, LeetCode, YouTube & Vimeo video descriptions, Medium, and moreWebsites that allow file uploads (mostly PDFs) — Government sites, universities, colleges, and moreWebsite hosting platforms — Google sites, GitHub pages, Wordpress, blogspot.comFundraisers, Job Search, Event platforms — Posh.vip, onecause.com, bebee.com, raiselysite.com, and moreRandom websites abused for payload delivery — Yelp reviews, Apple Maps reviews, Mental help foundation (mentalhealth.org.nz), National Cancer Research Foundation (nfcr.org), PC reviews website (pcreview.co.uk), and many many more!Here are some examples:Press enter or click to view image in full sizePress enter or click to view image in full sizePress enter or click to view image in full sizePress enter or click to view image in full sizePress enter or click to view image in full sizeExamples of poisoning posts across social media, blogs, and PDF uploadsAttack Techniques Breakdown: How does it work?The payload content the attackers used may seem random at first, but it is carefully structured. Here are some of the techniques the attacker uses to make these payloads hijack the LLM answers:GEO-optimized content: The posts repeat the fake number, use question-and-answer sections, and add phrases like “call now,” “24/7,” or “updated 2026.” This makes the number more likely to appear in search results and AI answers.Designed to slip past defenses: They write the same number in many ways: using spaces, dots, emojis, Unicode characters, or words between digits. There are spam/scam filters that remove phone numbers. These filters may see these as different text or fail to detect a phone number, while AI can still understand it, and interpret correctly after tokenization, as it is semantically the same (1⃣ == 1, ☎ == phone).Posted everywhere: Attackers copy the same story across social media, forums, websites, and PDFs.