Visual detection improves phishing website identification with yolo v8
A Computer Vision Approach to Visual Fraud Detection in Phishing Websites Using YOLOv8
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Phishing websites try to trick people by looking like real sites to steal passwords or credit card numbers. Instead of checking website addresses or code, the authors used a computer vision method that looks at website screenshots just like a person would. Their system learned to tell real sites from fake ones with 92% accuracy by focusing on logos, colors, and layout. They also made the system better at avoiding false alarms by training it on a variety of images. This helps add a new layer of protection alongside existing detection methods.
What this means in practice
- •For security operations teams: Integrate visual phishing detection to quickly identify fraudulent websites based on their appearance, providing timely alerts beyond URL analysis.
- •For web browser developers: Implement fast image-based checks in browsers to warn users when a website visually mimics a trusted brand, enhancing phishing protection.
Authors
Basil Sajid Shaikh, Hajar Homayouni
Abstract
Phishing remains one of the most common vectors for financial and identity fraud, and most detection systems still rely on inspecting a page's URL, HTML markup, or domain registration history. These signals are easy for an attacker to rotate or obfuscate, and they say very little about what actually convinces a victim to hand over a password or a card number: the way the page looks. This paper describes a visual, image-based approach to phishing detection that treats a rendered webpage the same way a human eye would, as a picture that either matches a trusted brand or doesn't. A YOLOv8 convolutional neural network was trained to classify full-page website screenshots as phishing or legitimate based on layout, logo placement, color scheme, and login-form structure, rather than on text extracted from the page. The system reached 92% classification accuracy on a held-out test set, processed a single screenshot in roughly 100 milliseconds, and, after a round of data augmentation aimed specifically at lighting, compression, and scaling variation, cut the false-positive rate by 11% relative to the pre-augmentation baseline. The paper walks through the dataset construction, the augmentation strategy, the model architecture and training setup, and the resulting performance, and closes with a discussion of where this kind of visual detector fits alongside, rather than instead of, existing URL- and content-based defenses.