The Thousands of Automated Friends we Make Along the Way: Battling Bots while Still Getting Sleep

• Type of Material: Presentation
• Authors: Beth Crompton, Emory University
• Language: English
• Date: 2025-06-11
• Publisher: Emory University Libraries
• Publication Version: Final Published Version
• Copyright Statement: Beth Crompton

Abstract

In today's digital landscape, libraries are experiencing a problematic increase in automated traffic. This includes AI crawlers, indexing bots, and university-mandated vulnerability scans at a pace that exceeds our ability to respond effectively. While often well-intentioned, these activities put immense strain on our search layer and digital repositories--critical resources that must remain operational 24/7 for students, researchers, and faculty. However, our support staff, while dedicated, are not network engineers. When faced with traffic spikes or disruptive behavior, the response burden often fails on teams without deep technical expertise, leading to service disruptions, frustrations, and burnout. To address this challenge, we have developed and documented a solution within AWS WAF that offers an easy way to diagnose and block poorly behaved automated traffic. This enables our less technical on-call support staff to swiftly implement the necessary protections with minimal impact on their work/life balance. This session will provide an overview of the issues institutions like ours face, explore the broader implications of unchecked automated traffic, and offer a practical, adaptable solution that others can implement to safeguard their digital infrastructure without requiring a dedicated network security team.