User-agent: * Disallow: /wp-content/uploads/2025/05/2025-OWL-K146-Online-Handouts-final.pdf Disallow: /wp-content/uploads/2025/05/2025-OWL-K146-Online-Handouts-final.docx Disallow: /wp-content/uploads/2025/05/2025-K146-Online-Manual-final-v2.pdf Disallow: /wp-content/uploads/2025/01/owl_training_roster-August-2023ROSTER.csv Disallow: /wp-content/uploads/2025/03/2025_Elementary_In-Person_Training_Manual_final31.pdf Disallow: /wp-content/uploads/2025/03/2025_Elementary_In-Person_Welcome_Letter_final1.docx Disallow: /wp-content/uploads/2025/04/2025-OWL-JrSr-High-In-Person-manual-final-pdf.pdf Disallow: /wp-content/uploads/2025/04/2025-JrSr-Welcome-Letter-final.docx Disallow: /wp-content/uploads/2024/07/2024-adult-levels-in-person-owl-training-manual.pdf Disallow: /wp-content/uploads/2025/03/2025_Elementary_In-Person_Trainer_Handouts_final21.pdf Disallow: /wp-content/uploads/2025/03/Elementary_In-Person_Participant_Handouts_final21.pdf Disallow: /wp-content/uploads/2025/04/2025-JrSr-Trainer-Handouts-final-pdf.pdf Disallow: /wp-content/uploads/2025/04/2025-JrSr-Participant-Handouts.pdf Disallow: /wp-content/uploads/2024/07/2024-adult-levels-owl-trainer-handouts.pdf Disallow: /wp-content/uploads/2024/07/2024-adult-levels-owl-participant-handouts.pdf Disallow: /wp-content/uploads/2024/07/2024-OWL-Online-Elementary-Facilitator-Training-Manual-final-update.pdf Disallow: /wp-content/uploads/2025/04/2025-OWL-JrSr-Online-Training-Manual-final-pdf.pdf Disallow: /wp-content/uploads/2024/07/2024-Elementary-OWL-Trainer-Handouts.pdf Disallow: /wp-content/uploads/2024/07/2024-Elementary-OWL-Participant-Handouts.pdf Disallow: /wp-content/uploads/2025/04/2025-JrSr-Online-Participant-Handouts-final-pdf.pdf Disallow: /wp-content/uploads/2025/01/OWL-Certificate-fillable-pdf.pdf Disallow: /wp-content/uploads/2025/01/OWL-posters-FINAL-2024.pdf Disallow: /wp-content/uploads/2025/01/Sept-2023-planning-guide-rev-1.pdf Disallow: /wp-content/uploads/2025/04/Trainer-Letter-of-Engagement-Template.docx Disallow: /wp-content/uploads/2025/04/Online-Training-Assignments-Tracking-Sheet.xlsx # --- Crawl-trap mitigation (added 2026-10-12, traffic investigation) --- # The Events Calendar renders a page for ANY date/pagination value, creating an # unbounded URL space (observed: 87,645 distinct eventDate= values incl. year 0000, # 32,205 distinct tribe-bar-date= values). Amazonbot crawled it continuously, # accounting for ~40% of all origin requests and defeating the CDN cache. Disallow: /*eventDate= Disallow: /*eventDisplay= Disallow: /*tribe-bar-date= Disallow: /*paged= Disallow: /events/list/page/ Disallow: /wp-login.php # --- Site search (added 2026-11-01) --- # Bingbot crawled ~900 search-result URLs a day (odd multi-word queries); a few # ran SearchWP relevance queries for 2-3 minutes after the page timed out. # Search results are not useful in an index. Disallow: /?s= Disallow: /*?s= Disallow: /*&s= Disallow: /search/ # --- Meta model-training crawler (added 2026-11-03) --- # UCC opted out of model-training use of its content. Meta says this crawler # honours a group that names it; link previews (facebookexternalhit) are unaffected. User-agent: meta-externalagent Disallow: / # --- Commercial SEO / brand-monitoring crawlers (added 2026-11-03) --- # QlyzeBot fetched ~10,700 pages on 2026-11-02 (church directory, devotionals); # AwarioBot has crawled since 2026-10-19 with many 404s. Neither serves UCC. User-agent: QlyzeBot Disallow: / User-agent: AwarioBot Disallow: / # --- Self-declared research scrapers (added 2026-11-12) --- # Assay (7.3k requests 11-07..08, old news slugs and feeds), HymnfoldResearch and # church-directory-research (church directory pages, up to ~1.1k requests an hour). User-agent: Assay Disallow: / User-agent: HymnfoldResearch Disallow: / User-agent: church-directory-research Disallow: / # --- Model-training, answer-engine indexers and SEO-data crawlers (added 2026-11-06) --- # Live logs Oct 23 - Nov 6, average per day: Amazonbot 12.7k requests (~2 h of PHP # time), ExaSearchBot 2.6k, SemrushBot 2.5k, YisouSpider 1.7k, ElevenlabsCrawler 1.1k, # plus model-training and smaller SEO-data crawlers. None of them sends visitors to # UCC. Regular search engines and user-triggered fetchers keep their access. # Same opt-out as the Meta group above. User-agent: Amazonbot User-agent: ExaSearchBot User-agent: YisouSpider User-agent: ElevenlabsCrawler User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Bytespider User-agent: Google-Extended User-agent: Applebot-Extended User-agent: SemrushBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: DotBot User-agent: DataForSeoBot User-agent: Barkrowler Disallow: /