User-agent: * Disallow: *.pdf # Block pdf files. Non-standard but works for major search engines. User-agent: GPTBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: barkrowler Disallow: / User-agent: MJ12bot Disallow: / User-agent: SemrushBot Disallow: / User-agent: crawler_eb_germany_2.0 Disallow: / User-agent: echobot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Yandex Disallow: / User-agent: PetalBot Disallow: /