Navigation

    APPDRAG Community

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Popular

    REMINDER

    Please be respectful of all AppDragers! Keep it really civil so that we can make the AppDrag community of builders as embracing, positive and inspiring as possible.

    UNSOLVED Unreachable: robots.txt

    Domains & Emails
    2
    6
    509
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      S Santiago last edited by

      I'm receiving the following error when trying to index my URL:

      Failed: Robots.txt unreachable

      Per Google:
      *Unreachable: robots.txt
      Before we crawled the pages of your site, we tried to check your robots.txt file to ensure we didn't crawl any pages that you had roboted out. However, your server returned a 5xx (unreachable) error when we tried to retrieve your robots.txt file. To make sure we didn't crawl any pages listed in that file, we postponed our crawl.

      Your hosting provider may be blocking Googlebot, or there may be a problem with the configuration of their firewall.

      Note: If the contents of the robots.txt file is different in your browser than what Google sees, work with your hosting company to remove any server rules that might show different robots.txt content to different user agents*.

      My robots.txt file is good and haven't had any issues until now.

      Any idea on why this is happening?
      thanks,

      Joseph Benguira 1 Reply Last reply Reply Quote 0
      • Joseph Benguira
        Joseph Benguira @S Santiago last edited by

        @s-santiago could you please share the url of your site so I can check that for you?

        S 1 Reply Last reply Reply Quote 0
        • S
          S Santiago @Joseph Benguira last edited by

          @joseph-benguira our url is appagreens.com

          Joseph Benguira 1 Reply Last reply Reply Quote 0
          • Joseph Benguira
            Joseph Benguira @S Santiago last edited by

            @s-santiago I can see the robots.txt is reachable:
            https://www.appagreens.com/robots.txt

            maybe it was a temporary error, please retry

            S 1 Reply Last reply Reply Quote 0
            • S
              S Santiago @Joseph Benguira last edited by

              @joseph-benguira Still having the same issue as of today (Nov. 8th). Any chance Googlebot might be blocked in someway on the hosting side or have some type of firewall configuration problem?

              The problem is that robots.txt is accessible, just not by googlebot for some reason

              Joseph Benguira 1 Reply Last reply Reply Quote 0
              • Joseph Benguira
                Joseph Benguira @S Santiago last edited by

                @s-santiago really strange, I never heard about similar issue before.
                Googlebot is not blocked by appdrag of course

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post