1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

Tip on indexing and robots.txt

Discussion in 'Link Building' started by krawling1, Feb 7, 2019.

  1. krawling1

    krawling1 Junior Member

    Joined:
    Oct 17, 2018
    Messages:
    112
    Likes Received:
    183
    Gender:
    Male
    Hi BHW.
    I've seen a lot of shares for creating profiles or other pages where it is possible to put a link. I want you to know, that a lot of them are blocked by robots.txt and would never be indexed even if you go to church and pray for that.

    Two tips on how to check whether a page is closed or no:
    1) look at the source page. Then press CTRL+F and put this into the field: "content". If a webmaster doesn't want this page to be seen you'll see "content = noindex, nofollow". In this case for Google it sounds like a recommendation. He probably would miss this page but if there are pages linking to this page he would crawl it.
    2) Go to https://search.google.com/test/mobile-friendly and put there a link to a page with your link
    If it is blocked by robots.txt you will see an error message.

    Be careful while you placing links. You can do a worthless business.


    For Example phpbb.com. You can create a profile on a VERY HIGH DA forum, you put a link and feel happy, but if you run your profile page through a mobile-friendly test you'll realize that it is blocked.
     
    • Thanks Thanks x 3
  2. gregz06

    gregz06 Regular Member

    Joined:
    Oct 30, 2018
    Messages:
    222
    Likes Received:
    120
    Gender:
    Male
    Occupation:
    SEO, design, tennis
    Location:
    Helsinki
    I believe robots.txt is a recommendation which can be easily neglected by crawlers, especially if some links are pointing to the page. Though, good advice for those waiting months for their profile links to get indexed.
     
  3. krawling1

    krawling1 Junior Member

    Joined:
    Oct 17, 2018
    Messages:
    112
    Likes Received:
    183
    Gender:
    Male
    I believe too. But there are some cases when the page can be totally closed at least for simple indexing links which people want to use for rankings.
     
  4. meandmyjetpack

    meandmyjetpack Registered Member

    Joined:
    Mar 1, 2019
    Messages:
    61
    Likes Received:
    6
    I tried method 1 on https://www.phpbb.com/community/mem...u=159755&sid=e7911ba39fd0ee2a0fd767598e5e4532

    Why doesn't the source page show the "content = noindex, nofollow" tag?

    ty
     
  5. krawling1

    krawling1 Junior Member

    Joined:
    Oct 17, 2018
    Messages:
    112
    Likes Received:
    183
    Gender:
    Male
    • Thanks Thanks x 1
  6. shaggy99

    shaggy99 Newbie

    Joined:
    Jan 12, 2014
    Messages:
    43
    Likes Received:
    2
    I have some profile backlinks that I suspect are no index.
    I searched in their page source, but I didn't found the noindex tag.
    Also, after making the mobile friendly test, I have some page resources that can't be loaded, and I have a few statuses "Googlebot blocked by https://ads.pubmatic.com/robots.txt" . But this kind of error I also have on pages that I'm sure are indexed.
    Where I should look more to be sure the pages are no index?