AdSense Publishers: Welcome Crawlers to Your Blog

As an AdSense Publisher, we know that it is very important to allow AdSense bot or AdSense Pub Crawler to crawl your site. The crawler will crawl your site to allow AdSense displays relevant ads to your site.

Is your site crawler friendly? Could the AdSense Pub Crawler crawl your site and is the crawler successfully crawl your site? You might need to remove roadblock for the crawler to crawl your site properly and improve the ads relevancy on your site.

1. Configuring Robot.txt:
You might be using robot.txt on your site which might be blocking AdSense crawler to crawl your site. Make sure that you are allowing AdSense crawler to crawl your site in your robot.txt, if not the AdSense ads on your site might not display properly.

You can add the following code to your robot.txt to allow AdSense crawler to crawl your site.
User-agent: Mediapartners-Google*
Disallow:

As for Blogger.com users, we will not be able to edit the robot.txt and it will not be a problem for AdSense crawler to crawl our blog. So, if you are using Blogger.com, then you do not need to worry about the robot.txt.

If you have a website that requires members to login to view certain pages and you are placing AdSense Ads on those pages. AdSense crawler might have a problem crawling those pages.

You can allow crawler to access login-protected pages by following the instructions stated here.

If you are not sure whether your site is blocking crawler or not, you can login to your AdSense account and go to "Site Diagnostic". This is the place where crawler will inform you the problem if crawler has problems crawling your site.