Robots.txt, if present, is the first thing that a search engine bot looks up when visiting your website. It is used by webmasters to provide instructions to the search engine bots visiting a site. If your robots.txt file carried instructions to discourage search engine crawling, your site may not get indexed. You can check your site\u2019s robots file by adding \/robots.txt<\/em> at the end of your domain name.<\/p>\n If you see the above instruction in your robots.txt file, it means that the search engine bots are discouraged from crawling your site\u2019s content. You can fix this issue easily by removing the \u2018\/\u2019 from this command. You can use the robots.txt file effectively to de-index specific folders in your site. Google\u2019s very own robots file<\/a> has covered an extensive list of commands. Check it out for your reference.<\/p>\n Meta Tags<\/b><\/span><\/p>\n <\/b>It is a common practice by webmasters to use meta \u2018robots\u2019 tag to prevent search engine bots from crawling a particular web page. If a particular page in your website doesn\u2019t get indexed, check its source code for this meta tag:<\/p>\n This tag will prevent the web page from being crawled by the search engine bots.\u00a0Some WordPress themes incorporate this meta tag by default. If your WP site has indexing problems, it is worth checking the theme header for this tag. Once this tag is removed from the source code, search engine bots can crawl the particular web page.<\/p>\n Sitemap Errors<\/b><\/span><\/p>\n A sitemap (XML or HTML) provides a search engine robot with the list of URLs to be crawled in your website. If for some reasons the sitemap in your website is not automatically updated, search engine bots may some time to discover those webpages and this would cause delay in indexing.<\/p>\n You need to check out the Webmaster tool (set up a Google Webmaster account for your site if you don\u2019t already have it) for issues related to the sitemap. The Sitemap Details page will inform you about the errors in it. Once the errors are fixed, you can generate a new xml sitemap<\/a> for your site and upload it in the root for your website.<\/p>\n Other Crawling Errors<\/b><\/span><\/p>\n Once you\u2019ve ruled out the above mentioned problems, head to your Google Webmaster Tools dashboard and check for crawling errors. The crawling errors need to be fixed as and when you see them. Some of the page crawling errors might be related URL parameter<\/a> issues, which mostly happen in the case of dynamic links. You can fix errors like this by using a simple 301 redirect.<\/p>\n Getting Indexed Under a Different Domain Name<\/b><\/span><\/p>\n Before coming to a conclusion that your website is not indexed, check out for other variations in your domain name that might have been indexed by search engines. For instance, http:\/\/www.yoursite.com and http:\/\/yoursite.com might look similar to you. However, they are two different URLs according to the search engines.<\/p>\n If you website is indexed in a different name than the one you preferred, you need to specify your preferred domain name in the Webmaster tools and use a 301 redirect<\/a>. You can also use canonical link element<\/a> for setting the most preferred URL version but a 301 redirect is more effective.<\/p>\n No Incoming Links<\/b><\/span><\/p>\n This issue is common in websites that are new to the web. If your website doesn\u2019t have any incoming links from other websites, it might take a while for Google to find your site. If you wish to get your site indexed soon, you may have to build some backlinks manually. Backlinks from high PR websites will provide more link weightage to your website thereby helping it get indexed soon.<\/p>\n Submitting your site to Google Plus will earn you an incoming link and help your site get discovered by the search engine bots. You can also submit your site URL to Google<\/a> through your Webmasters tool. These methods will significantly improve the chances of getting indexed by Google.<\/p>\n AJAX Applications<\/b><\/span><\/p>\n If your website runs AJAX application, it would contain AJAX URLs (a URL with # fragment) and the content of these URLs can\u2019t be crawled by search engine bots. This, in turn, would prevent the search engines from indexing your website. There is a separate process<\/a> to help search engine bots crawl AJAX content. The solution provided by Google Developers will help you fix this issue.<\/p>\n Bad Links & Google Reconsideration Request<\/b><\/span><\/p>\n It is important for a site owner to constantly assess the quality of incoming links. The inbound links to your website should not violate Google\u2019s guidelines<\/a> on link quality. You can detect these bad links through your Google Webmaster Tools. If your site has too many bad links, it will be de-indexed by Google and other search engines.<\/p>\n If by any chance your website has been de-indexed due to this issue, you need to disavow<\/a> these bad links completely and make sure that your site no longer violates the guidelines. Once this is fixed, you need to submit a reconsideration request to Google through your Webmaster tools.<\/p>\n SEO Spamming<\/b><\/span><\/p>\n There are many black hat SEO techniques practiced by spammers to get higher rankings instantly. Google constantly keeps tab of its search results quality and takes manual action<\/a> on sites that employ spamming techniques. Shortcut methods like cloaking, keyword stuffing, content scraping and spam links etc. will get your site permanently de-indexed.<\/p>\n The simple solution here is to clean up your site of all the spamming techniques and submit a reconsideration request<\/a> through your Webmaster tools.<\/p>\n Malware Ridden Websites<\/b><\/span><\/p>\n Websites infected with malwares may get disappeared from search results very soon. The silver lining here is that you can find the issue before your site gets de-indexed. Google Webmaster Tools will issue a warning that your site has been compromised. Once you\u2019ve discovered the issue, you can remove the malware and enhance your site\u2019s security. You can also submit a reconsideration request if your site has already been de-indexed.<\/p>\n Website Downtime<\/b><\/span><\/p>\n If your website is continuously down for a significant period (say a few weeks) of time, it might get de-indexed from the search engines. The only solution here is to fix the issue causing downtime and get your site back online. Once your site is back online, it may take a while to get your traffic back to its normal status. If your site isn\u2019t indexed even after fixing the issue, you may have to submit the reconsideration request in the Webmaster Tools.<\/p>\n![\"Robots](\"https:\/\/www.niyati.com\/blog\/wp-content\/uploads\/2013\/09\/robots-tag.jpg\" "\\"Robots")
<\/p>\n![\"meta](\"https:\/\/www.niyati.com\/blog\/wp-content\/uploads\/2013\/09\/meta-no-follow-tag.jpg\" "\\"meta")
<\/p>\n