There are several people that using WordPress and looking to be featured on first page of Google. In fact most of the users websites even not showing up on Google or any search engine. If that;s really happening with you then I have a quick solution for you to allow search engines to index your website.
You have to go to WordPress dashboard Setting -> Reading
After that you will see a “Discourage search engines from indexing this site” option.
Make sure that it will be not checked. If it is checked then unchecked it and click save button if you want search engines to crawl your website.
This is an easy way to prevent search engines from crawling your website.
WordPress 5.3 Will Fix That For You
The new WordPress version changes the way to prevent search engines from crawling website. WordPress 5.3 version drop the robots.txt method and come up with an updated robots meta tag to keep your site getting listed in search engines.
<meta name='robots' content='noindex,nofollow' />This is a new meta tag and of course a more precise way to keep your site from getting indexed or crawled.
Also the recent problem in the robots.txt is well-summarized by Jono Alderson.
