In this episode of BTW with BTM, we discussed a major feature change to the popular e-commerce platform, Shopify, and that’s the ability to edit a site’s robots.txt file. Confirmed by Shopify’s CEO, it was announced this week that the feature would be rolled out to all users. SEO’s rejoice as this is a feature many in the community have been asking for years to have access to.
What is a robots.txt file and why is it important?
For those not familiar with technical SEO and backend site structures, let’s take a look at what exactly a robots.txt file is and why it’s important to us.
A robots.txt file is a text file that webmasters create to instruct crawlers and bots (search engine robots) what and how to crawl pages or sections of a website. Think of it as a journey map or guide for search engines.
The reason robots.txt files are important is because it allows you to set rules in place on what pages and segments of the site you would or would not like to be crawled. This is especially important for content or pages with time sensitive content, or pages that are for select types of campaigns that you do not care to have indexed, or segments you do not care to have crawl budget wasted on.
In addition to what you’re allowing or disallowing to be crawled on a site, what is being outlined in a robots.txt file can also establish parameters on what you want to be crawled when. This priority can be vital to collections within a platform such as Shopify, or similar CMS structures.
What is an example of when you would want to edit a robots.txt file?
Since we’re dealing with Shopify and the subject of e-commerce, consider influencer marketing and the growth this has had in the space. Many times, major brands are working with various influencers who have pages and collections of products that are setup for exclusive deals or time sensitive promotions. This type of content is something your team might want to keep a tight rein on verses letting the crawlers loose on the content that is specific to one influencer.
Additional use cases that are very common and well documented include:
- Allowing or disallowing certain URLs from being crawled
- Adding crawl-delay rules for various crawlers and bots
- Adding in extra sitemap URLs – such as image sitemaps, etc
- Blocking certain crawlers
How do you access the robots.txt file in Shopify?
The robots.txt file in Shopify can be edited through the robots.txt.liquid theme template. This is the template that actually renders the robots.txt file for your Shopify based site. Additional tutorials and documentation have been built out nicely by the Shopify team, and those can act as your guide.
It’s strongly suggested that if you are not familiar with editing robots.txt files or understand the ramifications that can come from this, you work closely with your qualified SEO team, or web dev team.
For additional tips and insights on how you can improve the relevancy of your site online, please consider subscribing to our podcasts and monthly newsletter in the form below.
