Robots.txt é um simples arquivo de texto (assim como o nome sugere) que informa aos robôs rastreadores de mecanismo de pesquisa o que você quer que eles acessem ou não. Por padrão, os robôs tendem a rastrear todas as páginas de seu site, logo, a principal utilização do robots.txt é bloquear páginas que você não queria que esses robôs rastreiem.
Como criar um robots.txt
Por ser um arquivo .txt, criar um robots é bem simples, podendo ser feito até mesmo no bloco de notas. Você precisará saber de alguns comandos, como por exemplo, o nome dos robôs que você queria bloquear. Todos os mecanismo de busca tem o seu próprio robô, como por exemplo o Google que tem o Googlebot. Abaixo veja uma lista dos mecanismo mais famosos:
Como citado anteriormente, por padrão esses mecanismo irão percorrer todo o seu site em busca de informação para indexa-los. Como o maior uso do robots.txt é bloquear páginas que por alguma razão você não quer que seja indexada, o comando Disallow é o mais comum.
Se por exemplo, você gostaria de impedir que o Google acesse uma página ou diretório de seu site com o nome newsletter você adicionará o seguinte comando:
User-agent: Googlebot Disallow: /newsletter
Caso queria incluir todos os robôs de mecanismo de busca para que NÃO acessem alguma pasta/diretório (utilizando o exemplo do newsletter novamente), utilize o * para simbolizá-los. Exemplo:
User-agent: * Disallow: /newsletter
Outro comando interessante é o Sitemap, porém devido ao Google Webmaster Tools já fazer este trabalho e de maneira mais prática e aperfeiçoada, acaba não sendo muito relevante. Todavia, segue o exemplo de comando caso queira introduzir no seu robots.txt:
Pronto! Após as configurações deseja seu robots.txt está pronto para ser utilizado. Agora você precisará ter acesso ao seu servidor e usar um cliente FTP ou gerenciador de arquivos para colocá-lo dentro de seu site. Coloque-o na mesma pasta onde esta o seu site (normalmente localizado no public_html) e deverá estar funcionando corretamente!
Para checar se está tudo correto, você sempre pode digitar a URL do seu site seguido de /robots.txt!
Lei LGPD Brasil 2021
Para melhorar sua experiência usamos Cookies e Política de Privacidade, ao continuar você se declara ciente.
This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
Cookie
Duração
Descrição
cookielawinfo-checkbox-advertisement
1 year
Set by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Advertisement" category .
cookielawinfo-checkbox-advertisement
1 year
Set by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Advertisement" category .
cookielawinfo-checkbox-analytics
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-analytics
1 year
Set by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Analytics" category .
cookielawinfo-checkbox-functional
11 months
The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-necessary
1 year
Set by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Necessary" category .
cookielawinfo-checkbox-others
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-others
1 year
Set by the GDPR Cookie Consent plugin, this cookie is used to store the user consent for cookies in the category "Others".
cookielawinfo-checkbox-performance
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
cookielawinfo-checkbox-performance
1 year
Set by the GDPR Cookie Consent plugin, this cookie is used to store the user consent for cookies in the category "Performance".
elementor
never
This cookie is used by the website's WordPress theme. It allows the website owner to implement or change the website's content in real-time.
elementor
never
This cookie is used by the website's WordPress theme. It allows the website owner to implement or change the website's content in real-time.
viewed_cookie_policy
11 months
The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Cookie
Duração
Descrição
_ga
2 years
The _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_ga
2 years
The _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_ga_VFPSS1J8BL
2 years
This cookie is installed by Google Analytics.
_ga_VFPSS1J8BL
2 years
This cookie is installed by Google Analytics.
_gat_gtag_UA_27849698_1
1 minute
Set by Google to distinguish users.
_gat_gtag_UA_27849698_1
1 minute
Set by Google to distinguish users.
_gid
1 day
Installed by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
_gid
1 day
Installed by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
CONSENT
2 years
YouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
CONSENT
2 years
YouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.