Google has recently introduced a new bot, Google-CloudVertexBot, to its crawler documentation. This AI-powered crawler is part of Google’s Vertex AI product, designed to serve commercial clients. However, its exact functionality and scope have sparked some confusion due to unclear documentation.
Â
What is Google-CloudVertexBot?
Google-Cloud VertexBot is different from the usual bots associated with Google Search or advertising. Instead, this crawler is linked to Vertex AI, a tool used by businesses for machine learning and AI purposes. According to Google’s Cloud documentation, this bot helps Vertex AI clients ingest website content. The idea is that businesses using Vertex AI can set up data stores containing various types of data, including public website data, which are then used to build AI agents for tasks like searching or making recommendations.
Â
Types of Website Indexing
Google’s documentation mentions two kinds of website indexing:
- Basic Website Indexing:The details about this type of indexing are a bit unclear, and there’s no mention of whether site owner verification is required.
- Advanced Website Indexing: This type requires domain verification and has specific indexing quotas, suggesting that site owners have more control over what gets indexed.
Â
Â
Confusion Around Documentation
The documentation states that Google-CloudVertexBot crawls sites “on the site owners’ request,” implying that it might not crawl public websites without permission. However, the changelog for this new crawler indicates that it was introduced to help site owners recognize traffic from this bot, which has raised questions about whether it might crawl public sites without explicit site owner verification.
Â
Why It Matters
For site owners and SEO professionals, understanding the behavior of crawlers like Google-CloudVertexBot is crucial. If this bot crawls your site without proper verification, it could lead to unexpected traffic patterns or even data being used in ways you didn’t anticipate.
As always, staying informed and keeping an eye on your site’s analytics for any unusual activity from this new crawler is advisable. You might also want to keep your robots.txt file updated to manage how different bots interact with your site. Given the current uncertainties, site owners should remain cautious and proactive.
For those interested in learning more or adjusting their site settings, it’s worth reviewing Google’s official documentation on Google-CloudVertexBot. However, due to the ambiguity in the documentation, site owners should exercise caution and consider blocking this crawler as a precautionary measure if they have concerns about unintentional data scraping.
Read Out Google’s new documentation:
