Custom URL Inspection Tool Built Using Python on Google Search Console API

For Enterprise SEO Directors and CTOs, the margin for error is non-existent. When managing websites with hundreds of thousands of pages, data visibility isn't just a metric; it is the foundation of operational integrity. Yet, a significant "visibility gap" exists in how we monitor indexation at scale.

The industry is currently facing an inflection point. As websites grow more complex and reliant on JavaScript, the standard tools for monitoring indexation, specifically the Google Search Console (GSC) interface, are becoming insufficient for enterprise needs.

This playbook outlines a strategic framework for bypassing these limitations. We will move beyond off-the-shelf SaaS solutions to build a custom, "serverless" infrastructure. By leveraging the Google Search Console URL Inspection API and agentic workflows, we can operationalise indexation data without increasing server load or recurring costs.

The Visibility Crisis: Why the GSC Interface Isn't Enough

The Google Search Console interface is useful. But it has a ceiling that makes it structurally inadequate for enterprise SEO management.The hard limit is 1,000 rows. That's the maximum you can export from any standard GSC report. For a site with 20,000 URLs, that means 95% of your inventory is invisible in any given export. You're monitoring the top slice and hoping the rest is fine.The consequence isn't just incomplete data. It's misplaced confidence. You see a healthy indexation rate in your export, you report it upward, and meanwhile a category of transactional pages has quietly dropped out of the index because they fell below the 1,000-row cut-off.This is the visibility gap. And the only way to close it is to move beyond the interface entirely.

The API Paradox & Server Load Mitigation

Google recognised this visibility gap and released the Web Inspection API (also known as the URL Inspection Tool API), allowing developers to programmatically pull index status for up to 2,000 URLs per day per property.

Recognising this opportunity, popular third-party tools like the Screaming Frog API integration implemented solutions to retrieve this data. However, for the enterprise CTO, relying solely on third-party crawlers introduces a secondary set of challenges regarding infrastructure governance.

The Hidden Cost of "Active" Crawling

When you use a traditional crawler to query the Google SEO API, the tool typically crawls your site first to gather the URLs, and then queries Google. This creates a double-tax on your infrastructure:

1. The API Limit: You are still capped at 2,000 daily requests.
2. The Server Burden: Every time the crawler runs, it hits your server. If you are checking 2,000 URLs, that is 2,000 additional requests hitting your origin server daily.

Context: The Bot Traffic Surge

In the current digital landscape, server capacity is a crucial resource. We must be mindful not to overburden our infrastructure, specifically given the rise of AI bots.

Consider a standard 2,000-page enterprise site. Daily logs reveal a massive volume of crawl requests from disparate bots including Amazon, Google, ChatGPT, Bing, and Apple.

• The Strategic Question: In an environment where your server is already handling thousands of requests from AI agents, should we voluntarily add another 2,000 requests to the mix just to check indexation?.

For a resilient architecture, the answer is no. We need a solution that decouples data retrieval from server interaction.

The Solution: A "Serverless" Architecture

The solution lies in shifting the paradigm from "crawling the site" to "querying the database". We can architect a URL inspection tool workflow that interacts directly with Google's data, having zero interaction with your server.

Key Strategic Benefits

1. Zero SaaS Premium: This tool runs on its own using free infrastructure, eliminating monthly subscription fees associated with enterprise SEO tools.
2. Absolute Data Integrity: There is no ambiguity. If the API returns "not indexed", it is a definitive fact from Google's own database.
3. Historical Logging: By storing errors in Google Sheets, you create an audit trail to track if specific issues (like soft 404s or canonical errors) are resolving over time.

Use Case: Managing a 50,000-Page Site With a 2,000 URL Daily Limit?

This is the question that stops most SEOs from building something like this. The URL Inspection API allows up to 2,000 requests per day, per property. For a site with 50,000 URLs, that sounds like a hard ceiling. It isn't.The 2,000-per-day quota is not a limitation on what you can know. It's a pacing mechanism. And when you pair it with a crawler that runs automatically every day and tracks exactly which URLs were last checked and when, it becomes a continuous rolling window of fresh data across your entire inventory.Here is how the math works in practice.

At 2,000 URLs per day, you cover 14,000 URLs per week. A 50,000-page site gets a complete refresh every 25 days. More importantly, the tool prioritises the oldest data first so the URLs that haven't been checked in the longest time are always at the front of the queue. No URL sits unmonitored indefinitely.

For most enterprise sites, the pages that matter most transactional pages, category pages, high-revenue URLs are a fraction of the total inventory. The tool lets you segment your URL list by priority, so your most important pages are checked first and most frequently, while the long tail is covered in the background.

The result is a site of any size, managed continuously, with no manual intervention and no gaps in coverage.

How to Build It

The tool is a Python script. It requires no web server, no database, and no paid infrastructure. The only external dependency is a Google Cloud service account with access to the Search Console API.

What you need before you start:

A Google Cloud project with the Search Console API enabled. A service account created within that project, with a JSON key file downloaded. The service account email address added as a user in Google Search Console for the property you want to inspect. A CSV file containing the URLs you want to monitor.

The project structure:

web-inspection/
├── crawler.py
├── credentials.json        ← your service account key file
└── clients/
    └── client_name/
        ├── config.json     ← contains the site_url for this property
        ├── urls/
        │   └── urls.csv    ← your URL list
        └── output/
            ├── master_results.csv
            ├── error_log.csv
            └── daily_summary.csv

The config.json for each client is minimal:

{
  "site_url": "https://www.yoursite.com/"
}

The core logic:

The script authenticates using the service account credentials, reads the URL list from the CSV, checks which URLs have already been inspected within the last seven days, and processes the remainder one at a time. After each URL is processed, the result is written to disk immediately not held in memory and saved at the end. If the script is interrupted for any reason, the data already collected is safe, and the next run picks up exactly where the previous one stopped.

The API call itself is straightforward:

response = service.urlInspection( ).index().inspect(
    body={
        "inspectionUrl": url,
        "siteUrl": site_url,
        "languageCode": "en-US"
    }
).execute()

The response contains the verdict (PASS, NEUTRAL, FAIL), the coverage state, the last crawl date, and the canonical URLs as Google has recorded them. All of this goes into master_results.csv.

The re-check logic:

Any URL not inspected within seven days is automatically added back to the pending queue on the next run. This means the tool maintains a continuous rolling window of fresh data without any manual intervention. You don't need to manage which URLs need refreshing the script handles it.

Multiple clients:

If you're managing multiple properties, each client gets its own folder with its own URL list and its own output files. When the crawler runs, it processes all clients concurrently one independent process per property so the total run time is determined by the slowest single client, not the sum of all of them.

How to Deploy It

The simplest deployment is a scheduled task that runs once a day. No server required. The machine running it can be a Windows VM, a Mac, or a Linux box anything with Python installed and a stable internet connection.

On Windows (Task Scheduler):

Open Task Scheduler and create a new basic task. Set the trigger to Daily at a consistent time early morning works well, before the working day starts. Set the action to run the Python executable inside the virtual environment directly:

Program: C:\<fakepath>\Projects\web-inspection\
venv\Scripts\python.exe
Arguments: crawler.py
Start in: C:\<fakepath>\Projects\web-inspection

Pointing to the venv Python executable directly means the scheduled task uses the correct environment with all dependencies installed, without needing to activate the virtual environment manually.

The dashboard:

The output CSVs feed a Streamlit dashboard that visualises indexation trends, failure breakdowns by coverage state, and canonical conflicts across the property. Start it once with streamlit run app.py and it stays live, reading from the same files the crawler writes to.

streamlit run app.py

The dashboard is available at http://localhost:8501 by default. For teams that need it accessible to multiple people, it can be deployed to Streamlit Community Cloud or any server that supports Python the CSVs become the shared data layer.

Build vs. Buy

The case for building this rather than paying for a SaaS tool comes down to three things.

Data ownership:

The output lives in CSV files on your own machine. You control the retention, the format, and the access. There's no vendor lock-in, no API deprecation risk, and no subscription that disappears if a company shuts down.

Zero server interaction:

The tool queries Google's API directly. It never touches your site. For teams managing infrastructure carefully particularly those already dealing with elevated bot traffic this is a meaningful operational advantage.

Absolute data integrity:

When the API returns not indexed, that's a definitive statement from Google's own database. There's no ambiguity about whether the crawler was blocked, whether JavaScript rendered correctly, or whether the result reflects a cached state. It's what Google knows, at the time you asked.

The tool won't replace every use case for a traditional crawler. There are things a site crawl can tell you redirect chains, broken internal links, page speed data that the URL Inspection API doesn't surface. But for the specific question of whether Google has indexed a URL and what it thinks of its canonical, there is no more direct source of truth than the API itself.

That's the tool this is. Nothing more, nothing less.