Google Search Console URL Inspection Script

This script automatically inspects your new blog posts using Google Search Console’s URL Inspection API. It provides detailed indexing status and helps identify potential issues with your posts. Note: Google has deprecated the programmatic indexing request API, so this script focuses on inspection and monitoring.

Features

• 🔍 Multiple Detection Modes: Find new posts by recent changes, git commits, sitemap, or inspect specific URLs
• 🗺️ Sitemap Support: Automatically fetch and inspect all URLs from your sitemap
• 🚫 Find Unindexed Pages: Identify pages that are not in Google’s index
• 📊 CSV Export: Export inspection results to CSV for further analysis
• 🚀 Automatic URL Generation: Converts Jekyll post files to proper URLs based on your site structure
• ⚡ Rate Limited: Respects Google’s API rate limits with configurable delays
• 🔐 Secure Authentication: Uses OAuth2 for secure API access
• 📋 Detailed Inspection: Provides indexing status, coverage state, and fetch results
• 📝 Comprehensive Logging: Detailed logging and error handling
• 🧪 Dry Run Mode: Test what would be inspected without actually making API calls

Prerequisites

1. Google Cloud Console Setup

1. Go to the Google Cloud Console
2. Create a new project or select an existing one
3. Enable the Google Search Console API:
   • Navigate to “APIs & Services” > “Library”
   • Search for “Google Search Console API”
   • Click on it and press “Enable”

2. Create Credentials

1. Go to “APIs & Services” > “Credentials”
2. Click “Create Credentials” > “OAuth 2.0 Client ID”
3. If prompted, configure the OAuth consent screen first:
   • Choose “External” user type (unless you have a Google Workspace)
   • Fill in required fields (App name, User support email, Developer contact)
   • Add your email to test users if in testing mode
4. Choose “Desktop Application” as the application type
5. Name it something like “Blog GSC Submitter”
6. Add Authorized Redirect URIs (click “ADD URI” for each):
   • http://localhost:8080/
   • http://localhost:8081/
   • http://localhost:9090/
   • http://localhost/ (for fallback)
7. Download the JSON file and save it as credentials.json in this directory

3. Verify Your Website

Make sure your website is verified in Google Search Console.

Installation

1. Install Python Dependencies:

    pip install -r requirements.txt
   

2. Place Credentials:

   • Download your OAuth2 credentials from Google Cloud Console
   • Save the file as credentials.json in this directory

3. Configure the Script (optional):

   • The script will create a default config.json on first run
   • Modify the configuration as needed for your site

Usage

Quick Start: Find Unindexed Pages

The easiest way to find unindexed pages is to use the convenience script:

./find_unindexed.sh

This will automatically:

1. Fetch all URLs from your sitemap
2. Check their indexing status with Google Search Console
3. Filter for unindexed pages only
4. Export results to a CSV file with timestamp

You can also:

# Specify a custom CSV filename
./find_unindexed.sh my_unindexed_pages.csv

# Export all pages (not just unindexed)
./find_unindexed.sh --all

Basic Usage

Submit posts modified in the last 7 days:

python submit_posts.py

Submit posts modified in the last 3 days:

python submit_posts.py --mode recent --days 3

Git-Based Detection

Submit posts changed since the last commit:

python submit_posts.py --mode git

Submit posts changed since 5 commits ago:

python submit_posts.py --mode git --since HEAD~5

Submit posts changed since a specific commit:

python submit_posts.py --mode git --since abc1234

Submit Specific URL

python submit_posts.py --mode url --url "https://thinhdanggroup.github.io/your-post-title/"

Find Unindexed Pages from Sitemap

Inspect all URLs from your sitemap and find unindexed pages:

python submit_posts.py --mode sitemap --filter-unindexed

Export unindexed pages to CSV:

python submit_posts.py --mode sitemap --filter-unindexed --csv-filename unindexed_pages.csv

Inspect all URLs from sitemap (no filtering):

python submit_posts.py --mode sitemap --export-csv

Use a custom sitemap URL:

python submit_posts.py --mode sitemap --sitemap-url "https://example.com/custom-sitemap.xml" --filter-unindexed

Dry Run Mode

Test what would be submitted without actually submitting:

python submit_posts.py --dry-run

Verbose Logging

Enable detailed logging:

python submit_posts.py --verbose

Understanding Unindexed Pages

The script can identify various types of unindexed pages:

Unindexed States

• Discovered - currently not indexed: Google found the page but hasn’t indexed it yet
• Crawled - currently not indexed: Google crawled the page but chose not to index it
• URL is unknown to Google: The URL hasn’t been discovered by Google
• Page with redirect: The page redirects to another URL
• Duplicate content: Google detected duplicate content
• Not found (404): The page returns a 404 error
• Soft 404: The page returns 200 but appears to be a 404 page
• Blocked by robots.txt: The page is blocked by your robots.txt file
• Server error (5xx): The page returned a server error

Filter Options

• --filter-unindexed: Shows all pages that are NOT indexed (including errors)
• --filter-not-submitted: Shows only pages that are indexable (Verdict: PASS) but not submitted/indexed

CSV Export

The exported CSV includes these fields:

• url: The page URL
• success: Whether the inspection succeeded
• coverage_state: Current indexing state
• verdict: Overall verdict (PASS, FAIL, etc.)
• fetch_state: How Google fetched the page
• last_crawl_time: When Google last crawled the page
• indexing_allowed: Whether indexing is allowed
• user_canonical: Your specified canonical URL
• google_canonical: Google’s chosen canonical URL
• error: Any error messages

Configuration

The script uses a config.json file for configuration. Here are the available options:

{
    "site_url": "https://thinhdanggroup.github.io/",
    "base_url": "https://thinhdanggroup.github.io",
    "posts_pattern": "*.md",
    "excluded_files": ["README.md", "index.md"],
    "rate_limit_delay": 1.0,
    "max_urls_per_batch": 10
}

• site_url: Your site URL as registered in Google Search Console
• base_url: Base URL for generating post URLs
• posts_pattern: Glob pattern to match post files
• excluded_files: Files to exclude from submission
• rate_limit_delay: Delay between API requests (seconds)
• max_urls_per_batch: Maximum URLs to submit in one batch

Authentication Flow

On first run, the script will:

1. Open your web browser
2. Ask you to sign in to your Google account
3. Request permission to access Search Console
4. Save authentication tokens for future use

The tokens are saved in token.json and will be automatically refreshed as needed.

Examples

Find and Export All Unindexed Pages

1. Check which pages are unindexed:

python submit_posts.py --mode sitemap --filter-unindexed

1. Export to CSV for analysis:

python submit_posts.py --mode sitemap --filter-unindexed --csv-filename unindexed_$(date +%Y%m%d).csv

1. Inspect a specific unindexed URL for more details:

python submit_posts.py --mode url --url "https://thinhdanggroup.github.io/specific-post/" --export-csv

Check Recently Updated Posts

# Check posts updated in last 7 days
python submit_posts.py --mode recent --days 7 --export-csv

# Check only unindexed recent posts
python submit_posts.py --mode recent --days 7 --filter-unindexed --export-csv

Daily Automation

Add to your crontab to run daily:

# Submit new posts daily at 9 AM
0 9 * * * cd /path/to/script && python submit_posts.py --mode recent --days 1

CI/CD Integration

Add to your GitHub Actions workflow:

- name: Submit new posts to Google Search Console
  run: |
      cd script/google_search_console
      python submit_posts.py --mode git --since $
  env:
      GOOGLE_APPLICATION_CREDENTIALS: $

Post-Deployment Hook

Run after deploying new content:

# In your deployment script
python script/google_search_console/submit_posts.py --mode recent --days 1

Troubleshooting

Common Issues

1. “Credentials file not found”

   • Make sure credentials.json exists in the script directory
   • Download OAuth2 credentials from Google Cloud Console

2. “Permission denied”

   • Verify your website in Google Search Console
   • Check that the site URL in config matches exactly

3. “Rate limit exceeded”

   • Increase rate_limit_delay in config.json
   • Reduce max_urls_per_batch

4. “Invalid URL format”

   • Check your Jekyll permalink configuration
   • Verify the URL generation logic matches your site structure

5. OAuth/Redirect URI Issues

   • Make sure you created a “Desktop Application” (not Web Application)
   • Add all required redirect URIs in Google Cloud Console:
     • http://localhost:8080/
     • http://localhost:8081/
     • http://localhost:9090/
     • http://localhost/
   • If you see “redirect_uri_mismatch”, add the missing URI to your OAuth client
   • The script tries multiple ports and falls back to console auth if needed

Debug Mode

Run with verbose logging to see detailed information:

python submit_posts.py --verbose --dry-run

API Limits

• Google Search Console API has daily quotas
• The script includes rate limiting to avoid hitting limits
• Monitor your API usage in Google Cloud Console

Security Notes

• Keep credentials.json and token.json secure
• Add them to .gitignore to avoid committing to version control
• Use service accounts for production environments

Integration with Existing Scripts

This script complements your existing IndexNow script (../indexnow/main.py). You can run both:

1. IndexNow: For immediate notification to Bing and other search engines
2. Google Search Console: For Google-specific indexing

Consider running both scripts in sequence for maximum search engine coverage.