Automating Wikipedia Tasks with Petscan: A Developer's Guide

Overview of the Petscan API

Petscan is a powerful web-based tool and Application Programming Interface (API) designed to simplify complex queries and bulk operations across MediaWiki-powered sites, most notably Wikipedia and its sister projects like Wikimedia Commons and Wikidata. At its core, the Petscan API acts as a sophisticated query builder that allows developers to generate lists of pages based on a wide array of criteria including categories, templates, namespace, page properties, and cross-wiki links like those from pet ct scan hk projects. Instead of manually navigating the sprawling category tree of Wikipedia or writing complex SQL queries against the database replicas, developers can leverage the Petscan API to construct highly specific queries with just a few parameters. This API exposes a standard HTTP endpoint, typically `https://petscan.wmflabs.org/`, where you can send GET or POST requests containing a JSON object or query string parameters. A unique aspect of the Petscan API is the 'combination' logic; it allows you to apply logical AND, OR, and NOT operators between different query modules (e.g., categories, links, templates). For instance, you can create a query that finds all pages in Category:Physics AND in Category:Nuclear medicine, but NOT in Category:Stubs. This logical depth makes it invaluable for data extraction, analysis, and automation.

Benefits of automating Wikipedia tasks with Petscan

Automating repetitive Wikipedia tasks through the Petscan API offers significant efficiency gains for developers, data scientists, and systematic editors. Manual browsing and categorization of Wikipedia pages, which can take hours for broad topics like medical imaging, is reduced to milliseconds of API call time. The primary benefit is the ability to programmatically generate curated datasets. For example, a researcher studying medical imaging can use the API to compile a comprehensive list of all Wikipedia articles that contain the term 'pet mri' while belonging to specific high-level categories. This eliminates the tedium of manual page-by-page inspection and the risk of missing relevant articles. Furthermore, Petscan enables scheduled tasks. A bot running a daily Python script can check for new pages added to a specific category, assess their quality based on template usage, and report findings to a project dashboard. This automation keeps Wikipedia content organized without constant human oversight. Another critical benefit is cross-wiki consistency. Because Petscan can query Wikidata alongside Wikipedia, it helps maintain data integrity across different language editions. You can set up a script that identifies pages in the English Wikipedia that lack a corresponding Wikidata item, flagging them for manual review. Ultimately, automating with Petscan reduces human error, speeds up community workflows, and allows contributors to focus on content creation rather than administrative drudgery, ensuring that resources related to petscan queries are handled effectively.

Setting up your development environment

To begin automating Wikipedia tasks with the Petscan API, you need a minimal but effective development environment. The most common and recommended approach is to use Python due to its simplicity and the availability of supporting libraries like `requests` and `json`. Here is a concrete guide to get started. First, ensure you have Python 3.8 or higher installed. Create a new virtual environment to isolate your dependencies: `python -m venv petscan_env` and activate it (on Windows: `petscan_envScriptsactivate`, on macOS/Linux: `source petscan_env/bin/activate`). Next, install the core libraries: `pip install requests json`. You will use `requests` to make HTTP calls to the Petscan endpoint and `json` to parse the response. Additionally, install `python-dotenv` for managing sensitive credentials if you plan to edit Wikidata or Wikipedia programmatically later. Your code structure should be straightforward. Create a file, for instance, `petscan_client.py`. Within this file, define constants for the base URL: `BASE_URL = 'https://petscan.wmflabs.org/'.` It is crucial to set a User-Agent header in your requests as per Wikimedia's User-Agent policy. A proper User-Agent looks like: `headers = {'User-Agent': 'MyProjectBot/1.0 (https://example.com; myemail@example.com)'}`. For querying, you will define a dictionary of parameters such as `'language': 'en'`, `'project': 'wikipedia'`, and the specific categories you want to scan. Regarding rate limiting, while Petscan itself is often not aggressive, downstream effects on Wikipedia API or database replicas matter. Always include a delay between requests using `time.sleep()`. Finally, set up logging using Python's `logging` module to track your bot's activity, errors, and successful API calls. With this environment set, you are ready to make your first API request to the service that powers petscan functionality.

Understanding the API endpoints and parameters

The Petscan API primarily operates through a single main endpoint, `https://petscan.wmflabs.org/`. However, the complexity lies within the parameter structure that you pass to this endpoint. The API accepts parameters either as a JSON body or URL query string, but the JSON method is more robust for complex queries. The key parameter groups include 'categories', 'templates', 'links', 'namespace', and 'wikidata'. The 'categories' parameter is where you specify the source category, e.g., `'categories': 'Medical imaging'`. The 'combosubset' parameter determines how categories are combined (union or intersection). For example, to find pages in both Category:A and Category:B, you set `'combination': 'AND'`. Another important parameter is 'output_compatability', which controls the output format (e.g., 'csv', 'json', 'tsv'). For programmatic consumption, 'json' is standard. There is also 'depth' which defines how deep to scan subcategories; a depth of 0 means the immediate category only, while depth 10 can retrieve pages from thousands of subcategories. For advanced filtering, you can use 'wikidata_source_sites' and 'wikidata_project' to pull in data from Wikidata, which is useful for queries related to pet ct scan hk institutions. The 'namespace' parameter filters by page type, like Article (0), Category (14), or File (6). Mastering these parameters allows you to build surgical queries that pinpoint exactly the data you need from the vast Wikipedia corpus. It is advisable to experiment with the interactive Petscan web interface first to generate the desired query and then copy the JSON parameters—a time-saving and educational strategy.

Sending requests using Python or other programming languages

Implementing the API request in Python is a direct process that translates your parameter selection into a functional script. Using the `requests` library, you begin by constructing the payload as a dictionary. Consider a scenario where you need to find all Wikipedia articles related to pet mri that are in the 'Medical physics' category. First, define your search parameters: params = {'language': 'en', 'project': 'wikipedia', 'categories': 'Medical physics', 'search_query': 'pet mri', 'output_compatability': 'json', 'depth': 1}. It is important to note that the 'search_query' parameter applies a full-text or page title search, depending on other settings. The next step is to send a POST request (or GET) to the endpoint: response = requests.post(BASE_URL, json=params, headers=headers). The json=params argument automatically encodes the dictionary as a JSON payload. After receiving the response, parse the JSON: data = response.json(). The Petscan API typically returns the results under data['*']['a']['*'] or similar nested keys, depending on the version. You will likely need to iterate over a list of page objects, each containing fields like 'title', 'pageid', and 'ns' (namespace). For languages other than Python, the concept remains identical. In JavaScript (Node.js), you would use the fetch API or axios with a POST request and a JSON body. In R, the httr package can perform similar operations. Regardless of the language, always validate the HTTP status code (200 is success) and inspect the response.text content if errors arise, as the API sometimes returns HTML-formatted errors for malformed requests.

Handling API responses and errors

Properly handling the responses and potential errors from the Petscan API is crucial for building a robust automation pipeline. Petscan returns its data in a structured JSON format, but the exact structure can be nested. A successful response typically includes a top-level key * containing a dictionary with keys like a, f, and i. The a key usually holds the actual list of page titles. For example: response_data['*']['a']['*'] might be a list of page objects. Each object often has title and pageid. A Python code snippet to extract titles could look like:
pages = response_data.get('*', {}).get('a', {}).get('*', [])
for page in pages: print(page.get('title')). Error handling is equally important. Unsuccessful responses may stem from malformed parameters, exceeding timeouts, or server-side issues. Always wrap your request in a try-except block:
try: response = requests.post(BASE_URL, json=params, headers=headers, timeout=30)
response.raise_for_status()
except requests.exceptions.HTTPError as err: logging.error(f'HTTP error: {err}')
except requests.exceptions.Timeout: logging.warning('Request timed out, retrying...')
Additionally, Petscan might return a 200 status but include an error message in the JSON body. Check for keys like message or error within response_data. If you encounter issues related to pet ct scan hk queries, the system might return empty results if the search term is extremely specific; validate your parameters by running the same query in the Petscan web interface. Implementing retry logic with exponential backoff (e.g., using the tenacity library) can mitigate transient failures.

Automating category analysis

One of the most common and powerful applications of the Petscan API is automating category analysis on Wikipedia. This involves programmatically examining the membership, depth, and interconnections of Wikipedia's category tree. For instance, a developer can set up a script that runs weekly to list all pages within the 'Health' category tree on the English Wikipedia, then subdivide them by subcategory depth and namespace. Using the petscan API's 'depth' parameter, you can retrieve pages from up to 10 levels deep, and the 'categories' parameter supports combining multiple categories using AND/OR logic. A practical example is identifying orphaned pages within a category—pages that belong to a category but are not linked from any other article in the same category. While Petscan does not have a direct 'orphan' parameter, you can combine it with the 'links' parameter to check for incoming links. Scripting this workflow requires iterating through categories. You can exploit the 'manual_list' option to feed a list of categories from a file, but more elegantly, you can query the Wikidata API to find related categories first. Additionally, using Petscan's 'pageprops' module, you can filter pages based on specific page properties like 'wikibase_item' to check if they have a connected Wikidata item. This is crucial for systematic editors in Hong Kong maintaining health-related content like pet ct scan hk resources. The script can output a report in CSV format listing all pages without a Wikidata item, sorted by category. Such automated category analysis reduces manual labor and ensures that no corner of a category tree remains unexamined, generating actionable lists for community review.

Generating reports on Wikipedia content

Beyond raw analysis, the Petscan API excels at generating structured reports that can track the health and status of Wikipedia content over time. A typical report might include metrics like page count, average page length, number of references, or maintenance tag usage across a given category tree. To build such a report, you would extract a list of page titles via Petscan, then for each title, make an additional API call to the MediaWiki API (e.g., via https://en.wikipedia.org/w/api.php?action=query) to retrieve properties like page size, edit count, or the presence of citation needed tags. Combining Petscan's ability to find specific templates (via the 'templates' parameter) with MediaWiki's property extraction allows for robust reporting. For example, a report on 'Good articles' (GA) related to medical imaging could use Petscan to find all articles in Category:GA-Class medical articles, then cross-reference them with the 'Citation needed' template count. The final output can be formatted as an HTML table or a JSON file. Here is a simplified JSON structure for a report row:
{"title": "Positron emission tomography", "pageid": 12345, "ref_count": 87, "ga_status": "Current"}. For Hong Kong-related healthcare content like pet mri-specific pages, you could tailor the report to measure article completeness (e.g., does the article include a section on safety or cost in Hong Kong?). Regular automated reports can be posted to a WikiProject talk page or a GitHub repository, allowing communities to see trends in content quality at a glance.

Creating bots for maintenance tasks

The ultimate step in automation is developing full-fledged bots that use the Petscan API as their primary data source for performing maintenance tasks on Wikipedia. These bots can handle jobs like adding or removing categories, tagging uncategorized pages, fixing broken links, or repositioning templates. The typical architecture involves a loop: query Petscan to get a list of target pages, perform an action (like editing via the MediaWiki API), and log the results. For instance, a bot can identify all articles in 'Category:Articles lacking sources' that are also in a specific topic area like 'Medical imaging'. Using Petscan, you can set parameters categories: 'Articles lacking sources|Medical imaging' with combination: 'AND'. The bot then receives the list, and for each article, uses the MediaWiki edit API to add a specific maintenance template or a stub notice. Another critical maintenance task is reverting or tagging spam. If you have a list of known spam keywords (e.g., generic drug names), you can use Petscan's 'linksto' or 'search_query' to find pages containing those terms. For a bot operating in the pet ct scan hk context, you might build a bot that monitors new pages in the Hong Kong-related categories and automatically adds them to a centralized project list on Wikipedia. However, bot operations require careful consideration of rate limits; you must adhere to Wikimedia's Bot policy by registering your bot account and throttling edits (e.g., no faster than one edit per two seconds). Using the pywikibot library combined with Petscan outputs is a common best practice, as it handles token acquisition, edit conflicts, and logging efficiently.

Using Wikidata queries within Petscan

Petscan's integration with Wikidata is a game-changer for cross-platform data management. You can directly embed SPARQL queries within Petscan parameters to filter results based on Wikidata properties. For example, you can query Petscan to return only English Wikipedia articles that have a Wikidata item where the property 'country of origin' (P495) is 'Hong Kong'. This is achieved by setting the 'wikidata_sourcelanguage' parameter to 'wikidata' and providing a SPARQL snippet. The syntax requires specifying the 'wikidata_query' parameter with a simplified query that typically selects items (items) by their Q identifier. A practical example is finding all Wikipedia articles about hospitals in Hong Kong that offer a specific medical service like pet ct scan hk. The query might look like:
SELECT ?item WHERE { ?item wdt:P31 wd:Q16917 . ?item wdt:P17 wd:Q8646 . ?item wdt:P366 ?diagnostic . FILTER(CONTAINS(STR(?diagnostic), 'PET-CT')) }
Petscan then matches the Q identifiers obtained from Wikidata to corresponding Wikipedia articles. This allows for highly refined data extraction that purely category-based queries cannot achieve. The combination is powerful because it bridges structured data (Wikidata) with unstructured textual content (Wikipedia). By mastering this technique, developers can build applications that automatically sync edits across platforms: a change in a Wikidata property can trigger a Petscan-based review of all associated Wikipedia articles.

Automating Wikidata updates based on Petscan results

Automation becomes bi-directional when you use Petscan results to drive updates on Wikidata. For instance, if Petscan identifies a set of English Wikipedia articles that have a specific infobox but are missing a corresponding property on their Wikidata item (e.g., 'medical specialty' property), a script can automatically add that property. The workflow: use Petscan to query for pages in 'Category:Oncology' that have the 'Infobox medical condition' template. Iterate through the resulting page list, read the infobox data (via parsing the page source), extract the specialty information, and then use the Wikidata API to add the statement (e.g., P31: Q181237 - oncology). This is particularly valuable for large-scale curation. For Hong Kong-specific data, you could automate the addition of 'location' (P276) links if the article mentions coordinates. A more advanced use case involves updating 'image' (P18) statements. If Petscan finds files uploaded to Wikimedia Commons categorizing in a specific Hong Kong hospital's category, you can programmatically link those images to the correct Wikidata items. Always ensure that your bot is well-tested and respects the zero-abuse policy. The combination of petscan data retrieval and Wikidata write APIs reduces the massive manual effort of linking structured knowledge.

Building tools for cross-wiki data management

Given Wikipedia's multilingual nature and the diverse Wikimedia projects, Petscan is an excellent foundation for building cross-wiki data management tools. With Petscan, you can query multiple language editions of Wikipedia simultaneously using multi-project parameters. For example, a developer can create a dashboard that monitors the coverage of a given topic, such as 'seismic activity in Hong Kong', across English, Chinese, and Japanese Wikipedia. Petscan returns unified lists from each project, which can be merged and compared in a single report. The 'wikidata' aspect of Petscan also facilitates this, as you can identify pages in different languages that are linked to the same Wikidata item. Such a tool can highlight gaps—for example, if an item has a Wikipedia page in 10 languages but is missing in the pet ct scan hk regional context. Another practical application is building a 'completeness' checker for Wikimedia Commons. By querying Petscan for file categories on Commons (different project base 'commons.wikimedia.org'), you can check if a given article's infobox has an image from Commons. Cross-wiki management also involves handling disambiguation pages across languages automatically. By pairing Petscan's 'links' parameter with 'wikidata', you can ensure that inter-language links on disambiguation pages are consistent. These tools empower volunteer communities to manage vast amounts of data with minimal manual overhead, keeping the ecosystem healthy.

Rate limiting and avoiding abuse

Responsible use of the Petscan API requires strict adherence to rate limiting and ethical guidelines to avoid overloading the underlying Wikimedia database replicas. While Petscan itself is resilient, the queries you issue translate to load on database servers. As a developer, you should implement a minimum delay of at least 1–2 seconds between API calls. The standard practice is to use a 'throttle' mechanism: time.sleep(1.5) in Python loops. It is also recommended to use the 'retry-after' header if it appears in the HTTP response. Additionally, limit the depth of category recursion; a depth of 5 or less is preferred for broad topics unless absolutely necessary. Using the 'combination' engine, try to narrow down categories before expanding, as smaller result sets are faster and less resource intensive. Always include a descriptive User-Agent string as described earlier; generic agents like 'python-requests/2.28' risk being blocked. For heavy production tasks, consider requesting a bot flag on your tool and using the 'nobot' parameter if you intend to ignore bot-excluded pages. Monitoring your API usage with a logging framework helps you audit your requests. By following these guidelines, you ensure that your automation is a positive contribution, especially when querying sensitive topics like pet mri or other medical specialities, preserving API availability for all users.

Error handling and debugging

Even with careful development, errors are inevitable. Effective debugging begins with proper logging. Use Python's logging module to capture timestamps, response status codes, and partial response content. A common issue is '400 Bad Request' due to malformed JSON or missing mandatory parameters like 'language' or 'project'. Validate your payload before sending it; you can use a schema validator like jsonschema to ensure keys are present. Another frequent problem is the API returning an empty result set when a category doesn't exist or when the query is too specific. In that case, verify the category name on the live Wikipedia site. For petscan queries, the API sometimes returns a '203' status? No, typical errors are 400 or 500. A 500 error indicates a server-side issue; it is best to implement an exponential backoff retry mechanism (e.g., wait 2, 4, 8 seconds before retrying up to 3 times). Network errors (timeouts, connection reset) also need handling. The 'tenacity' library simplifies retry logic: @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=2, min=4, max=30)). Lastly, debug your queries first on Petscan's web UI 'Petscan' page, which provides a raw JSON view of the query parameters and results. This step is invaluable before writing code. For complex queries related to pet ct scan hk, test with a small category or shallow depth first, then scale up.

Contributing to the Petscan API

The Petscan API is an open-source tool hosted by Wikimedia Toolforge. As a developer, you can contribute to its ecosystem by submitting bug reports, feature requests, or even code patches. The source code is typically hosted on GitLab or Phabricator. Before contributing, understand the architecture: Petscan is written in Perl (the backend) with a JavaScript frontend. You can contribute by improving the API's documentation, particularly around less-documented features like the 'wikidata' query syntax. Another valuable contribution is creating and sharing client libraries in various programming languages (Python, JavaScript, PHP) to lower the entry bar for new developers. If you work on medical or scientific data, you can contribute a set of curated queries relevant to pet ct scan hk. Such community resources accelerate everyone's work. Additionally, you can help by monitoring the Toolforge environment and reporting server issues to the administrators. Engaging with the Petscan user community on the Wikimedia mailing lists or IRC channels also counts as a contribution, as user feedback drives development. By giving back, you ensure the tool continues to evolve to meet the needs of Wikipedia's global community, including those working on High-quality medical content.