Introduction¶

When agents scrape web pages, they often encounter HTTP scraping that returns empty shells or is blocked by anti-scraping measures. better-crawler-4-agent is an MCP server that uses real browser rendering to solve these problems.

Core Features¶

The core value of this plugin is an MCP server that turns URLs into clean main content. It uses real browser rendering, so sites like Zhihu, CSDN, and Juejin—where HTTP scraping may return empty shells—can still return content.

After installation, the model in the DSH profile gains three tools:

fetch_url(url, max_chars) - fetch a single page
fetch_urls(urls, max_chars, max_concurrent) - fetch concurrently, with each item succeeding or failing independently, supporting up to 20 concurrent tasks
crawler_status() - check browser status and diagnose startup failures

Internally, the plugin uses a tiered fallback strategy: browser rendering (primary path) -> trafilatura direct extraction (fast path) -> httpx + full browser fingerprint (fallback). When the browser is unavailable, it automatically falls back to the last two tiers, and crawler_status provides the reason.

Installation and Activation¶

In a DSH environment, this is a standard plugin package. The installation command is as follows:

dsh plugin --profile web add github:TC635807/better-crawler4agent

After installation, the plugin uses dsh.bundle.patch to register the repository’s MCP server into the current profile. The model can then directly use the three tools above.

Usage¶

This is a standard stdio MCP server and is not bound to any client. In addition to being used as a plugin in DSH, it is compatible with the generic mcpServers format and can be integrated with clients such as Claude Code, Cursor, Windsurf, and Cline.

If you need to call it directly from code, you can use the Python interface:

import sys
sys.path.insert(0, r"D:\better-crawler4agent\src")

from better_crawler import Fetcher

async def main():
    f = Fetcher()
    await f.start()                                  # 预热浏览器
    result = await f.fetch("https://example.com/article")
    if result.ok:
        print(result.content)
        print(f"命中层级: {result.tier}, {result.content_length} 字符")
    else:
        print(f"失败: {result.error}")
    await f.close()

Notes¶

When using it, pay attention to the following points:

Playwright version pinning: The plugin pins playwright==1.61.0. When upgrading Playwright, you must reinstall the browsers in sync; otherwise, it may fail silently due to a revision mismatch.
Protocol restriction: Only the http and https protocols are allowed.
Fallback mechanism: When the browser is unavailable, it automatically falls back to trafilatura or httpx.
Concurrency limitation: fetch_urls supports at most 20 concurrent tasks, and it is recommended not to exceed this number in a single call.
Security: Access to intranet addresses is not allowed by default. For debugging, you can set the environment variable BETTER_CRAWLER_ALLOW_PRIVATE=1.

Conclusion¶

better-crawler-4-agent is a practical tool for solving web scraping challenges for agents. Through real browser rendering and a tiered fallback strategy, it can extract clean main content on most sites.

GitHub repository: https://github.com/TC635807/better-crawler4agent
Skill directory: https://www.skillhub.cn/plugins/TC635807/better-crawler4agent

better-crawler-4-agent