Preface¶

The extension approach for DSH is plugin-oriented: models, backends, and UI capabilities can be integrated as independent plugins. For those working on intelligent agents, a common bottleneck is: text-based chat works fine, but screenshots, error images, charts, and design mockups must first be converted to text before they can be handed off to a text model for further processing; when images are needed, the generated results often end up scattered across external services.

The usual approach involves maintaining vision APIs, image APIs, transcription prompts, caching, and failure retries yourself. dsh-multimodal consolidates these steps into the DeepSeek Harness conversation pipeline: first reads the image, then lets DeepSeek continue processing; when an image is needed, displays the generated image directly within the conversation.

What This Is¶

dsh-multimodal is a DeepSeek Harness (DSH) plugin, maintained by MC5lan. It provides three types of capabilities: vision transcription, OCR, and text-to-image generation, all executed via OpenAI-compatible backends.

It does not include built-in models, providers, or backends. Vision endpoints, image backends, and models all require user configuration.

Core Features¶

Image Transcription Followed by Continued Answering¶

After pasting a screenshot or image, the plugin first calls the user-configured vision provider for verbatim transcription, then DeepSeek continues processing the question within the same turn.

Transcription cache can be reused for the same image and context; the cache is LRU-based and scoped per session.

If the primary vision provider fails or is rate-limited, it can fall back to fallbackProviders; parallelImages enables parallel transcription of multiple images; small images can be routed to cost-effective providers based on costMaxPixels.

OCR and Contextual Transcription¶

The plugin supports calling extract_text for OCR, outputting Markdown, plain text, or JSON.

Built-in transcribeMode presets:

• error-fix
• chart-sql
• design-code

Text-to-Image and Image Cards¶

When the user requests images, DeepSeek can call generate_image, and the generated image is displayed within the conversation.

Generated image cards support: thumbnails, click-to-zoom, one-click download, prompt and model metadata, crop-to-ask, copy params, and failure retry.

Image generation backend failover attempts the next backend on failure; AUTH/aborted skips failover.

Supports integrating any non-OpenAI/DashScope image API via the custom backend.

Vision Provider Integration¶

extraProviders accepts any OpenAI-compatible vision endpoint, with presets for: Zhipu, Bailian, Xfyun, ModelScope, SiliconFlow, Qianfan, and local Ollama.

Version 0.7+ supports pasting an API key for automatic platform detection and setup.

The settings page supports JSON import/export for configuration.

Installation and Activation¶

First, verify the environment: official DeepSeek Harness 0.1.0-rc.6+ and Node.js 18+.

Then execute:

dsh plugin --profile web add https://github.com/MC5lan/dsh-multimodal

After the above steps, restart dsh web. In the Settings sidebar, a “Multimodal” page will appear, where all vision and image-related configuration is completed in one place.

Typical Usage¶

1. Quick setup (0.7+): Open Settings → Multimodal → Quick Setup, paste the API key, and the platform is automatically detected and configured.

2. Manual setup: Click the platform preset card (Zhipu / Bailian / Xfyun / ModelScope / SiliconFlow / Qianfan / local Ollama) and paste the key, or declare any OpenAI-compatible endpoint in extraProviders.

3. Select vision routing: Choose the deepseek-vision model in the Web UI model selector; sessions routed through this model gain vision capabilities.

4. Start using: Paste or drag in an image and ask questions during normal chat, or directly request image generation.

Use Cases and Notes¶

Suitable for users who already have OpenAI-compatible vision APIs, image APIs, or local Ollama vision endpoints and want to read images, extract text, and generate accompanying images directly within DSH conversations.

Notes before use:

• The plugin does not include models, providers, or backends; vision endpoints, image backends, and models all require self-configuration.
• The plugin runs with the current DSH process permissions; review the source code and license before installation; the license follows what is declared in the repository.
• Security limits include: API key allow-list, trustedBaseUrls, SSRF guard, no forwarding of sessionId, prompt-injection markers, sensitive data redaction, audit logs, and LRU memory limits.
• Explicitly listed local endpoints (such as http://localhost:11434) are allowed to use plain HTTP without requiring a key.

Links¶

• Catalog page: The DSH plugin catalog is a community directory, not equivalent to an official app store; you can search for dsh-multimodal in the catalog.
• GitHub repository: https://github.com/MC5lan/dsh-multimodal