跳转到内容
Hashee Docs

切换 LLM Provider

锁死一个 LLM provider 风险高（涨价 / 服务变更 / 区域可用性）。本页讲怎么 做轻量抽象层，让切换 / A/B / fallback 都 1 行配置改动。

抽象层接口

src/llm/types.ts
export interface ChatMessage {
  role: "system" | "user" | "assistant" | "tool";
  content: string;
  name?: string;
}


export interface LlmProvider {
  complete(messages: ChatMessage[], options?: { model?: string; maxTokens?: number; temperature?: number }): Promise<string>;
  stream(messages: ChatMessage[], options?: { model?: string; maxTokens?: number; temperature?: number }): AsyncIterable<string>;
}

OpenAI 实现

src/llm/openai-provider.ts
import OpenAI from "openai";
import type { LlmProvider, ChatMessage } from "./types";


export class OpenAiProvider implements LlmProvider {
  private client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });


  async complete(messages: ChatMessage[], opts = {}): Promise<string> {
    const r = await this.client.chat.completions.create({
      model: opts.model ?? "gpt-4o-mini",
      messages,
      max_tokens: opts.maxTokens,
      temperature: opts.temperature,
    });
    return r.choices[0]?.message?.content ?? "";
  }


  async *stream(messages: ChatMessage[], opts = {}): AsyncIterable<string> {
    const s = await this.client.chat.completions.create({
      model: opts.model ?? "gpt-4o-mini",
      messages, stream: true,
      max_tokens: opts.maxTokens,
      temperature: opts.temperature,
    });
    for await (const chunk of s) {
      const delta = chunk.choices[0]?.delta?.content;
      if (delta) yield delta;
    }
  }
}

Anthropic 实现

src/llm/anthropic-provider.ts
import Anthropic from "@anthropic-ai/sdk";
import type { LlmProvider, ChatMessage } from "./types";


export class AnthropicProvider implements LlmProvider {
  private client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });


  async complete(messages: ChatMessage[], opts = {}): Promise<string> {
    // Anthropic 把 system 拆出来
    const system = messages.find((m) => m.role === "system")?.content;
    const conv = messages.filter((m) => m.role !== "system").map((m) => ({
      role: m.role === "assistant" ? "assistant" : "user",
      content: m.content,
    }));
    const r = await this.client.messages.create({
      model: opts.model ?? "claude-sonnet-4-6",
      max_tokens: opts.maxTokens ?? 1024,
      system,
      messages: conv,
      temperature: opts.temperature,
    });
    const block = r.content[0];
    return block.type === "text" ? block.text : "";
  }


  async *stream(messages: ChatMessage[], opts = {}): AsyncIterable<string> {
    const system = messages.find((m) => m.role === "system")?.content;
    const conv = messages.filter((m) => m.role !== "system").map((m) => ({
      role: m.role === "assistant" ? "assistant" : "user",
      content: m.content,
    }));
    const s = this.client.messages.stream({
      model: opts.model ?? "claude-sonnet-4-6",
      max_tokens: opts.maxTokens ?? 1024,
      system,
      messages: conv,
      temperature: opts.temperature,
    });
    for await (const event of s) {
      if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
        yield event.delta.text;
      }
    }
  }
}

Ollama 实现（本地）

src/llm/ollama-provider.ts
import type { LlmProvider, ChatMessage } from "./types";


export class OllamaProvider implements LlmProvider {
  constructor(private baseUrl = "http://localhost:11434") {}


  async complete(messages: ChatMessage[], opts = {}): Promise<string> {
    const r = await fetch(`${this.baseUrl}/api/chat`, {
      method: "POST",
      body: JSON.stringify({
        model: opts.model ?? "llama3.1",
        messages,
        stream: false,
        options: { num_predict: opts.maxTokens, temperature: opts.temperature },
      }),
    }).then((r) => r.json());
    return r.message?.content ?? "";
  }


  async *stream(messages: ChatMessage[], opts = {}): AsyncIterable<string> {
    const r = await fetch(`${this.baseUrl}/api/chat`, {
      method: "POST",
      body: JSON.stringify({
        model: opts.model ?? "llama3.1",
        messages, stream: true,
        options: { num_predict: opts.maxTokens, temperature: opts.temperature },
      }),
    });
    const reader = r.body!.getReader();
    const decoder = new TextDecoder();
    let buf = "";
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      buf += decoder.decode(value);
      let lines = buf.split("\n");
      buf = lines.pop() ?? "";
      for (const line of lines) {
        if (!line.trim()) continue;
        const obj = JSON.parse(line);
        if (obj.message?.content) yield obj.message.content;
      }
    }
  }
}

切换 (env)

src/llm/index.ts
import { OpenAiProvider } from "./openai-provider";
import { AnthropicProvider } from "./anthropic-provider";
import { OllamaProvider } from "./ollama-provider";


export const llm = ({
  openai: () => new OpenAiProvider(),
  anthropic: () => new AnthropicProvider(),
  ollama: () => new OllamaProvider(process.env.OLLAMA_URL),
})[process.env.LLM_PROVIDER ?? "openai"]();

业务侧用法

import { llm } from "./llm";


agent.addMessageHandler(async (msg) => {
  const reply = await llm.complete([
    { role: "system", content: "你是助理。" },
    { role: "user", content: msg.payload!.text! },
  ]);
  await agent.send(msg.conversation_id, { type: "text", text: reply });
});

切换 provider 只改 LLM_PROVIDER=anthropic。

灰度 A/B 切换

// 按用户 hash 决定走哪个 provider（10% Anthropic + 90% OpenAI）
import { createHash } from "node:crypto";


function chooseProvider(userId: string): LlmProvider {
  const hash = createHash("sha256").update(userId).digest()[0];
  return hash < 25 ? new AnthropicProvider() : new OpenAiProvider();
}


agent.addMessageHandler(async (msg) => {
  const provider = chooseProvider(msg.sender_id);
  const reply = await provider.complete([/* ... */]);
  // 记录 provider 用于后续质量对比
  await db.query(
    "INSERT INTO llm_calls (user_id, provider, reply_len, latency_ms) VALUES ($1, $2, $3, $4)",
    [msg.sender_id, provider.constructor.name, reply.length, /* latency */],
  );
});

Fallback（首选失败时换备用）

async function completeWithFallback(messages: ChatMessage[]): Promise<string> {
  try {
    return await new OpenAiProvider().complete(messages, { model: "gpt-4o-mini" });
  } catch (e) {
    console.warn("openai failed, falling back to anthropic", e);
    return new AnthropicProvider().complete(messages);
  }
}

Provider 对比（参考，2026 中）

Provider适合优势劣势
OpenAI gpt-4o-mini通用 / 客服 / 短任务便宜（$0.15/$0.60 per 1M I/O） / 快 / 工具支持成熟长 context 一般
OpenAI gpt-4o长 context / 复杂推理128k 窗口 / vision贵 ($2.50/$10 per 1M)
Anthropic Claude Sonnet 4.6长文档 / 编码 / 严谨任务200k 窗口 / 思维链清晰贵 ($3/$15 per 1M)
Anthropic Claude Haiku 4.5快速分类 / 摘要便宜 / 快推理能力弱
Ollama 本地 Llama 3.1隐私 / 离线 / 成本 0零成本 / 完全离线慢（除非 GPU） / 能力弱于云模型

何时该真正多 provider

  • 冗余：单 provider 宕机不影响业务
  • 成本：把简单任务路由便宜模型
  • 能力：vision 用 GPT-4o + 推理用 Claude + 翻译用 Llama
  • 合规：欧盟客户的请求强制走欧盟 region 的 provider

相关页面