voicetype: Apple 端侧 + LLM 异步精修,可选 xunfei/dual

新增 apple-llm 引擎(默认):Apple SFSpeechRecognizer → 贴原文 → LLM (~1s,
thinking=disabled) → 退格覆盖精修版。LLM 走 ~/ssw.ini [llm-api],无配置
静默回退 Apple 原文。

- 删 localClean(中文无空格,正则误伤'好啊'等正常词)
- pickEngine() 默认 apple-llm
- LLMCleaner 自动带 thinking: {type: 'disabled'} (GLM 7s+ → ~1s)
- build.sh 新增 apple / apple-llm 模式,默认 apple-llm
- README + AGENTS.md 同步更新

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-10-02 19:23:47 +08:00
commit b380eaf252
7 files changed
+1252

No files matched your search

+4
View File
@@ -0,0 +1,4 @@
.DS_Store
*.swp
*.bak
.history/
+67
View File
@@ -0,0 +1,67 @@
# voicetype / VoiceType
macOS 工具:**长按鼠标左键 → 说话 → 松开自动粘贴到光标处**(任意 app 输入框;微信窗口内自动放行给微信自己的语音输入)。
## 布局
```
src/voicetype.swift 全部源码(单文件探针,~330 行)
src/Info.plist app 包描述(权限声明)
src/build.sh 构建+签名+安装+重启一条龙
docs/voicetype/ 详细文档(用法/架构/踩坑)
```
## 常用命令
```bash
src/build.sh # 编译 → 签名 → 装到 /Applications → apple-llm 启动 → 打日志(默认)
src/build.sh apple # 纯端侧(无 LLM 精修)
src/build.sh apple-llm # 端侧 + LLM 异步精修
src/build.sh xunfei # 同上,但经 xop 注入讯飞凭据 → 讯飞 IAT v2 WebSocket 引擎
src/build.sh dual # apple + xunfei 并行 + LLM 精修(需 xop 注入)
tail -f /tmp/voicetype.log # 看判定/识别/粘贴日志
killall VoiceType # 停止
```
## 引擎切换(两个互不重叠的层面)
1. **持久切换**:`echo apple|apple-llm|xunfei|dual > ~/.voicetype-engine`,任意方式启动都生效。
2. **启动方式注入**:`src/build.sh xunfei` 用 `xop run --secret ... -- open ...` 临时把讯飞凭据塞进进程环境变量,**只对当次启动有效**,不会写到磁盘/源码。
讯飞凭据一次性入库(从 TTY 读值,源码/git/日志都不出现):
```bash
xop set xf_vt_APPID # APPID
xop set xf_vt_APISecret # APISecret
xop set xf_vt_APIKey # APIKey
```
LLM 配置(`apple-llm` / `dual` 需要;无 INI 则静默回退 apple 原文):
```ini
# ~/ssw.ini
[llm-api]
key=sk_xxxxxxxxxxxx
base-url=https://your-llm-proxy/v1
model=GLM-5.3-Flash
```
容忍全角破折号 `—` 与半角 `-` 等价。引擎请求时强制带 `thinking: {type: "disabled"`,绕过 GLM 推理模式(7s → 1s)。
## 红线(违反必翻车)
1. **签名必须用固定 Apple Development 证书**(`src/build.sh` 里的 `IDENT`)。ad-hoc `-s -` 每次重编译身份都变,TCC 当成新 app,三项授权全作废。
2. **必须以 .app 形式装在 /Applications 并 LaunchServices 注册**。裸 CLI 的 TCC 归属是宿主终端,行为混乱;/tmp 下的 app 进不了辅助功能列表。
3. **权限三件套缺一不可**:麦克风(录音)、语音识别(apple 引擎)/ 辅助功能(全模式)。日志 `trusted: true` + `mic granted: true` + `speech auth: 3` 全绿才算就绪。
4. **长按状态机:一次按下只判定一次**(激活/跳过后进 `.dead`,松开才复位)。打回 idle 会在按住期间循环重启录音引擎(已踩过)。
5. **讯飞凭据永不落盘**:源码/git/日志/argv/历史都不能出现 APPID/APISecret/APIKey 明文。运行时统一经 `xop run --secret KEY=xf_vt_* -- open ...` 注入到 `ProcessInfo.processInfo.environment["XUNFEI_*"]`。引擎拿到空 env 时**直接报错退出**,不静默回退 apple(避免"以为用了讯飞结果识别差"的误判)。
6. **LLM 凭据明文存于 `~/ssw.ini`**(用户偏好,不进 xop);`key` 字段不进源码/git/日志。LLM 失败/超时/未配 → 保留 Apple 原文,不静默丢弃。
## 核心设计(为什么这么做)
- **鼠标左键轮询零权限可用**(`CGEventSource.buttonState`),Fn/Globe 键则轮询不可见——所以触发方式选左键而不是 Fn(实验证据见 GOTCHAS §1)。
- **判定用"光标下最顶层窗口归属"**(CGWindowList z 序),不用 `frontmostApplication`——微信悬浮窗会霸占键盘焦点导致误判(踩坑见 GOTCHAS §5)。
- **判定三层**:AX 焦点角色严格判断 → `AXManualAccessibility` 强开 Chromium 系 AX 再查 → 完全不透明则 app 级兜底放行。
- **注入 = 写剪贴板 + 合成 ⌘V**,1.5s 后恢复原剪贴板(与微信同款)。
- **识别引擎可插拔**:`protocol ASREngine` + `pickEngine()` 读 `~/.voicetype-engine`。默认 **Apple 端侧 SFSpeechRecognizer**(zh-CN,离线流式);备选 **讯飞 IAT v2 流式 WebSocket**(端点 `wss://iat-api.xfyun.cn/v2/iat`,HMAC-SHA256 鉴权,40ms 切片推送,服务端逐条返回拼接)。微信自家 ASR 是私有鉴权协议,外部不可用。
详细原理与全部踩坑:`docs/voicetype/README.md`、`docs/voicetype/GOTCHAS.md`。
详细原理与全部踩坑:`docs/voicetype/README.md`、`docs/voicetype/GOTCHAS.md`。
+70
View File
@@ -0,0 +1,70 @@
# GOTCHAS — VoiceType 踩坑全记录(2026-10-02 spike)
## 1. Fn(🌐) 键轮询不可见;鼠标左键零权限可见
写了 `CGEventSource.flagsState` 双视图(combinedSessionState + hidSystemState)轮询探针,用户两轮实按 **8ms 采样粒度下一个翻转都抓不到**——Apple Silicon 的 Globe 键在更底层被系统消费,只有 Fn+字母组合时修饰标志才出现。而 `CGEventSource.buttonState(.combinedSessionState, .left)` 一次就干净抓到左键 DOWN/UP,**不需要任何权限**。→ 这就是触发方式选左键不选 Fn 的实验依据;若将来要做 Fn 触发,必须走辅助功能 CGEventTap(flagsChanged/keycode 63)。
## 2. 微信的"长按 Fn/鼠标说话"是微信自己的全局劫持
- 不是 macOS 系统功能;4.1.7+ 加入,授权「控制你的 Mac」后在微信外生效,结果走**剪贴板渠道**注入,可在微信 设置→快捷键 关闭。
- 二进制证据(热更新副本 `~/Library/Containers/com.tencent.xinWeChat/Data/Documents/app_data/xplugin/plugins/Magicdy/<ver>/extracted/wechat.dylib` 的导入符号):`mmui::VoiceInputKeyboardListener / VoiceInputMouseListener / VoiceInputMouseSessionController / VoiceInputFocusListener` + `AXIsProcessTrusted` + `CGEventTapCreateForPid`(按前台 app 建 tap,所以不出现在 CGGetEventTapList 全局列表) + `CGEventSourceButtonState/FlagsState`(轮询兜底) + `NSPasteboard`/`CGEventPost`。ASR 走自家 CGI(`voicetrans`/`uploadvoice`),私有鉴权协议,外部不可用 → 本工具用 Apple 端侧识别。
- 包内 grep 中文字符串全空:微信 UI 串打包时不可直接 strings,要靠符号表和 `mmui::` 类名。
## 3. TCC 权限:裸 CLI 归属混乱,必须打包 .app + 固定签名
- 裸 CLI 从终端跑:麦克风 requestAccess 直接 false,授权对象是"责任进程"(宿主终端),行为不可预期。
- 打包最小 .app(Info.plist 带 NSMicrophoneUsageDescription/NSSpeechRecognitionUsageDescription)+ `open` 启动 → TCC 弹窗正常附着,授权立即可用。
- **ad-hoc 签名(-s -)每次重编译 cdhash 都变 → TCC 当成新 app,授权作废**。用钥匙串里已有的 Apple Development 证书签名,身份锚定证书,重编译授权保留。
- /tmp 下的 app 不进「辅助功能」列表(未注册 LaunchServices)→ 装到 /Applications + `lsregister -f`。
## 4. AX API 三连坑
- 新 SDK 把 `AXUIElementCopyAttributeValue` 导入成 **3 参数版**(没有 error 出参):`(AXUIElement, CFString, UnsafeMutablePointer<CFTypeRef?>) -> AXError`。写 4 参报"extra argument in call",用小文件穷举签名确认。
- 常量 `kAXFocusedUIElementAttribute` 是 String,要 `as CFString` 传。
- 查询失败码要打出来看:`-25204`(CannotComplete)/`-25212`(NoValue=app 没暴露聚焦元素)是完全不同的病因。
## 5. 判定"长按发生在哪个 app":用光标下窗口,别用 frontmostApplication
用户在备忘录长按,日志却显示 frontmost=微信——微信悬浮窗/语音面板会**霸占键盘焦点**,`NSWorkspace.frontmostApplication` 跟着错。改为 `CGWindowListCopyWindowInfo(optionOnScreenOnly)` 按 z 序找包含光标点的第一个 layer-0 窗口的 owner pid,所见即所得。
## 6. Chromium/Electron app 的 AX 树默认不建
`kAXFocusedUIElementAttribute` 返回 -25212。对 app 元素设置 `AXManualAccessibility` 和 `AXEnhancedUserInterface` = true,~250ms 后重查,多数 Chromium 系会建树。仍不行的(HexHub 实测)只能 app 级兜底放行。
## 7. 状态机 flap 循环
按下→判定失败→打回 `.idle`→下一轮询周期又当新按下→重启录音引擎……按住期间每 ~350ms 循环一次,表现为菜单栏录音图标狂闪、永远到不了激活。修法:**一次按下只判定一次**,判定完进 `.dead`,左键抬起才回 `.idle`。
## 8. 短句丢头 → 预卷录音
v1 在激活(350ms)后才启动录音引擎,用户按下即开口,前几百毫秒语音丢失 → ASR 报 `No speech detected (1110)`。修法:**鼠标按下瞬间**就启动引擎收缓冲,激活时才挂识别任务(`SFSpeechAudioBufferRecognitionRequest` 允许先 append 后建 task),普通点击抬起则丢弃。
## 9. 讯飞 IAT v2 鉴权在 URL query 里,不在 header
`URLSessionWebSocketTask` 用 `webSocketTask(with: URL)` 时**所有参数都得到 URL 上**,没有 header 钩子(Authorization 不能直接挂)。所以讯飞的鉴权模式:
1. RFC1123 GMT 时间串 `date`
2. 拼接 `host: iat-api.xfyun.cn\ndate: <date>\nGET /v2/iat HTTP/1.1` → HMAC-SHA256(secret) → base64 → `signature`
3. 拼 `api_key="...", algorithm="hmac-sha256", headers="host date request-line", signature="..."` → 再 base64 → `authorization`
4. URL = `wss://iat-api.xfyun.cn/v2/iat?authorization=<>&date=<>&host=iat-api.xfyun.cn`
注意 URL 里的日期**不能跨分钟**:同一连接生命周期里服务端会校验(签名原始串里的 date 与请求 URL 里的 date 一致)。长会话需要周期重连(本工具单次录音 ≤60s,无此问题)。
## 10. AVAudioConverter 系统原生 PCM → 16k Int16 mono
`AVAudioEngine.inputNode.outputFormat(forBus: 0)` 出来通常是 48k 或 44.1k Float32。讯飞要 16k/16bit/mono PCM,所以:
- `AVAudioConverter(from: inputFmt, to: targetFmt)` 直接搭好
- `convert(to:from:error:)` 一行转完(回调式只用于压缩输入的流式场景,我们用不上)
- 输出 `AVAudioPCMBuffer.int16ChannelData[0]` → `UnsafeRawBufferPointer(start:count:)` → `Data.append(contentsOf:)`
- 注意 `Int16` 指针不能直接喂 `Data.append(_:count:)`,必须先按字节重解释
## 11. xop 注入 env 经 `open` 传递到 .app
`xop run --secret X=xf_vt_X -- open /Applications/VoiceType.app` 时,env 经 `xop → open → LaunchServices → app` 这条链路一直传到底,`ProcessInfo.processInfo.environment["XUNFEI_X"]` 在 app 内能拿到。但有几种场景会丢:
- macOS 14+ 的 `open --new` 在某些沙盒场景下被 launchd 接管,env 被截断 → 备用 `xop run --secret ... -- /Applications/VoiceType.app/Contents/MacOS/VoiceType`(直接执行二进制,TCC 仍生效,因为同一 .app bundle)
- 用户从 Finder 双击 .app:不走 xop,env 没有,引擎立刻报错退出
## 复现/验证工具
- 事件 tap 全列表:`CGGetEventTapList`(源码曾存 /tmp/taplist.swift,随 /tmp 已失传,可按 GOTCHAS §2 结论重写)
- Fn/左键轮询探针:`CGEventSource.flagsState/.buttonState` + 8ms 循环
- 日志判据:每次判定一行 `activate/skip/allow`,带 app bundle id + AX 角色/错误码
+100
View File
@@ -0,0 +1,100 @@
# VoiceType — 长按左键语音输入(原型文档)
灵感来自微信 Mac 版的"按住说话":在**任意 app 的输入框**长按鼠标左键说话,松开文字自动出现在光标处。本工具把它做成全局的,且微信窗口内自动放行。
## 使用
1. `src/build.sh` 构建安装(首次需手动开「系统设置 → 隐私与安全性 → 辅助功能 → VoiceType」,麦克风/语音识别弹窗点允许)
2. 点击输入框定位光标 → **按住左键 ≥0.35s**(哔一声开始)→ 说话 → **松开** → 出字
3. 位移超过 10px 视为拖选,不触发;微信窗口内长按交给微信自己的语音输入
日志:`/tmp/voicetype.log`(每次判定都记录走了哪条路径、看到什么角色、识别文本、粘贴结果)。
## 引擎选择
`~/.voicetype-engine` 文件内容决定用哪个 ASR 引擎(空文件 = `apple-llm`):
| 值 | 引擎 | 备注 |
|---|---|---|
| `apple-llm` (默认) | Apple 端侧 + LLM 异步精修 | 无凭据;LLM 走 `~/ssw.ini`;无 INI 则回退 Apple 原文 |
| `apple` | SFSpeechRecognizer 端侧 zh-CN | 离线,无凭据,响应快,无 LLM 清洗 |
| `xunfei` | 讯飞 IAT v2 WebSocket 流式 | **需凭据**;首次须 `xop set` 三个 vault 条目,启动走 `src/build.sh xunfei` |
| `dual` | apple + xunfei 并行 + LLM 精修 | 苹果先贴,讯飞覆盖,LLM 再精修 |
### apple-llm / dual 的 LLM 配置(`~/ssw.ini`)
LLM 走 OpenAI Chat Completions 兼容协议(任何 zhipu/openai/anthropic-compatible 代理都行)。
```ini
[llm-api]
key=sk_xxxxxxxxxxxx
base-url=https://your-llm-proxy/v1
model=GLM-5.3-Flash
```
容忍全角破折号 `—` 与半角 `-` 等价(`[llm—api]` / `base—url` 都能识别)。
引擎会自动在请求里带 `thinking: {type: "disabled"`,绕开 GLM 系列推理模式(7s+ → 1s,实测 `thinking=disabled` 后稳定 ~0.9s)。
**失败兜底**: 无 INI / key 错 / 网络断 / 超时 → 保留 Apple 原文,绝不静默丢内容。
### 一次性:把讯飞凭据放进 xop vault(仅 xunfei/dual 需要)
凭据**只**从 TTY 读取,不会出现在源码/git/日志/argv/历史里。`xop set <name>` 然后回车输入值。
```bash
xop set xf_vt_APPID # APPID
xop set xf_vt_APISecret # APISecret
xop set xf_vt_APIKey # APIKey
```
### 启动对应引擎
```bash
src/build.sh # 默认 = apple-llm
src/build.sh apple # 纯端侧(无 LLM 精修)
src/build.sh apple-llm # 端侧 + LLM(默认)
src/build.sh xunfei # 讯飞(需 xop 注入)
src/build.sh dual # 双引擎(需 xop 注入)
```
这一步做三件事:
1. 把 `~/.voicetype-engine` 写成对应模式(持久切换)
2. `xunfei` / `dual` 模式会 `xop run --secret XUNFEI_APPID=xf_vt_APPID --secret ... -- open /Applications/VoiceType.app` —— 凭据仅在子进程 env 里存在
3. 引擎读 `ProcessInfo.processInfo.environment["XUNFEI_*"]`,**任一缺失则报错并丢弃本次录音**(不静默回退 apple,避免你以为用了讯飞但实际是 apple 给你识别差)。
## 判定流程(按住 350ms 时触发一次)
```
光标下窗口归属(CGWindowList z序)
├─ 微信(com.tencent.xinWeChat*) → 放行,静默
└─ 其他 app:
① AX 聚焦元素角色 ∈ {AXTextField, AXTextArea, AXComboBox, AXSearchField} → 激活
② 查不到 → 设置 AXManualAccessibility + AXEnhancedUserInterface,250ms 后重查 → 同①
③ 仍查不到(AX 完全不透明,如 HexHub) → app 级兜底放行
```
## 架构(单文件)
| 组件 | 职责 | 权限 |
|---|---|---|
| 轮询线程(~125Hz) | `CGEventSource.buttonState` 检测左键,状态机 idle→press→recording/dead | 无 |
| 预卷录音 | **鼠标按下瞬间**就开始收音(`AVAudioEngine` tap),激活时才挂识别任务——短句不丢头 | 麦克风 |
| `AppleEngine` | `SFSpeechRecognizer(zh-CN)` 流式,`requiresOnDeviceRecognition = true`,松开 ~70ms 出结果 | 语音识别 |
| `AppleLLMEngine` | 包 AppleEngine + LLMCleaner;Apple 出文立刻贴,LLM 异步 ~1s 后回来若非空且不同则退格再贴 | 语音识别 + LLM |
| `XunfeiEngine` | `URLSessionWebSocketTask` → `wss://iat-api.xfyun.cn/v2/iat`,HMAC-SHA256 鉴权 URL,系统原生 PCM → 16k/16bit/mono 后 1280 字节/帧推送,逐条收 `data_status` 拼结果 | 麦克风(无语音识别权限要求) |
| `DualEngine` | apple + xunfei 并行,apple 先贴 → xunfei 覆盖 → LLM 异步精修再覆盖(可关) | 语音识别 + 麦克风 + LLM |
| `LLMCleaner` | OpenAI Chat Completions 兼容;强制带 `thinking: disabled`;`refine()` 异步回调,`clean()` 同步 1.5s 兜底 | — |
| `pickEngine()` | 读 `~/.voicetype-engine`,默认 `apple-llm` | — |
| 注入 | 写剪贴板 → 合成 ⌘V → 1.5s 后恢复原剪贴板 | 辅助功能 |
| 拖选抑制 tap | 录音期间吞掉 leftMouseDragged,防拖选文字 | 辅助功能 |
| 聚焦查询 | AX 系统级/per-app `kAXFocusedUIElementAttribute` + 角色 | 辅助功能 |
## 已知边界(探针范围外)
- AX 完全不透明的 app 里是 **app 级兜底**:长按它任何位置都触发,不只输入框
- 剪贴板恢复只管文本(图片/文件内容不还原)
- 录音上限 60s;350ms 内慢速起步(<10px)的拖选理论可能误判
- LSUIElement 无界面:无菜单栏开关、无开机自启、无常量配置面板(阈值都是源码顶部常量)
- 讯飞 IAT v2 单次请求 60s 上限(服务端);长语音会被截断
- 讯飞计费按调用次数+音频时长,开发期注意用量
+15
View File
@@ -0,0 +1,15 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleIdentifier</key><string>com.xs.voicetype</string>
<key>CFBundleName</key><string>VoiceType</string>
<key>CFBundleExecutable</key><string>VoiceType</string>
<key>CFBundlePackageType</key><string>APPL</string>
<key>CFBundleShortVersionString</key><string>0.1</string>
<key>LSMinimumSystemVersion</key><string>14.0</string>
<key>LSUIElement</key><true/>
<key>NSMicrophoneUsageDescription</key><string>长按鼠标左键时录音进行语音输入</string>
<key>NSSpeechRecognitionUsageDescription</key><string>将录音转成文字</string>
</dict>
</plist>
Executable
+86
View File
@@ -0,0 +1,86 @@
#!/bin/bash
# 构建 VoiceType.app 并安装到 /Applications
# 红线: 必须用固定 Apple Development 证书签名 —— ad-hoc(-s -) 每次重编译身份都变,
# TCC 会当成新 app,麦克风/语音/辅助功能授权全部作废
#
# 用法:
# src/build.sh 编译+签名+安装,默认启动 apple-llm(端侧+LLM 精修)
# src/build.sh apple 纯端侧(无 LLM,最低延迟)
# src/build.sh apple-llm 端侧识别 + LLM 异步精修(默认;需 ~/ssw.ini [llm-api])
# src/build.sh xunfei 讯飞识别(端点 ~1s)
# src/build.sh dual apple + xunfei 并行 + LLM 精修(xunfei 覆盖后)
#
# 引擎持久切换(两种启动方式之外):
# echo apple > ~/.voicetype-engine
# echo apple-llm > ~/.voicetype-engine
# echo xunfei > ~/.voicetype-engine
# echo dual > ~/.voicetype-engine
#
# 凭据(讯飞,一次性,只需跑一次,xop 从 TTY 读值):
# xop set xf_vt_APPID # APPID
# xop set xf_vt_APISecret # APISecret
# xop set xf_vt_APIKey # APIKey
#
# LLM(apple-llm/dual 需要,否则静默回退到 apple 原文):
# ~/ssw.ini → [llm-api] key=… base-url=… model=…
# (apple-llm 无 ini 也照常工作,只是不精修)
set -e
cd "$(dirname "$0")"
IDENT="Apple Development: [email protected] (Y4AJXC7Q9S)"
APP=/Applications/VoiceType.app
BIN=/tmp/voicetype-build
MODE="${1:-apple-llm}"
swiftc -O voicetype.swift -o "$BIN"
mkdir -p "$APP/Contents/MacOS"
cp "$BIN" "$APP/Contents/MacOS/VoiceType"
cp Info.plist "$APP/Contents/Info.plist"
codesign --force --sign "$IDENT" "$APP"
killall VoiceType 2>/dev/null || true
sleep 0.3
: > /tmp/voicetype.log
case "$MODE" in
xunfei)
# 把 ~/.voicetype-engine 也设上,使得日后 `open /Applications/VoiceType.app`
# 直接双击也能继续用 xunfei。凭据永远只走 xop 注入,不落盘不传 argv。
current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )"
[ "$current" != "xunfei" ] && echo xunfei > ~/.voicetype-engine
xop run \
--secret XUNFEI_APPID=xf_vt_APPID \
--secret XUNFEI_API_SECRET=xf_vt_APISecret \
--secret XUNFEI_API_KEY=xf_vt_APIKey \
-- open "$APP"
;;
dual)
current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )"
[ "$current" != "dual" ] && echo dual > ~/.voicetype-engine
xop run \
--secret XUNFEI_APPID=xf_vt_APPID \
--secret XUNFEI_API_SECRET=xf_vt_APISecret \
--secret XUNFEI_API_KEY=xf_vt_APIKey \
-- open "$APP"
;;
apple)
current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )"
[ -n "$current" ] && [ "$current" != "apple" ] && echo apple > ~/.voicetype-engine
open "$APP"
;;
apple-llm|*)
current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )"
[ -n "$current" ] && [ "$current" != "apple-llm" ] && echo apple-llm > ~/.voicetype-engine
[ -z "$current" ] && echo apple-llm > ~/.voicetype-engine
open "$APP"
;;
esac
sleep 2
cat /tmp/voicetype.log
echo
echo "installed: $APP (pid $(pgrep -f VoiceType)) engine=$MODE"
[ "$MODE" = "xunfei" ] || [ "$MODE" = "dual" ] && echo "vault used: xf_vt_APPID / xf_vt_APISecret / xf_vt_APIKey (no values in logs/argv)"
[ "$MODE" = "apple-llm" ] && grep -q '\[llm-api\]' ~/ssw.ini 2>/dev/null && echo "llm: ~/ssw.ini [llm-api] 已配 → LLM 异步精修生效;否则回退 apple 原文"
+910
View File
@@ -0,0 +1,910 @@
// voicetype: 长按鼠标左键 → 说话 → 松开自动粘贴到光标处
// 引擎可选: apple / xunfei / dual(apple+xunfei 并行,apple 先贴)
// 凭据(仅 xunfei 需要): 经 xop run --secret 注入到环境变量,源码/git/日志绝不出现
// LLM 清洗(可选): ~/ssw.ini 的 [llm-api] 段,key/base-url/model;无配置则跳过
// 权限: 麦克风 + 语音识别(apple) / 麦克风 + 辅助功能(全模式)
import AppKit
import ApplicationServices
import AVFoundation
import CoreGraphics
import CryptoKit
import Speech
// ---------- 配置 ----------
let holdMs: Double = 350 // 长按激活阈值
let movePx: CGFloat = 10 // 位移超过此值视为拖选,不激活
let maxRecSec: Double = 60 // 最长录音(到点直接贴当前已识别部分,不继续等)
let pollUs: UInt32 = 8_000 // 轮询周期 8ms (~125Hz)
let weChatPrefix = "com.tencent.xinWeChat"
let engineConfigPath = NSHomeDirectory() + "/.voicetype-engine"
let llmIniPath = NSHomeDirectory() + "/ssw.ini"
// ---------- INI 读取(仅支持 section/key=value 简单格式) ----------
// 容忍 section/key 名里的全角破折号 `—` 与半角 `-` 等价(~/ssw.ini 历史上用 —)
func normINIKey(_ s: String) -> String {
s.replacingOccurrences(of: "—", with: "-").trimmingCharacters(in: .whitespaces)
}
func loadINI(_ path: String) -> [String: [String: String]] {
guard let raw = try? String(contentsOfFile: path, encoding: .utf8) else { return [:] }
var sections: [String: [String: String]] = [:]
var current: String?
for line in raw.split(whereSeparator: { $0 == "\n" || $0 == "\r" }) {
let s = line.trimmingCharacters(in: .whitespaces)
if s.isEmpty || s.hasPrefix("#") || s.hasPrefix(";") { continue }
if s.hasPrefix("[") && s.hasSuffix("]") {
current = normINIKey(String(s.dropFirst().dropLast()))
sections[current!] = [:]
} else if let cur = current, let eq = s.firstIndex(of: "=") {
let k = normINIKey(String(s[..<eq]))
let v = String(s[s.index(after: eq)...]).trimmingCharacters(in: .whitespaces)
sections[cur]?[k] = v
}
}
return sections
}
struct LLMConfig {
let key: String
let baseURL: String // 例如 https://api.openai.com/v1
let model: String // 例如 gpt-4o-mini
static func load() -> LLMConfig? {
let s = loadINI(llmIniPath)["llm-api"] ?? [:]
guard let k = s["key"], !k.isEmpty,
let u = s["base-url"], !u.isEmpty,
let m = s["model"], !m.isEmpty else {
log("llm: ~/ssw.ini [llm-api] 缺失或字段不全,跳过 LLM 清洗")
return nil
}
log("llm: 加载 [llm-api] model=\(m) base=\(u.prefix(30))…")
return LLMConfig(key: k, baseURL: u, model: m)
}
}
// 讯飞 IAT v2(参考 xfyun 开放平台"语音听写(流式版)WebAPI")
let xunfeiURL = "wss://iat-api.xfyun.cn/v2/iat"
let xunfeiHost = "iat-api.xfyun.cn"
setbuf(stdout, nil)
func ts() -> String { let f = DateFormatter(); f.dateFormat = "HH:mm:ss.SSS"; return f.string(from: Date()) }
func log(_ s: String) { print("[\(ts())] \(s)") }
// ---------- 共享状态(仅 poller 线程写) ----------
final class Box<T> { var v: T; init(_ v: T) { self.v = v } }
enum St { case idle; case press(origin: CGPoint, start: Date); case recording(origin: CGPoint, start: Date); case dead }
let state = Box<St>(.idle)
let maxMove = Box<CGFloat>(0)
let suppressDrag = Box<Bool>(false)
// ---------- ASR 引擎抽象 ----------
protocol ASREngine {
func prepare() // 鼠标按下瞬间: 预卷/连WS
func beginTask() // 长按激活: 开始识别
func cancel() // 普通点击抬起: 丢弃
func stop() -> String // 录音中抬起: 阻塞等最终结果(dual 模式下返回 apple 的)
func flushNow() -> String // force-stop / maxRecSec: 不等,直接拿当前累计文本
var readyForBegin: Bool { get } // prepare() 是否成功(凭据检查等)
}
func pickEngine() -> ASREngine {
let raw = (try? String(contentsOfFile: engineConfigPath, encoding: .utf8))?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? "apple-llm"
switch raw {
case "xunfei":
log("engine: xunfei (require XUNFEI_APPID/API_SECRET/API_KEY in env)")
return XunfeiEngine()
case "dual":
log("engine: dual (apple first, xunfei overwrite later, llm refine)")
return DualEngine()
case "apple-llm", "llm":
log("engine: apple + LLM refine (~/ssw.ini [llm-api] required, no INI → silent fallback to apple)")
return AppleLLMEngine()
default:
log("engine: apple (on-device, no LLM)")
return AppleEngine()
}
}
final class Engine {
static var current: ASREngine = pickEngine()
static var ready: Bool { current.readyForBegin }
}
// ---------- Apple 端侧 SFSpeechRecognizer ----------
final class AppleEngine: ASREngine {
static let q = DispatchQueue(label: "voice.apple.rec")
let engine = AVAudioEngine()
var request: SFSpeechAudioBufferRecognitionRequest?
var task: SFSpeechRecognitionTask?
var recognizer = SFSpeechRecognizer(locale: Locale(identifier: "zh-CN"))
var text = ""
var finalSem = DispatchSemaphore(value: 0)
var prepared = false
var readyForBegin: Bool { prepared }
func prepare() {
text = ""
finalSem = DispatchSemaphore(value: 0)
let req = SFSpeechAudioBufferRecognitionRequest()
req.shouldReportPartialResults = true
req.addsPunctuation = true
if recognizer?.supportsOnDeviceRecognition == true {
req.requiresOnDeviceRecognition = true
} else {
log("apple: on-device unavailable → server fallback")
}
let input = engine.inputNode
let fmt = input.outputFormat(forBus: 0)
input.removeTap(onBus: 0)
input.installTap(onBus: 0, bufferSize: 2048, format: fmt) { [req] buf, _ in
req.append(buf)
}
request = req
do { try engine.start() } catch { log("apple engine start failed: \(error)") }
prepared = true
}
func beginTask() {
guard let req = request, let rec = recognizer, prepared else { return }
task = rec.recognitionTask(with: req) { [weak self] r, err in
guard let self = self else { return }
if let r = r {
let t = r.bestTranscription.formattedString
if !t.isEmpty { self.text = t }
if r.isFinal { self.finalSem.signal() }
}
if err != nil { log("ASR err: \(err!)"); self.finalSem.signal() }
}
log("apple: recording (on-device: \(self.recognizer?.supportsOnDeviceRecognition == true))")
}
func cancel() {
guard prepared else { return }
engine.inputNode.removeTap(onBus: 0)
engine.stop()
request = nil
task = nil
prepared = false
}
func flushNow() -> String {
// SFSpeech 不太好中途取结果;返回当前 text(可能部分)
let t = text
cancel()
return t
}
func stop() -> String {
engine.inputNode.removeTap(onBus: 0)
engine.stop()
request?.endAudio()
_ = finalSem.wait(timeout: .now() + 3)
task?.finish()
let t = text
text = ""
request = nil
task = nil
prepared = false
return t
}
}
// ---------- Apple 端侧 + LLM 异步精修 ----------
// 流程: Apple 识别 → 贴出原文 → LLM 异步精修 → 若非空且不同则删原文再贴精修版
// 失败/超时/未配 INI: 不精修,保留 Apple 原文(不翻车)
final class AppleLLMEngine: ASREngine {
let apple = AppleEngine()
let cleaner: LLMCleaner
var lastRaw = "" // 覆盖阶段用: 知道要删多少字
init() {
self.cleaner = LLMCleaner(cfg: LLMConfig.load())
}
var readyForBegin: Bool { apple.readyForBegin }
func prepare() { apple.prepare() }
func beginTask() { apple.beginTask() }
func cancel() { apple.cancel() }
func flushNow() -> String { return apple.flushNow() }
func stop() -> String {
let raw = apple.stop()
lastRaw = raw
// 异步 LLM 精修: 非空且不同 → 删原文再贴
DispatchQueue.global().async { [weak self] in
guard let self = self else { return }
self.cleaner.refine(raw) { [weak self] refined in
guard let self = self else { return }
guard !refined.isEmpty, refined != raw else {
log("apple-llm: refine skipped (empty or identical)")
return
}
log("apple-llm: llm refine overwrite (\(raw.count)→\(refined.count) chars)")
self.deleteChars(raw.count)
paste(refined)
}
}
return raw
}
private func deleteChars(_ n: Int) {
guard n > 0 else { return }
let src = CGEventSource(stateID: .combinedSessionState)
for _ in 0..<n {
let kd = CGEvent(keyboardEventSource: src, virtualKey: 51, keyDown: true)
kd?.post(tap: .cghidEventTap)
let ku = CGEvent(keyboardEventSource: src, virtualKey: 51, keyDown: false)
ku?.post(tap: .cghidEventTap)
usleep(3_000)
}
}
}
// 凭据来自环境变量(XUNFEI_APPID / XUNFEI_API_SECRET / XUNFEI_API_KEY),
// 由 xop run --secret 在启动前注入。引擎自身不读配置文件、不打 fallback —— 安静回退会让用户误以为"用了讯飞识别差"。
final class XunfeiEngine: NSObject, ASREngine {
let q = DispatchQueue(label: "voice.xunfei")
let engine = AVAudioEngine()
var session: URLSession?
var wsTask: URLSessionWebSocketTask?
var pcmBuf = Data() // 16k Int16 mono PCM 累积
var rawBuf = Data() // 硬件格式 PCM 累积(Float32 mono @ hwRate,待手动转换)
var hwFmt: AVAudioFormat? // 输入节点硬件格式(用于算下采样比)
var resultText = "" // 累积识别结果
var waitingFinal = false // status=2 已发,等任何一条后续回复即视为结束
var firstFrameSent = false
var frameTimer: DispatchSourceTimer?
var finalSem = DispatchSemaphore(value: 0)
var openedSem = DispatchSemaphore(value: 0)
var opened = false
var prepared = false // prepare() 成功(ws open + lastErr==nil)就 true
var lastErr: String?
var appID = ""
var readyForBegin: Bool { prepared }
private func envCreds() -> (id: String, secret: String, key: String)? {
let env = ProcessInfo.processInfo.environment
guard let id = env["XUNFEI_APPID"], !id.isEmpty,
let s = env["XUNFEI_API_SECRET"], !s.isEmpty,
let k = env["XUNFEI_API_KEY"], !k.isEmpty else { return nil }
return (id, s, k)
}
private func rfc1123() -> String {
let f = DateFormatter()
f.locale = Locale(identifier: "en_US_POSIX")
f.timeZone = TimeZone(identifier: "GMT")
f.dateFormat = "EEE, dd MMM yyyy HH:mm:ss z"
return f.string(from: Date())
}
// HMAC-SHA256 鉴权 URL(同步在 IAT v2 文档)
private func authURL(appID: String, secret: String, apiKey: String) -> URL? {
let date = rfc1123()
let signOrigin = "host: \(xunfeiHost)\ndate: \(date)\nGET /v2/iat HTTP/1.1"
let mac = HMAC<SHA256>.authenticationCode(for: Data(signOrigin.utf8), using: SymmetricKey(data: Data(secret.utf8)))
let sig = Data(mac).base64EncodedString()
let authOrigin = "api_key=\"\(apiKey)\", algorithm=\"hmac-sha256\", headers=\"host date request-line\", signature=\"\(sig)\""
let auth = Data(authOrigin.utf8).base64EncodedString()
return URL(string: "\(xunfeiURL)?authorization=\(auth)&date=\(date)&host=\(xunfeiHost)")
}
func prepare() {
// 先收尾旧的(session、wsTask),避免和上一轮并发触发 delegate 回调
cleanup()
guard let creds = envCreds() else {
lastErr = "missing env XUNFEI_APPID/API_SECRET/API_KEY — launch via `xop run --secret`"
log("XUNFEI: \(lastErr!)")
return
}
appID = creds.id
// tap 必须用输入节点硬件格式 (Float32, 通常 44.1k 或 48k) —— 不然 installTap 抛 NSException
// 转码(浮点→int16 + 下采样)放到 serial queue 上做,绝不在音频线程跑。
let input = engine.inputNode
hwFmt = input.outputFormat(forBus: 0)
guard let url = authURL(appID: creds.id, secret: creds.secret, apiKey: creds.key) else {
lastErr = "auth URL build failed"
log("XUNFEI: \(lastErr!)")
return
}
// 用串行 queue 保证 delegate 回调按序触发
let delegateQ = OperationQueue()
delegateQ.maxConcurrentOperationCount = 1
session = URLSession(configuration: .default, delegate: self, delegateQueue: delegateQ)
wsTask = session!.webSocketTask(with: url)
opened = false
wsTask?.resume()
// tap 回调是音频实时线程,绝对禁止在这里 log / 阻塞 / 分配大对象。
// 只做一次 unsafe 指针复制 + 把字节扔到 serial queue。
engine.inputNode.removeTap(onBus: 0)
engine.inputNode.installTap(onBus: 0, bufferSize: 2048, format: hwFmt!) { [weak self] buf, _ in
guard let self = self,
let ch = buf.floatChannelData?[0] else { return }
let n = Int(buf.frameLength)
if n == 0 { return }
let raw = UnsafeRawBufferPointer(start: ch, count: n * 4) // Float32 = 4 bytes
let copy = Data(raw) // 拷贝出音频线程作用域
self.q.async { self.rawBuf.append(copy); self.drainAndConvert() }
}
do { try engine.start() } catch {
lastErr = "engine start failed: \(error)"
log("XUNFEI: \(lastErr!)")
return
}
prepared = true
log("xunfei: ws connecting + recording (hw=\(hwFmt!.sampleRate)Hz fmt=\(hwFmt!.commonFormat.rawValue))")
}
private func cleanup() {
frameTimer?.cancel(); frameTimer = nil
wsTask?.cancel(with: .goingAway, reason: nil)
wsTask = nil
session?.invalidateAndCancel()
session = nil
engine.inputNode.removeTap(onBus: 0)
if engine.isRunning { engine.stop() }
pcmBuf = Data(); rawBuf = Data(); resultText = ""
firstFrameSent = false
hwFmt = nil
waitingFinal = false
finalSem = DispatchSemaphore(value: 0)
openedSem = DispatchSemaphore(value: 0)
opened = false
}
private func sendFrames() {
q.async {
// 先把硬件格式的 rawBuf 转码成 16k int16 mono 进 pcmBuf
self.drainAndConvert()
// 1280 bytes = 640 samples × 2 bytes/sample = 40ms @ 16kHz
while self.pcmBuf.count >= 1280 {
guard let ws = self.wsTask, ws.state == .running else { return } // ws 还没握上,等下一帧再试
let chunk = self.pcmBuf.prefix(1280)
self.pcmBuf.removeFirst(1280)
let isFirst = !self.firstFrameSent
self.send(status: self.firstFrameSent ? 1 : 0, audio: Data(chunk))
self.firstFrameSent = true
if isFirst { log("xunfei: sent first chunk (\(chunk.count)B) ws.state=\(ws.state.rawValue)") }
}
}
}
// 把 rawBuf(硬件格式 Float32 mono @ hwRate)手动转成 16k int16 mono,入 pcmBuf。
// 只在 serial queue 上跑,绝不在音频线程跑。
// 解: Float32 -> Int16 (clip); 48k -> 16k 用 3:1 抽取(够 ASR 用,免 AVAudioConverter 崩)
private func drainAndConvert() {
guard hwFmt != nil else { return }
let hwFrameBytes = 4 // Float32 mono = 4 bytes/frame
let frameCount = rawBuf.count / hwFrameBytes
guard frameCount >= 3 else { return } // 至少一帧 16k 输出
let ratio = Int((hwFmt!.sampleRate / 16000).rounded())
let outFrames = frameCount / max(ratio, 1)
guard outFrames > 0 else { return }
rawBuf.withUnsafeBytes { rawPtr in
let floats = rawPtr.bindMemory(to: Float32.self).baseAddress!
var out = [Int16](repeating: 0, count: outFrames)
for i in 0..<outFrames {
let s = floats[i * ratio]
let clipped = max(-1.0, min(1.0, s))
out[i] = Int16(clipped * 32767)
}
out.withUnsafeBufferPointer { bp in
pcmBuf.append(Data(buffer: bp))
}
}
rawBuf.removeFirst(outFrames * ratio * hwFrameBytes)
}
private func send(status: Int, audio: Data) {
guard let ws = wsTask else { return }
let payload: [String: Any] = [
"common": ["app_id": appID],
"business": ["language": "zh_cn", "domain": "iat", "accent": "mandarin", "vad_eos": 10000, "ptt": 1],
"data": ["status": status, "format": "audio/L16;rate=16000", "encoding": "raw", "audio": audio.base64EncodedString()]
]
guard let data = try? JSONSerialization.data(withJSONObject: payload) else { return }
log("xunfei: send status=\(status) bytes=\(audio.count) ws=\(ws.state.rawValue)")
ws.send(.data(data)) { err in
if let err = err { log("xunfei send err: \(err)") }
}
}
private func scheduleRecv() {
wsTask?.receive { [weak self] result in
guard let self = self else { return }
switch result {
case .success(let msg):
self.handle(msg)
if self.lastErr == nil { self.scheduleRecv() }
case .failure(let err):
log("xunfei recv err: \(err)")
self.lastErr = err.localizedDescription
self.finalSem.signal()
}
}
}
private func handle(_ msg: URLSessionWebSocketTask.Message) {
let data: Data?
switch msg {
case .string(let s): data = s.data(using: .utf8)
case .data(let d): data = d
@unknown default: data = nil
}
guard let d = data, let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any] else { return }
let code = j["code"] as? Int ?? -1
let payload = j["data"] as? [String: Any]
log("xunfei recv: code=\(code) payload=\(payload ?? [:])")
if code != 0 {
lastErr = "信令码 \(code) \(j["message"] ?? "")"
log("XUNFEI: \(lastErr!)")
finalSem.signal()
return
}
guard let payload = j["data"] as? [String: Any] else { return }
if let result = payload["result"] as? [String: Any],
let wsArr = result["ws"] as? [[String: Any]] {
var piece = ""
for w in wsArr {
if let cw = w["cw"] as? [[String: Any]] {
for c in cw {
if let ww = c["w"] as? String { piece += ww }
}
}
}
// pgs: "rpl" 替换全文;"apd" 追加。"apd" 时将 piece 接到尾部
let pgs = result["pgs"] as? String ?? "rpl"
if pgs == "apd" { resultText += piece } else { resultText = piece }
log("xunfei recv rg: pgs=\(pgs) piece=\"\(piece)\" total=\"\(resultText)\" ls=\(payload["ls"] ?? false)")
}
if (payload["ls"] as? Bool) == true || waitingFinal {
// status=2 之后任何一条回复 = 服务端把最终答案给我了;ls=true 也算
finalSem.signal()
}
}
func beginTask() {
guard prepared, lastErr == nil else {
log("XUNFEI: beginTask refused (not prepared: \(lastErr ?? "?"))")
return
}
// 每 40ms 切一帧推上去
let t = DispatchSource.makeTimerSource(queue: q)
t.schedule(deadline: .now() + 0.04, repeating: 0.04)
t.setEventHandler { [weak self] in self?.sendFrames() }
t.resume()
frameTimer = t
scheduleRecv()
log("xunfei: begin")
}
func cancel() {
cleanup()
lastErr = nil
prepared = false
}
func flushNow() -> String {
// 不发 status=2、不等;直接返回当前 resultText(中间识别累计)
let t = resultText
cleanup()
lastErr = nil
prepared = false
return t
}
func stop() -> String {
frameTimer?.cancel(); frameTimer = nil
if prepared, wsTask != nil {
// 最后一帧: 先把 rawBuf 转完,再把 pcmBuf 尾巴推上去 + status=2
waitingFinal = true // 发完 status=2 后,handle() 收到任何一条回复就 signal
q.sync {
self.drainAndConvert()
let tail = self.pcmBuf
self.pcmBuf = Data()
self.send(status: 2, audio: tail)
}
// 等服务端回完最终那条(经验 100~500ms),1s 兜底(waitingFinal + ws close 都会 signal)
_ = finalSem.wait(timeout: .now() + 1)
}
// 必须在 cleanup 前读 resultText —— cleanup 会把它清空
let t = resultText
cleanup()
lastErr = nil
prepared = false
return t
}
}
// ---------- LLM 清洗(可选):OpenAI Chat Completions 兼容 ----------
final class LLMCleaner {
let cfg: LLMConfig?
init(cfg: LLMConfig?) { self.cfg = cfg }
static let prompt = """
你是语音转写后处理助手。把下面这段口述语音转写文字清理成自然书面语中文:
- 去掉"嗯/啊/呃/那个/就是/然后/其实/对"等口水词
- 加标点(句号、逗号、问号等),分句
- 重组零碎短句为通顺长句
- 保留全部原意,不增删内容
只输出清理后的文字,不要任何解释、标签或前后缀。
"""
func clean(_ raw: String) -> String {
// 同步接口(给单引擎调用):1.5s 兜底超时,默认返 raw
guard let cfg = cfg, !raw.isEmpty else { return raw }
let url = URL(string: cfg.baseURL.trimmingCharacters(in: CharacterSet(charactersIn: "/")) + "/chat/completions")!
var req = URLRequest(url: url, timeoutInterval: 1.5)
req.httpMethod = "POST"
req.setValue("application/json", forHTTPHeaderField: "Content-Type")
req.setValue("Bearer \(cfg.key)", forHTTPHeaderField: "Authorization")
var body: [String: Any] = [
"model": cfg.model,
"messages": [
["role": "system", "content": Self.prompt],
["role": "user", "content": raw],
],
"temperature": 0.1,
]
// 强制关推理: GLM-5.3-Flash 默认有 reasoning_content,7s+;关掉 <1s 且内容字段干净。
// 不支持 thinking 的代理会静默忽略。
body["thinking"] = ["type": "disabled"]
req.httpBody = try? JSONSerialization.data(withJSONObject: body)
let sem = DispatchSemaphore(value: 0)
var result = raw
URLSession.shared.dataTask(with: req) { data, _, _ in
defer { sem.signal() }
guard let data = data,
let j = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let choices = j["choices"] as? [[String: Any]],
let msg = choices.first?["message"] as? [String: Any],
let content = msg["content"] as? String else { return }
let cleaned = content.trimmingCharacters(in: .whitespacesAndNewlines)
if !cleaned.isEmpty { result = cleaned }
}.resume()
_ = sem.wait(timeout: .now() + 1.5)
return result
}
// 异步精修(给 apple-llm / dual 引擎用):长 timeout,回调返 cleaned
func refine(_ raw: String, timeout: TimeInterval = 3, _ cb: @escaping (String) -> Void) {
guard let cfg = cfg, !raw.isEmpty else { cb(raw); return }
let url = URL(string: cfg.baseURL.trimmingCharacters(in: CharacterSet(charactersIn: "/")) + "/chat/completions")!
var req = URLRequest(url: url, timeoutInterval: timeout)
req.httpMethod = "POST"
req.setValue("application/json", forHTTPHeaderField: "Content-Type")
req.setValue("Bearer \(cfg.key)", forHTTPHeaderField: "Authorization")
var body: [String: Any] = [
"model": cfg.model,
"messages": [
["role": "system", "content": Self.prompt],
["role": "user", "content": raw],
],
"temperature": 0.1,
]
body["thinking"] = ["type": "disabled"]
req.httpBody = try? JSONSerialization.data(withJSONObject: body)
let done = DispatchSemaphore(value: 0)
URLSession.shared.dataTask(with: req) { data, _, _ in
defer { done.signal() }
guard let data = data,
let j = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let choices = j["choices"] as? [[String: Any]],
let msg = choices.first?["message"] as? [String: Any],
let content = msg["content"] as? String else {
cb(raw); return
}
let cleaned = content.trimmingCharacters(in: .whitespacesAndNewlines)
cb(cleaned.isEmpty ? raw : cleaned)
}.resume()
_ = done.wait(timeout: .now() + timeout + 0.5)
}
}
// ---------- 本地清洗(无) ----------
// LLM 之前实验过本地正则去"嗯/啊/那个"等口水词,但中文没空格,
// 任何字符级启发式都会误伤"好啊""什么啊"等正常语料 → 已弃用,清洗只走 LLM
// ---------- Dual 引擎:apple + xunfei 并行,apple 先贴、xunfei 后续覆盖 ----------
final class DualEngine: ASREngine {
let apple = AppleEngine()
let xunfei = XunfeiEngine()
let cleaner: LLMCleaner
var lastAppleText = "" // 给覆盖阶段用:知道要删多少字
var overwriteScheduled = false
init() {
self.cleaner = LLMCleaner(cfg: LLMConfig.load())
}
var readyForBegin: Bool { apple.readyForBegin && xunfei.readyForBegin }
func prepare() {
apple.prepare()
xunfei.prepare()
}
func beginTask() {
apple.beginTask()
xunfei.beginTask()
}
func cancel() {
apple.cancel()
xunfei.cancel()
}
func flushNow() -> String {
return apple.flushNow()
}
func stop() -> String {
// 1) 收 apple(~1s) → 立刻贴出原始文本(不本地清洗)
let appleRaw = apple.stop()
lastAppleText = appleRaw
// 2) 异步: xunfei 识别完 → 覆盖 apple;再走 LLM 异步精修 → 若更优再覆盖一次
DispatchQueue.global().async { [weak self] in
guard let self = self else { return }
let xfRaw = self.xunfei.stop()
if !xfRaw.isEmpty, xfRaw != appleRaw {
log("dual: xunfei overwrite (\(appleRaw.count)→\(xfRaw.count) chars)")
self.deleteChars(appleRaw.count)
paste(xfRaw)
}
// 3) LLM 异步精修(可关): 配了 ~/ssw.ini 就跑,返非空且不同时再覆盖一次
self.cleaner.refine(xfRaw) { [weak self] refined in
guard let self = self else { return }
guard !refined.isEmpty, refined != xfRaw else { return }
log("dual: llm refine overwrite (\(xfRaw.count)→\(refined.count) chars)")
self.deleteChars(xfRaw.count)
paste(refined)
}
}
return appleRaw
}
private func deleteChars(_ n: Int) {
guard n > 0 else { return }
let src = CGEventSource(stateID: .combinedSessionState)
// 51 = kVK_Delete(退格)
for _ in 0..<n {
let kd = CGEvent(keyboardEventSource: src, virtualKey: 51, keyDown: true)
kd?.post(tap: .cghidEventTap)
let ku = CGEvent(keyboardEventSource: src, virtualKey: 51, keyDown: false)
ku?.post(tap: .cghidEventTap)
usleep(3_000) // 3ms / 字符,13 字 ≈ 40ms
}
}
}
extension XunfeiEngine: URLSessionWebSocketDelegate {
func urlSession(_ session: URLSession, webSocketTask: URLSessionWebSocketTask,
didOpenWithProtocol p: String?) {
opened = true
openedSem.signal()
log("xunfei: ws open")
}
func urlSession(_ session: URLSession, webSocketTask: URLSessionWebSocketTask,
didCloseWith code: URLSessionWebSocketTask.CloseCode, reason: Data?) {
log("xunfei: ws close \(code.rawValue)")
// 对端结束也算"最后一帧到了":把 finalSem 抬起来,stop() 别再傻等
finalSem.signal()
}
}
// ---------- 注入:剪贴板 + 合成 Cmd+V ----------
let vKey: CGKeyCode = 9 // kVK_ANSI_V
func paste(_ s: String) {
let pb = NSPasteboard.general
let old = pb.string(forType: .string)
pb.clearContents()
pb.setString(s, forType: .string)
let src = CGEventSource(stateID: .combinedSessionState)
let kd = CGEvent(keyboardEventSource: src, virtualKey: vKey, keyDown: true)
kd?.flags = .maskCommand
kd?.post(tap: .cghidEventTap)
usleep(60_000)
let ku = CGEvent(keyboardEventSource: src, virtualKey: vKey, keyDown: false)
ku?.flags = .maskCommand
ku?.post(tap: .cghidEventTap)
log("pasted \(s.count) chars -> cursor")
DispatchQueue.global().asyncAfter(deadline: .now() + 1.5) {
if let old = old, pb.changeCount != 0 {
pb.clearContents(); pb.setString(old, forType: .string)
log("clipboard restored")
}
}
}
func finalize() {
suppressDrag.v = false
DispatchQueue.global().async {
let t = Engine.current.stop()
log("transcript: \"\(t)\"")
if !t.isEmpty { paste(t) }
}
}
// ---------- 光标定位 app ----------
func appAtCursor(_ pt: CGPoint) -> (pid: pid_t, bundle: String?) {
guard let list = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID) as? [[String: Any]] else {
return (NSWorkspace.shared.frontmostApplication?.processIdentifier ?? 0,
NSWorkspace.shared.frontmostApplication?.bundleIdentifier)
}
for w in list {
guard (w[kCGWindowLayer as String] as? Int) == 0,
let b = w[kCGWindowBounds as String] as? [String: CGFloat] else { continue }
if CGRect(x: b["X"] ?? 0, y: b["Y"] ?? 0, width: b["Width"] ?? 0, height: b["Height"] ?? 0).contains(pt) {
let pid = pid_t(w[kCGWindowOwnerPID as String] as? Int ?? 0)
return (pid, NSRunningApplication(processIdentifier: pid)?.bundleIdentifier)
}
}
return (0, nil)
}
// ---------- 微信过滤 + AX 输入框判定 ----------
let textRoles: Set<String> = ["AXTextField", "AXTextArea", "AXComboBox", "AXSearchField"]
func frontmostIsWeChat(_ bundle: String?) -> Bool {
bundle?.hasPrefix(weChatPrefix) == true
}
func axCopy(_ el: AXUIElement, _ attr: String, _ tag: String) -> CFTypeRef? {
var val: CFTypeRef?
let err: AXError = AXUIElementCopyAttributeValue(el, attr as CFString, &val)
if err != .success { log(" ax[\(tag)] \(attr) err=\(err.rawValue)") }
return err == .success ? val : nil
}
func axSet(_ el: AXUIElement, _ attr: String, _ v: CFTypeRef) {
_ = AXUIElementSetAttributeValue(el, attr as CFString, v)
}
func nudgeAX(_ pid: pid_t) {
let el = AXUIElementCreateApplication(pid)
axSet(el, "AXManualAccessibility", kCFBooleanTrue)
axSet(el, "AXEnhancedUserInterface", kCFBooleanTrue)
}
func focusedTextRole(pid: pid_t) -> String? {
if let el = axCopy(AXUIElementCreateSystemWide(), kAXFocusedUIElementAttribute, "syswide"),
let role = axCopy(el as! AXUIElement, kAXRoleAttribute, "syswide") as? String {
return role
}
if let el = axCopy(AXUIElementCreateApplication(pid), kAXFocusedUIElementAttribute, "perapp"),
let role = axCopy(el as! AXUIElement, kAXRoleAttribute, "perapp") as? String {
return role
}
return nil
}
// ---------- 长按检测线程 ----------
Thread.detachNewThread {
while true {
let down = CGEventSource.buttonState(.combinedSessionState, button: .left)
let loc = CGEvent(source: nil)?.location ?? .zero
switch (state.v, down) {
case (.idle, true):
state.v = .press(origin: loc, start: Date())
maxMove.v = 0
DispatchQueue.global().async { Engine.current.prepare() }
case (.press(let origin, let start), true):
maxMove.v = max(maxMove.v, hypot(loc.x - origin.x, loc.y - origin.y))
if Date().timeIntervalSince(start) * 1000 >= holdMs {
if maxMove.v <= movePx {
let (pid, bundle) = appAtCursor(loc)
if frontmostIsWeChat(bundle) {
log("skip: WeChat under cursor (\(bundle ?? "?"))")
state.v = .dead
DispatchQueue.global().async { Engine.current.cancel() }
} else if pid != 0 {
var role = focusedTextRole(pid: pid)
if role == nil {
nudgeAX(pid); usleep(250_000)
role = focusedTextRole(pid: pid)
}
func startRec() {
log("activate @ \(bundle ?? "?") role=\(role ?? "opaque")")
NSSound.beep()
if !Engine.ready {
log("engine not ready: \(Engine.current is XunfeiEngine ? "xunfei creds/ws?" : "apple init?")")
state.v = .dead
DispatchQueue.global().async { Engine.current.cancel() }
return
}
suppressDrag.v = true
state.v = .recording(origin: origin, start: start)
DispatchQueue.global().async { Engine.current.beginTask() }
}
if let role, textRoles.contains(role) {
startRec()
} else if let role {
log("skip: role=\(role)")
state.v = .dead
DispatchQueue.global().async { Engine.current.cancel() }
} else {
log("allow: AX-opaque app \(bundle ?? "?") (fallback)")
startRec()
}
} else {
state.v = .dead
DispatchQueue.global().async { Engine.current.cancel() }
}
} else {
state.v = .dead
DispatchQueue.global().async { Engine.current.cancel() }
}
}
case (.recording(_, let start), true):
if Date().timeIntervalSince(start) >= maxRecSec {
log("max record time, force-stop (paste current)")
DispatchQueue.global().async {
let t = Engine.current.flushNow()
log("force-transcript: \"\(t)\"")
if !t.isEmpty { paste(t) }
}
state.v = .idle
}
case (.dead, true):
break
case (_, false):
switch state.v {
case .recording: finalize()
case .press: DispatchQueue.global().async { Engine.current.cancel() }
default: break
}
state.v = .idle
}
usleep(pollUs)
}
}
// ---------- 拖选抑制 tap ----------
let tapPort = Box<CFMachPort?>(nil)
func tapCallback(_ proxy: CGEventTapProxy, _ type: CGEventType, _ ev: CGEvent, _ ud: UnsafeMutableRawPointer?) -> Unmanaged<CGEvent>? {
if type == .tapDisabledByTimeout || type == .tapDisabledByUserInput {
if let p = tapPort.v { CGEvent.tapEnable(tap: p, enable: true) }
return Unmanaged.passUnretained(ev)
}
return suppressDrag.v ? nil : Unmanaged.passUnretained(ev)
}
Thread.detachNewThread {
let mask = CGEventMask(1 << CGEventType.leftMouseDragged.rawValue)
guard let port = CGEvent.tapCreate(tap: .cgSessionEventTap, place: .headInsertEventTap,
options: .defaultTap, eventsOfInterest: mask,
callback: tapCallback, userInfo: nil) else {
log("drag-tap install failed (no Accessibility) — 录音期间移动鼠标会拖选文字")
return
}
let src = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, port, 0)
CFRunLoopAddSource(CFRunLoopGetCurrent(), src, .commonModes)
tapPort.v = port
CFRunLoopRun()
}
// ---------- 启动自检 + 权限请求 ----------
freopen("/tmp/voicetype.log", "a", stdout)
setbuf(stdout, nil)
log("process=\(ProcessInfo.processInfo.processName) bundle=\(Bundle.main.bundleIdentifier ?? "nil")")
log("Accessibility trusted: \(AXIsProcessTrusted())")
if !AXIsProcessTrusted() {
log(">>> 系统设置 → 隐私与安全性 → 辅助功能 → 勾选 VoiceType <<<")
}
AVCaptureDevice.requestAccess(for: .audio) { ok in log("mic granted: \(ok)") }
SFSpeechRecognizer.requestAuthorization { st in log("speech auth: \(st.rawValue) (1=authorized)") }
dispatchMain()