voicetype: Apple 端侧 + LLM 异步精修,可选 xunfei/dual
新增 apple-llm 引擎(默认):Apple SFSpeechRecognizer → 贴原文 → LLM (~1s,
thinking=disabled) → 退格覆盖精修版。LLM 走 ~/ssw.ini [llm-api],无配置
静默回退 Apple 原文。
- 删 localClean(中文无空格,正则误伤'好啊'等正常词)
- pickEngine() 默认 apple-llm
- LLMCleaner 自动带 thinking: {type: 'disabled'} (GLM 7s+ → ~1s)
- build.sh 新增 apple / apple-llm 模式,默认 apple-llm
- README + AGENTS.md 同步更新
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
commit
b380eaf252
7 files changed
+1252
No files matched your search
@@ -0,0 +1,4 @@
|
||||
.DS_Store
|
||||
*.swp
|
||||
*.bak
|
||||
.history/
|
||||
@@ -0,0 +1,67 @@
|
||||
# voicetype / VoiceType
|
||||
|
||||
macOS 工具:**长按鼠标左键 → 说话 → 松开自动粘贴到光标处**(任意 app 输入框;微信窗口内自动放行给微信自己的语音输入)。
|
||||
|
||||
## 布局
|
||||
|
||||
```
|
||||
src/voicetype.swift 全部源码(单文件探针,~330 行)
|
||||
src/Info.plist app 包描述(权限声明)
|
||||
src/build.sh 构建+签名+安装+重启一条龙
|
||||
docs/voicetype/ 详细文档(用法/架构/踩坑)
|
||||
```
|
||||
|
||||
## 常用命令
|
||||
|
||||
```bash
|
||||
src/build.sh # 编译 → 签名 → 装到 /Applications → apple-llm 启动 → 打日志(默认)
|
||||
src/build.sh apple # 纯端侧(无 LLM 精修)
|
||||
src/build.sh apple-llm # 端侧 + LLM 异步精修
|
||||
src/build.sh xunfei # 同上,但经 xop 注入讯飞凭据 → 讯飞 IAT v2 WebSocket 引擎
|
||||
src/build.sh dual # apple + xunfei 并行 + LLM 精修(需 xop 注入)
|
||||
tail -f /tmp/voicetype.log # 看判定/识别/粘贴日志
|
||||
killall VoiceType # 停止
|
||||
```
|
||||
|
||||
## 引擎切换(两个互不重叠的层面)
|
||||
|
||||
1. **持久切换**:`echo apple|apple-llm|xunfei|dual > ~/.voicetype-engine`,任意方式启动都生效。
|
||||
2. **启动方式注入**:`src/build.sh xunfei` 用 `xop run --secret ... -- open ...` 临时把讯飞凭据塞进进程环境变量,**只对当次启动有效**,不会写到磁盘/源码。
|
||||
|
||||
讯飞凭据一次性入库(从 TTY 读值,源码/git/日志都不出现):
|
||||
```bash
|
||||
xop set xf_vt_APPID # APPID
|
||||
xop set xf_vt_APISecret # APISecret
|
||||
xop set xf_vt_APIKey # APIKey
|
||||
```
|
||||
|
||||
LLM 配置(`apple-llm` / `dual` 需要;无 INI 则静默回退 apple 原文):
|
||||
```ini
|
||||
# ~/ssw.ini
|
||||
[llm-api]
|
||||
key=sk_xxxxxxxxxxxx
|
||||
base-url=https://your-llm-proxy/v1
|
||||
model=GLM-5.3-Flash
|
||||
```
|
||||
容忍全角破折号 `—` 与半角 `-` 等价。引擎请求时强制带 `thinking: {type: "disabled"`,绕过 GLM 推理模式(7s → 1s)。
|
||||
|
||||
## 红线(违反必翻车)
|
||||
|
||||
1. **签名必须用固定 Apple Development 证书**(`src/build.sh` 里的 `IDENT`)。ad-hoc `-s -` 每次重编译身份都变,TCC 当成新 app,三项授权全作废。
|
||||
2. **必须以 .app 形式装在 /Applications 并 LaunchServices 注册**。裸 CLI 的 TCC 归属是宿主终端,行为混乱;/tmp 下的 app 进不了辅助功能列表。
|
||||
3. **权限三件套缺一不可**:麦克风(录音)、语音识别(apple 引擎)/ 辅助功能(全模式)。日志 `trusted: true` + `mic granted: true` + `speech auth: 3` 全绿才算就绪。
|
||||
4. **长按状态机:一次按下只判定一次**(激活/跳过后进 `.dead`,松开才复位)。打回 idle 会在按住期间循环重启录音引擎(已踩过)。
|
||||
5. **讯飞凭据永不落盘**:源码/git/日志/argv/历史都不能出现 APPID/APISecret/APIKey 明文。运行时统一经 `xop run --secret KEY=xf_vt_* -- open ...` 注入到 `ProcessInfo.processInfo.environment["XUNFEI_*"]`。引擎拿到空 env 时**直接报错退出**,不静默回退 apple(避免"以为用了讯飞结果识别差"的误判)。
|
||||
6. **LLM 凭据明文存于 `~/ssw.ini`**(用户偏好,不进 xop);`key` 字段不进源码/git/日志。LLM 失败/超时/未配 → 保留 Apple 原文,不静默丢弃。
|
||||
|
||||
## 核心设计(为什么这么做)
|
||||
|
||||
- **鼠标左键轮询零权限可用**(`CGEventSource.buttonState`),Fn/Globe 键则轮询不可见——所以触发方式选左键而不是 Fn(实验证据见 GOTCHAS §1)。
|
||||
- **判定用"光标下最顶层窗口归属"**(CGWindowList z 序),不用 `frontmostApplication`——微信悬浮窗会霸占键盘焦点导致误判(踩坑见 GOTCHAS §5)。
|
||||
- **判定三层**:AX 焦点角色严格判断 → `AXManualAccessibility` 强开 Chromium 系 AX 再查 → 完全不透明则 app 级兜底放行。
|
||||
- **注入 = 写剪贴板 + 合成 ⌘V**,1.5s 后恢复原剪贴板(与微信同款)。
|
||||
- **识别引擎可插拔**:`protocol ASREngine` + `pickEngine()` 读 `~/.voicetype-engine`。默认 **Apple 端侧 SFSpeechRecognizer**(zh-CN,离线流式);备选 **讯飞 IAT v2 流式 WebSocket**(端点 `wss://iat-api.xfyun.cn/v2/iat`,HMAC-SHA256 鉴权,40ms 切片推送,服务端逐条返回拼接)。微信自家 ASR 是私有鉴权协议,外部不可用。
|
||||
|
||||
详细原理与全部踩坑:`docs/voicetype/README.md`、`docs/voicetype/GOTCHAS.md`。
|
||||
|
||||
详细原理与全部踩坑:`docs/voicetype/README.md`、`docs/voicetype/GOTCHAS.md`。
|
||||
@@ -0,0 +1,70 @@
|
||||
# GOTCHAS — VoiceType 踩坑全记录(2026-10-02 spike)
|
||||
|
||||
## 1. Fn(🌐) 键轮询不可见;鼠标左键零权限可见
|
||||
|
||||
写了 `CGEventSource.flagsState` 双视图(combinedSessionState + hidSystemState)轮询探针,用户两轮实按 **8ms 采样粒度下一个翻转都抓不到**——Apple Silicon 的 Globe 键在更底层被系统消费,只有 Fn+字母组合时修饰标志才出现。而 `CGEventSource.buttonState(.combinedSessionState, .left)` 一次就干净抓到左键 DOWN/UP,**不需要任何权限**。→ 这就是触发方式选左键不选 Fn 的实验依据;若将来要做 Fn 触发,必须走辅助功能 CGEventTap(flagsChanged/keycode 63)。
|
||||
|
||||
## 2. 微信的"长按 Fn/鼠标说话"是微信自己的全局劫持
|
||||
|
||||
- 不是 macOS 系统功能;4.1.7+ 加入,授权「控制你的 Mac」后在微信外生效,结果走**剪贴板渠道**注入,可在微信 设置→快捷键 关闭。
|
||||
- 二进制证据(热更新副本 `~/Library/Containers/com.tencent.xinWeChat/Data/Documents/app_data/xplugin/plugins/Magicdy/<ver>/extracted/wechat.dylib` 的导入符号):`mmui::VoiceInputKeyboardListener / VoiceInputMouseListener / VoiceInputMouseSessionController / VoiceInputFocusListener` + `AXIsProcessTrusted` + `CGEventTapCreateForPid`(按前台 app 建 tap,所以不出现在 CGGetEventTapList 全局列表) + `CGEventSourceButtonState/FlagsState`(轮询兜底) + `NSPasteboard`/`CGEventPost`。ASR 走自家 CGI(`voicetrans`/`uploadvoice`),私有鉴权协议,外部不可用 → 本工具用 Apple 端侧识别。
|
||||
- 包内 grep 中文字符串全空:微信 UI 串打包时不可直接 strings,要靠符号表和 `mmui::` 类名。
|
||||
|
||||
## 3. TCC 权限:裸 CLI 归属混乱,必须打包 .app + 固定签名
|
||||
|
||||
- 裸 CLI 从终端跑:麦克风 requestAccess 直接 false,授权对象是"责任进程"(宿主终端),行为不可预期。
|
||||
- 打包最小 .app(Info.plist 带 NSMicrophoneUsageDescription/NSSpeechRecognitionUsageDescription)+ `open` 启动 → TCC 弹窗正常附着,授权立即可用。
|
||||
- **ad-hoc 签名(-s -)每次重编译 cdhash 都变 → TCC 当成新 app,授权作废**。用钥匙串里已有的 Apple Development 证书签名,身份锚定证书,重编译授权保留。
|
||||
- /tmp 下的 app 不进「辅助功能」列表(未注册 LaunchServices)→ 装到 /Applications + `lsregister -f`。
|
||||
|
||||
## 4. AX API 三连坑
|
||||
|
||||
- 新 SDK 把 `AXUIElementCopyAttributeValue` 导入成 **3 参数版**(没有 error 出参):`(AXUIElement, CFString, UnsafeMutablePointer<CFTypeRef?>) -> AXError`。写 4 参报"extra argument in call",用小文件穷举签名确认。
|
||||
- 常量 `kAXFocusedUIElementAttribute` 是 String,要 `as CFString` 传。
|
||||
- 查询失败码要打出来看:`-25204`(CannotComplete)/`-25212`(NoValue=app 没暴露聚焦元素)是完全不同的病因。
|
||||
|
||||
## 5. 判定"长按发生在哪个 app":用光标下窗口,别用 frontmostApplication
|
||||
|
||||
用户在备忘录长按,日志却显示 frontmost=微信——微信悬浮窗/语音面板会**霸占键盘焦点**,`NSWorkspace.frontmostApplication` 跟着错。改为 `CGWindowListCopyWindowInfo(optionOnScreenOnly)` 按 z 序找包含光标点的第一个 layer-0 窗口的 owner pid,所见即所得。
|
||||
|
||||
## 6. Chromium/Electron app 的 AX 树默认不建
|
||||
|
||||
`kAXFocusedUIElementAttribute` 返回 -25212。对 app 元素设置 `AXManualAccessibility` 和 `AXEnhancedUserInterface` = true,~250ms 后重查,多数 Chromium 系会建树。仍不行的(HexHub 实测)只能 app 级兜底放行。
|
||||
|
||||
## 7. 状态机 flap 循环
|
||||
|
||||
按下→判定失败→打回 `.idle`→下一轮询周期又当新按下→重启录音引擎……按住期间每 ~350ms 循环一次,表现为菜单栏录音图标狂闪、永远到不了激活。修法:**一次按下只判定一次**,判定完进 `.dead`,左键抬起才回 `.idle`。
|
||||
|
||||
## 8. 短句丢头 → 预卷录音
|
||||
|
||||
v1 在激活(350ms)后才启动录音引擎,用户按下即开口,前几百毫秒语音丢失 → ASR 报 `No speech detected (1110)`。修法:**鼠标按下瞬间**就启动引擎收缓冲,激活时才挂识别任务(`SFSpeechAudioBufferRecognitionRequest` 允许先 append 后建 task),普通点击抬起则丢弃。
|
||||
|
||||
## 9. 讯飞 IAT v2 鉴权在 URL query 里,不在 header
|
||||
|
||||
`URLSessionWebSocketTask` 用 `webSocketTask(with: URL)` 时**所有参数都得到 URL 上**,没有 header 钩子(Authorization 不能直接挂)。所以讯飞的鉴权模式:
|
||||
1. RFC1123 GMT 时间串 `date`
|
||||
2. 拼接 `host: iat-api.xfyun.cn\ndate: <date>\nGET /v2/iat HTTP/1.1` → HMAC-SHA256(secret) → base64 → `signature`
|
||||
3. 拼 `api_key="...", algorithm="hmac-sha256", headers="host date request-line", signature="..."` → 再 base64 → `authorization`
|
||||
4. URL = `wss://iat-api.xfyun.cn/v2/iat?authorization=<>&date=<>&host=iat-api.xfyun.cn`
|
||||
|
||||
注意 URL 里的日期**不能跨分钟**:同一连接生命周期里服务端会校验(签名原始串里的 date 与请求 URL 里的 date 一致)。长会话需要周期重连(本工具单次录音 ≤60s,无此问题)。
|
||||
|
||||
## 10. AVAudioConverter 系统原生 PCM → 16k Int16 mono
|
||||
|
||||
`AVAudioEngine.inputNode.outputFormat(forBus: 0)` 出来通常是 48k 或 44.1k Float32。讯飞要 16k/16bit/mono PCM,所以:
|
||||
- `AVAudioConverter(from: inputFmt, to: targetFmt)` 直接搭好
|
||||
- `convert(to:from:error:)` 一行转完(回调式只用于压缩输入的流式场景,我们用不上)
|
||||
- 输出 `AVAudioPCMBuffer.int16ChannelData[0]` → `UnsafeRawBufferPointer(start:count:)` → `Data.append(contentsOf:)`
|
||||
- 注意 `Int16` 指针不能直接喂 `Data.append(_:count:)`,必须先按字节重解释
|
||||
|
||||
## 11. xop 注入 env 经 `open` 传递到 .app
|
||||
|
||||
`xop run --secret X=xf_vt_X -- open /Applications/VoiceType.app` 时,env 经 `xop → open → LaunchServices → app` 这条链路一直传到底,`ProcessInfo.processInfo.environment["XUNFEI_X"]` 在 app 内能拿到。但有几种场景会丢:
|
||||
- macOS 14+ 的 `open --new` 在某些沙盒场景下被 launchd 接管,env 被截断 → 备用 `xop run --secret ... -- /Applications/VoiceType.app/Contents/MacOS/VoiceType`(直接执行二进制,TCC 仍生效,因为同一 .app bundle)
|
||||
- 用户从 Finder 双击 .app:不走 xop,env 没有,引擎立刻报错退出
|
||||
|
||||
## 复现/验证工具
|
||||
|
||||
- 事件 tap 全列表:`CGGetEventTapList`(源码曾存 /tmp/taplist.swift,随 /tmp 已失传,可按 GOTCHAS §2 结论重写)
|
||||
- Fn/左键轮询探针:`CGEventSource.flagsState/.buttonState` + 8ms 循环
|
||||
- 日志判据:每次判定一行 `activate/skip/allow`,带 app bundle id + AX 角色/错误码
|
||||
@@ -0,0 +1,100 @@
|
||||
# VoiceType — 长按左键语音输入(原型文档)
|
||||
|
||||
灵感来自微信 Mac 版的"按住说话":在**任意 app 的输入框**长按鼠标左键说话,松开文字自动出现在光标处。本工具把它做成全局的,且微信窗口内自动放行。
|
||||
|
||||
## 使用
|
||||
|
||||
1. `src/build.sh` 构建安装(首次需手动开「系统设置 → 隐私与安全性 → 辅助功能 → VoiceType」,麦克风/语音识别弹窗点允许)
|
||||
2. 点击输入框定位光标 → **按住左键 ≥0.35s**(哔一声开始)→ 说话 → **松开** → 出字
|
||||
3. 位移超过 10px 视为拖选,不触发;微信窗口内长按交给微信自己的语音输入
|
||||
|
||||
日志:`/tmp/voicetype.log`(每次判定都记录走了哪条路径、看到什么角色、识别文本、粘贴结果)。
|
||||
|
||||
## 引擎选择
|
||||
|
||||
`~/.voicetype-engine` 文件内容决定用哪个 ASR 引擎(空文件 = `apple-llm`):
|
||||
|
||||
| 值 | 引擎 | 备注 |
|
||||
|---|---|---|
|
||||
| `apple-llm` (默认) | Apple 端侧 + LLM 异步精修 | 无凭据;LLM 走 `~/ssw.ini`;无 INI 则回退 Apple 原文 |
|
||||
| `apple` | SFSpeechRecognizer 端侧 zh-CN | 离线,无凭据,响应快,无 LLM 清洗 |
|
||||
| `xunfei` | 讯飞 IAT v2 WebSocket 流式 | **需凭据**;首次须 `xop set` 三个 vault 条目,启动走 `src/build.sh xunfei` |
|
||||
| `dual` | apple + xunfei 并行 + LLM 精修 | 苹果先贴,讯飞覆盖,LLM 再精修 |
|
||||
|
||||
### apple-llm / dual 的 LLM 配置(`~/ssw.ini`)
|
||||
|
||||
LLM 走 OpenAI Chat Completions 兼容协议(任何 zhipu/openai/anthropic-compatible 代理都行)。
|
||||
|
||||
```ini
|
||||
[llm-api]
|
||||
key=sk_xxxxxxxxxxxx
|
||||
base-url=https://your-llm-proxy/v1
|
||||
model=GLM-5.3-Flash
|
||||
```
|
||||
|
||||
容忍全角破折号 `—` 与半角 `-` 等价(`[llm—api]` / `base—url` 都能识别)。
|
||||
|
||||
引擎会自动在请求里带 `thinking: {type: "disabled"`,绕开 GLM 系列推理模式(7s+ → 1s,实测 `thinking=disabled` 后稳定 ~0.9s)。
|
||||
|
||||
**失败兜底**: 无 INI / key 错 / 网络断 / 超时 → 保留 Apple 原文,绝不静默丢内容。
|
||||
|
||||
### 一次性:把讯飞凭据放进 xop vault(仅 xunfei/dual 需要)
|
||||
|
||||
凭据**只**从 TTY 读取,不会出现在源码/git/日志/argv/历史里。`xop set <name>` 然后回车输入值。
|
||||
|
||||
```bash
|
||||
xop set xf_vt_APPID # APPID
|
||||
xop set xf_vt_APISecret # APISecret
|
||||
xop set xf_vt_APIKey # APIKey
|
||||
```
|
||||
|
||||
### 启动对应引擎
|
||||
|
||||
```bash
|
||||
src/build.sh # 默认 = apple-llm
|
||||
src/build.sh apple # 纯端侧(无 LLM 精修)
|
||||
src/build.sh apple-llm # 端侧 + LLM(默认)
|
||||
src/build.sh xunfei # 讯飞(需 xop 注入)
|
||||
src/build.sh dual # 双引擎(需 xop 注入)
|
||||
```
|
||||
|
||||
这一步做三件事:
|
||||
1. 把 `~/.voicetype-engine` 写成对应模式(持久切换)
|
||||
2. `xunfei` / `dual` 模式会 `xop run --secret XUNFEI_APPID=xf_vt_APPID --secret ... -- open /Applications/VoiceType.app` —— 凭据仅在子进程 env 里存在
|
||||
3. 引擎读 `ProcessInfo.processInfo.environment["XUNFEI_*"]`,**任一缺失则报错并丢弃本次录音**(不静默回退 apple,避免你以为用了讯飞但实际是 apple 给你识别差)。
|
||||
|
||||
## 判定流程(按住 350ms 时触发一次)
|
||||
|
||||
```
|
||||
光标下窗口归属(CGWindowList z序)
|
||||
├─ 微信(com.tencent.xinWeChat*) → 放行,静默
|
||||
└─ 其他 app:
|
||||
① AX 聚焦元素角色 ∈ {AXTextField, AXTextArea, AXComboBox, AXSearchField} → 激活
|
||||
② 查不到 → 设置 AXManualAccessibility + AXEnhancedUserInterface,250ms 后重查 → 同①
|
||||
③ 仍查不到(AX 完全不透明,如 HexHub) → app 级兜底放行
|
||||
```
|
||||
|
||||
## 架构(单文件)
|
||||
|
||||
| 组件 | 职责 | 权限 |
|
||||
|---|---|---|
|
||||
| 轮询线程(~125Hz) | `CGEventSource.buttonState` 检测左键,状态机 idle→press→recording/dead | 无 |
|
||||
| 预卷录音 | **鼠标按下瞬间**就开始收音(`AVAudioEngine` tap),激活时才挂识别任务——短句不丢头 | 麦克风 |
|
||||
| `AppleEngine` | `SFSpeechRecognizer(zh-CN)` 流式,`requiresOnDeviceRecognition = true`,松开 ~70ms 出结果 | 语音识别 |
|
||||
| `AppleLLMEngine` | 包 AppleEngine + LLMCleaner;Apple 出文立刻贴,LLM 异步 ~1s 后回来若非空且不同则退格再贴 | 语音识别 + LLM |
|
||||
| `XunfeiEngine` | `URLSessionWebSocketTask` → `wss://iat-api.xfyun.cn/v2/iat`,HMAC-SHA256 鉴权 URL,系统原生 PCM → 16k/16bit/mono 后 1280 字节/帧推送,逐条收 `data_status` 拼结果 | 麦克风(无语音识别权限要求) |
|
||||
| `DualEngine` | apple + xunfei 并行,apple 先贴 → xunfei 覆盖 → LLM 异步精修再覆盖(可关) | 语音识别 + 麦克风 + LLM |
|
||||
| `LLMCleaner` | OpenAI Chat Completions 兼容;强制带 `thinking: disabled`;`refine()` 异步回调,`clean()` 同步 1.5s 兜底 | — |
|
||||
| `pickEngine()` | 读 `~/.voicetype-engine`,默认 `apple-llm` | — |
|
||||
| 注入 | 写剪贴板 → 合成 ⌘V → 1.5s 后恢复原剪贴板 | 辅助功能 |
|
||||
| 拖选抑制 tap | 录音期间吞掉 leftMouseDragged,防拖选文字 | 辅助功能 |
|
||||
| 聚焦查询 | AX 系统级/per-app `kAXFocusedUIElementAttribute` + 角色 | 辅助功能 |
|
||||
|
||||
## 已知边界(探针范围外)
|
||||
|
||||
- AX 完全不透明的 app 里是 **app 级兜底**:长按它任何位置都触发,不只输入框
|
||||
- 剪贴板恢复只管文本(图片/文件内容不还原)
|
||||
- 录音上限 60s;350ms 内慢速起步(<10px)的拖选理论可能误判
|
||||
- LSUIElement 无界面:无菜单栏开关、无开机自启、无常量配置面板(阈值都是源码顶部常量)
|
||||
- 讯飞 IAT v2 单次请求 60s 上限(服务端);长语音会被截断
|
||||
- 讯飞计费按调用次数+音频时长,开发期注意用量
|
||||
@@ -0,0 +1,15 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>CFBundleIdentifier</key><string>com.xs.voicetype</string>
|
||||
<key>CFBundleName</key><string>VoiceType</string>
|
||||
<key>CFBundleExecutable</key><string>VoiceType</string>
|
||||
<key>CFBundlePackageType</key><string>APPL</string>
|
||||
<key>CFBundleShortVersionString</key><string>0.1</string>
|
||||
<key>LSMinimumSystemVersion</key><string>14.0</string>
|
||||
<key>LSUIElement</key><true/>
|
||||
<key>NSMicrophoneUsageDescription</key><string>长按鼠标左键时录音进行语音输入</string>
|
||||
<key>NSSpeechRecognitionUsageDescription</key><string>将录音转成文字</string>
|
||||
</dict>
|
||||
</plist>
|
||||
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/bin/bash
|
||||
# 构建 VoiceType.app 并安装到 /Applications
|
||||
# 红线: 必须用固定 Apple Development 证书签名 —— ad-hoc(-s -) 每次重编译身份都变,
|
||||
# TCC 会当成新 app,麦克风/语音/辅助功能授权全部作废
|
||||
#
|
||||
# 用法:
|
||||
# src/build.sh 编译+签名+安装,默认启动 apple-llm(端侧+LLM 精修)
|
||||
# src/build.sh apple 纯端侧(无 LLM,最低延迟)
|
||||
# src/build.sh apple-llm 端侧识别 + LLM 异步精修(默认;需 ~/ssw.ini [llm-api])
|
||||
# src/build.sh xunfei 讯飞识别(端点 ~1s)
|
||||
# src/build.sh dual apple + xunfei 并行 + LLM 精修(xunfei 覆盖后)
|
||||
#
|
||||
# 引擎持久切换(两种启动方式之外):
|
||||
# echo apple > ~/.voicetype-engine
|
||||
# echo apple-llm > ~/.voicetype-engine
|
||||
# echo xunfei > ~/.voicetype-engine
|
||||
# echo dual > ~/.voicetype-engine
|
||||
#
|
||||
# 凭据(讯飞,一次性,只需跑一次,xop 从 TTY 读值):
|
||||
# xop set xf_vt_APPID # APPID
|
||||
# xop set xf_vt_APISecret # APISecret
|
||||
# xop set xf_vt_APIKey # APIKey
|
||||
#
|
||||
# LLM(apple-llm/dual 需要,否则静默回退到 apple 原文):
|
||||
# ~/ssw.ini → [llm-api] key=… base-url=… model=…
|
||||
# (apple-llm 无 ini 也照常工作,只是不精修)
|
||||
set -e
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
IDENT="Apple Development: [email protected] (Y4AJXC7Q9S)"
|
||||
APP=/Applications/VoiceType.app
|
||||
BIN=/tmp/voicetype-build
|
||||
|
||||
MODE="${1:-apple-llm}"
|
||||
|
||||
swiftc -O voicetype.swift -o "$BIN"
|
||||
|
||||
mkdir -p "$APP/Contents/MacOS"
|
||||
cp "$BIN" "$APP/Contents/MacOS/VoiceType"
|
||||
cp Info.plist "$APP/Contents/Info.plist"
|
||||
codesign --force --sign "$IDENT" "$APP"
|
||||
|
||||
killall VoiceType 2>/dev/null || true
|
||||
sleep 0.3
|
||||
: > /tmp/voicetype.log
|
||||
|
||||
case "$MODE" in
|
||||
xunfei)
|
||||
# 把 ~/.voicetype-engine 也设上,使得日后 `open /Applications/VoiceType.app`
|
||||
# 直接双击也能继续用 xunfei。凭据永远只走 xop 注入,不落盘不传 argv。
|
||||
current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )"
|
||||
[ "$current" != "xunfei" ] && echo xunfei > ~/.voicetype-engine
|
||||
xop run \
|
||||
--secret XUNFEI_APPID=xf_vt_APPID \
|
||||
--secret XUNFEI_API_SECRET=xf_vt_APISecret \
|
||||
--secret XUNFEI_API_KEY=xf_vt_APIKey \
|
||||
-- open "$APP"
|
||||
;;
|
||||
dual)
|
||||
current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )"
|
||||
[ "$current" != "dual" ] && echo dual > ~/.voicetype-engine
|
||||
xop run \
|
||||
--secret XUNFEI_APPID=xf_vt_APPID \
|
||||
--secret XUNFEI_API_SECRET=xf_vt_APISecret \
|
||||
--secret XUNFEI_API_KEY=xf_vt_APIKey \
|
||||
-- open "$APP"
|
||||
;;
|
||||
apple)
|
||||
current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )"
|
||||
[ -n "$current" ] && [ "$current" != "apple" ] && echo apple > ~/.voicetype-engine
|
||||
open "$APP"
|
||||
;;
|
||||
apple-llm|*)
|
||||
current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )"
|
||||
[ -n "$current" ] && [ "$current" != "apple-llm" ] && echo apple-llm > ~/.voicetype-engine
|
||||
[ -z "$current" ] && echo apple-llm > ~/.voicetype-engine
|
||||
open "$APP"
|
||||
;;
|
||||
esac
|
||||
|
||||
sleep 2
|
||||
cat /tmp/voicetype.log
|
||||
echo
|
||||
echo "installed: $APP (pid $(pgrep -f VoiceType)) engine=$MODE"
|
||||
[ "$MODE" = "xunfei" ] || [ "$MODE" = "dual" ] && echo "vault used: xf_vt_APPID / xf_vt_APISecret / xf_vt_APIKey (no values in logs/argv)"
|
||||
[ "$MODE" = "apple-llm" ] && grep -q '\[llm-api\]' ~/ssw.ini 2>/dev/null && echo "llm: ~/ssw.ini [llm-api] 已配 → LLM 异步精修生效;否则回退 apple 原文"
|
||||
@@ -0,0 +1,910 @@
|
||||
// voicetype: 长按鼠标左键 → 说话 → 松开自动粘贴到光标处
|
||||
// 引擎可选: apple / xunfei / dual(apple+xunfei 并行,apple 先贴)
|
||||
// 凭据(仅 xunfei 需要): 经 xop run --secret 注入到环境变量,源码/git/日志绝不出现
|
||||
// LLM 清洗(可选): ~/ssw.ini 的 [llm-api] 段,key/base-url/model;无配置则跳过
|
||||
// 权限: 麦克风 + 语音识别(apple) / 麦克风 + 辅助功能(全模式)
|
||||
import AppKit
|
||||
import ApplicationServices
|
||||
import AVFoundation
|
||||
import CoreGraphics
|
||||
import CryptoKit
|
||||
import Speech
|
||||
|
||||
// ---------- 配置 ----------
|
||||
let holdMs: Double = 350 // 长按激活阈值
|
||||
let movePx: CGFloat = 10 // 位移超过此值视为拖选,不激活
|
||||
let maxRecSec: Double = 60 // 最长录音(到点直接贴当前已识别部分,不继续等)
|
||||
let pollUs: UInt32 = 8_000 // 轮询周期 8ms (~125Hz)
|
||||
let weChatPrefix = "com.tencent.xinWeChat"
|
||||
let engineConfigPath = NSHomeDirectory() + "/.voicetype-engine"
|
||||
let llmIniPath = NSHomeDirectory() + "/ssw.ini"
|
||||
|
||||
// ---------- INI 读取(仅支持 section/key=value 简单格式) ----------
|
||||
// 容忍 section/key 名里的全角破折号 `—` 与半角 `-` 等价(~/ssw.ini 历史上用 —)
|
||||
func normINIKey(_ s: String) -> String {
|
||||
s.replacingOccurrences(of: "—", with: "-").trimmingCharacters(in: .whitespaces)
|
||||
}
|
||||
|
||||
func loadINI(_ path: String) -> [String: [String: String]] {
|
||||
guard let raw = try? String(contentsOfFile: path, encoding: .utf8) else { return [:] }
|
||||
var sections: [String: [String: String]] = [:]
|
||||
var current: String?
|
||||
for line in raw.split(whereSeparator: { $0 == "\n" || $0 == "\r" }) {
|
||||
let s = line.trimmingCharacters(in: .whitespaces)
|
||||
if s.isEmpty || s.hasPrefix("#") || s.hasPrefix(";") { continue }
|
||||
if s.hasPrefix("[") && s.hasSuffix("]") {
|
||||
current = normINIKey(String(s.dropFirst().dropLast()))
|
||||
sections[current!] = [:]
|
||||
} else if let cur = current, let eq = s.firstIndex(of: "=") {
|
||||
let k = normINIKey(String(s[..<eq]))
|
||||
let v = String(s[s.index(after: eq)...]).trimmingCharacters(in: .whitespaces)
|
||||
sections[cur]?[k] = v
|
||||
}
|
||||
}
|
||||
return sections
|
||||
}
|
||||
|
||||
struct LLMConfig {
|
||||
let key: String
|
||||
let baseURL: String // 例如 https://api.openai.com/v1
|
||||
let model: String // 例如 gpt-4o-mini
|
||||
static func load() -> LLMConfig? {
|
||||
let s = loadINI(llmIniPath)["llm-api"] ?? [:]
|
||||
guard let k = s["key"], !k.isEmpty,
|
||||
let u = s["base-url"], !u.isEmpty,
|
||||
let m = s["model"], !m.isEmpty else {
|
||||
log("llm: ~/ssw.ini [llm-api] 缺失或字段不全,跳过 LLM 清洗")
|
||||
return nil
|
||||
}
|
||||
log("llm: 加载 [llm-api] model=\(m) base=\(u.prefix(30))…")
|
||||
return LLMConfig(key: k, baseURL: u, model: m)
|
||||
}
|
||||
}
|
||||
|
||||
// 讯飞 IAT v2(参考 xfyun 开放平台"语音听写(流式版)WebAPI")
|
||||
let xunfeiURL = "wss://iat-api.xfyun.cn/v2/iat"
|
||||
let xunfeiHost = "iat-api.xfyun.cn"
|
||||
|
||||
setbuf(stdout, nil)
|
||||
func ts() -> String { let f = DateFormatter(); f.dateFormat = "HH:mm:ss.SSS"; return f.string(from: Date()) }
|
||||
func log(_ s: String) { print("[\(ts())] \(s)") }
|
||||
|
||||
// ---------- 共享状态(仅 poller 线程写) ----------
|
||||
final class Box<T> { var v: T; init(_ v: T) { self.v = v } }
|
||||
enum St { case idle; case press(origin: CGPoint, start: Date); case recording(origin: CGPoint, start: Date); case dead }
|
||||
let state = Box<St>(.idle)
|
||||
let maxMove = Box<CGFloat>(0)
|
||||
let suppressDrag = Box<Bool>(false)
|
||||
|
||||
// ---------- ASR 引擎抽象 ----------
|
||||
protocol ASREngine {
|
||||
func prepare() // 鼠标按下瞬间: 预卷/连WS
|
||||
func beginTask() // 长按激活: 开始识别
|
||||
func cancel() // 普通点击抬起: 丢弃
|
||||
func stop() -> String // 录音中抬起: 阻塞等最终结果(dual 模式下返回 apple 的)
|
||||
func flushNow() -> String // force-stop / maxRecSec: 不等,直接拿当前累计文本
|
||||
var readyForBegin: Bool { get } // prepare() 是否成功(凭据检查等)
|
||||
}
|
||||
|
||||
func pickEngine() -> ASREngine {
|
||||
let raw = (try? String(contentsOfFile: engineConfigPath, encoding: .utf8))?
|
||||
.trimmingCharacters(in: .whitespacesAndNewlines) ?? "apple-llm"
|
||||
switch raw {
|
||||
case "xunfei":
|
||||
log("engine: xunfei (require XUNFEI_APPID/API_SECRET/API_KEY in env)")
|
||||
return XunfeiEngine()
|
||||
case "dual":
|
||||
log("engine: dual (apple first, xunfei overwrite later, llm refine)")
|
||||
return DualEngine()
|
||||
case "apple-llm", "llm":
|
||||
log("engine: apple + LLM refine (~/ssw.ini [llm-api] required, no INI → silent fallback to apple)")
|
||||
return AppleLLMEngine()
|
||||
default:
|
||||
log("engine: apple (on-device, no LLM)")
|
||||
return AppleEngine()
|
||||
}
|
||||
}
|
||||
|
||||
final class Engine {
|
||||
static var current: ASREngine = pickEngine()
|
||||
static var ready: Bool { current.readyForBegin }
|
||||
}
|
||||
|
||||
// ---------- Apple 端侧 SFSpeechRecognizer ----------
|
||||
final class AppleEngine: ASREngine {
|
||||
static let q = DispatchQueue(label: "voice.apple.rec")
|
||||
let engine = AVAudioEngine()
|
||||
var request: SFSpeechAudioBufferRecognitionRequest?
|
||||
var task: SFSpeechRecognitionTask?
|
||||
var recognizer = SFSpeechRecognizer(locale: Locale(identifier: "zh-CN"))
|
||||
var text = ""
|
||||
var finalSem = DispatchSemaphore(value: 0)
|
||||
var prepared = false
|
||||
|
||||
var readyForBegin: Bool { prepared }
|
||||
|
||||
func prepare() {
|
||||
text = ""
|
||||
finalSem = DispatchSemaphore(value: 0)
|
||||
let req = SFSpeechAudioBufferRecognitionRequest()
|
||||
req.shouldReportPartialResults = true
|
||||
req.addsPunctuation = true
|
||||
if recognizer?.supportsOnDeviceRecognition == true {
|
||||
req.requiresOnDeviceRecognition = true
|
||||
} else {
|
||||
log("apple: on-device unavailable → server fallback")
|
||||
}
|
||||
let input = engine.inputNode
|
||||
let fmt = input.outputFormat(forBus: 0)
|
||||
input.removeTap(onBus: 0)
|
||||
input.installTap(onBus: 0, bufferSize: 2048, format: fmt) { [req] buf, _ in
|
||||
req.append(buf)
|
||||
}
|
||||
request = req
|
||||
do { try engine.start() } catch { log("apple engine start failed: \(error)") }
|
||||
prepared = true
|
||||
}
|
||||
|
||||
func beginTask() {
|
||||
guard let req = request, let rec = recognizer, prepared else { return }
|
||||
task = rec.recognitionTask(with: req) { [weak self] r, err in
|
||||
guard let self = self else { return }
|
||||
if let r = r {
|
||||
let t = r.bestTranscription.formattedString
|
||||
if !t.isEmpty { self.text = t }
|
||||
if r.isFinal { self.finalSem.signal() }
|
||||
}
|
||||
if err != nil { log("ASR err: \(err!)"); self.finalSem.signal() }
|
||||
}
|
||||
log("apple: recording (on-device: \(self.recognizer?.supportsOnDeviceRecognition == true))")
|
||||
}
|
||||
|
||||
func cancel() {
|
||||
guard prepared else { return }
|
||||
engine.inputNode.removeTap(onBus: 0)
|
||||
engine.stop()
|
||||
request = nil
|
||||
task = nil
|
||||
prepared = false
|
||||
}
|
||||
|
||||
func flushNow() -> String {
|
||||
// SFSpeech 不太好中途取结果;返回当前 text(可能部分)
|
||||
let t = text
|
||||
cancel()
|
||||
return t
|
||||
}
|
||||
|
||||
func stop() -> String {
|
||||
engine.inputNode.removeTap(onBus: 0)
|
||||
engine.stop()
|
||||
request?.endAudio()
|
||||
_ = finalSem.wait(timeout: .now() + 3)
|
||||
task?.finish()
|
||||
let t = text
|
||||
text = ""
|
||||
request = nil
|
||||
task = nil
|
||||
prepared = false
|
||||
return t
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- Apple 端侧 + LLM 异步精修 ----------
|
||||
// 流程: Apple 识别 → 贴出原文 → LLM 异步精修 → 若非空且不同则删原文再贴精修版
|
||||
// 失败/超时/未配 INI: 不精修,保留 Apple 原文(不翻车)
|
||||
final class AppleLLMEngine: ASREngine {
|
||||
let apple = AppleEngine()
|
||||
let cleaner: LLMCleaner
|
||||
var lastRaw = "" // 覆盖阶段用: 知道要删多少字
|
||||
|
||||
init() {
|
||||
self.cleaner = LLMCleaner(cfg: LLMConfig.load())
|
||||
}
|
||||
|
||||
var readyForBegin: Bool { apple.readyForBegin }
|
||||
|
||||
func prepare() { apple.prepare() }
|
||||
func beginTask() { apple.beginTask() }
|
||||
func cancel() { apple.cancel() }
|
||||
func flushNow() -> String { return apple.flushNow() }
|
||||
|
||||
func stop() -> String {
|
||||
let raw = apple.stop()
|
||||
lastRaw = raw
|
||||
// 异步 LLM 精修: 非空且不同 → 删原文再贴
|
||||
DispatchQueue.global().async { [weak self] in
|
||||
guard let self = self else { return }
|
||||
self.cleaner.refine(raw) { [weak self] refined in
|
||||
guard let self = self else { return }
|
||||
guard !refined.isEmpty, refined != raw else {
|
||||
log("apple-llm: refine skipped (empty or identical)")
|
||||
return
|
||||
}
|
||||
log("apple-llm: llm refine overwrite (\(raw.count)→\(refined.count) chars)")
|
||||
self.deleteChars(raw.count)
|
||||
paste(refined)
|
||||
}
|
||||
}
|
||||
return raw
|
||||
}
|
||||
|
||||
private func deleteChars(_ n: Int) {
|
||||
guard n > 0 else { return }
|
||||
let src = CGEventSource(stateID: .combinedSessionState)
|
||||
for _ in 0..<n {
|
||||
let kd = CGEvent(keyboardEventSource: src, virtualKey: 51, keyDown: true)
|
||||
kd?.post(tap: .cghidEventTap)
|
||||
let ku = CGEvent(keyboardEventSource: src, virtualKey: 51, keyDown: false)
|
||||
ku?.post(tap: .cghidEventTap)
|
||||
usleep(3_000)
|
||||
}
|
||||
}
|
||||
}
|
||||
// 凭据来自环境变量(XUNFEI_APPID / XUNFEI_API_SECRET / XUNFEI_API_KEY),
|
||||
// 由 xop run --secret 在启动前注入。引擎自身不读配置文件、不打 fallback —— 安静回退会让用户误以为"用了讯飞识别差"。
|
||||
final class XunfeiEngine: NSObject, ASREngine {
|
||||
let q = DispatchQueue(label: "voice.xunfei")
|
||||
let engine = AVAudioEngine()
|
||||
var session: URLSession?
|
||||
var wsTask: URLSessionWebSocketTask?
|
||||
var pcmBuf = Data() // 16k Int16 mono PCM 累积
|
||||
var rawBuf = Data() // 硬件格式 PCM 累积(Float32 mono @ hwRate,待手动转换)
|
||||
var hwFmt: AVAudioFormat? // 输入节点硬件格式(用于算下采样比)
|
||||
var resultText = "" // 累积识别结果
|
||||
var waitingFinal = false // status=2 已发,等任何一条后续回复即视为结束
|
||||
var firstFrameSent = false
|
||||
var frameTimer: DispatchSourceTimer?
|
||||
var finalSem = DispatchSemaphore(value: 0)
|
||||
var openedSem = DispatchSemaphore(value: 0)
|
||||
var opened = false
|
||||
var prepared = false // prepare() 成功(ws open + lastErr==nil)就 true
|
||||
var lastErr: String?
|
||||
var appID = ""
|
||||
|
||||
var readyForBegin: Bool { prepared }
|
||||
|
||||
private func envCreds() -> (id: String, secret: String, key: String)? {
|
||||
let env = ProcessInfo.processInfo.environment
|
||||
guard let id = env["XUNFEI_APPID"], !id.isEmpty,
|
||||
let s = env["XUNFEI_API_SECRET"], !s.isEmpty,
|
||||
let k = env["XUNFEI_API_KEY"], !k.isEmpty else { return nil }
|
||||
return (id, s, k)
|
||||
}
|
||||
|
||||
private func rfc1123() -> String {
|
||||
let f = DateFormatter()
|
||||
f.locale = Locale(identifier: "en_US_POSIX")
|
||||
f.timeZone = TimeZone(identifier: "GMT")
|
||||
f.dateFormat = "EEE, dd MMM yyyy HH:mm:ss z"
|
||||
return f.string(from: Date())
|
||||
}
|
||||
|
||||
// HMAC-SHA256 鉴权 URL(同步在 IAT v2 文档)
|
||||
private func authURL(appID: String, secret: String, apiKey: String) -> URL? {
|
||||
let date = rfc1123()
|
||||
let signOrigin = "host: \(xunfeiHost)\ndate: \(date)\nGET /v2/iat HTTP/1.1"
|
||||
let mac = HMAC<SHA256>.authenticationCode(for: Data(signOrigin.utf8), using: SymmetricKey(data: Data(secret.utf8)))
|
||||
let sig = Data(mac).base64EncodedString()
|
||||
let authOrigin = "api_key=\"\(apiKey)\", algorithm=\"hmac-sha256\", headers=\"host date request-line\", signature=\"\(sig)\""
|
||||
let auth = Data(authOrigin.utf8).base64EncodedString()
|
||||
return URL(string: "\(xunfeiURL)?authorization=\(auth)&date=\(date)&host=\(xunfeiHost)")
|
||||
}
|
||||
|
||||
func prepare() {
|
||||
// 先收尾旧的(session、wsTask),避免和上一轮并发触发 delegate 回调
|
||||
cleanup()
|
||||
|
||||
guard let creds = envCreds() else {
|
||||
lastErr = "missing env XUNFEI_APPID/API_SECRET/API_KEY — launch via `xop run --secret`"
|
||||
log("XUNFEI: \(lastErr!)")
|
||||
return
|
||||
}
|
||||
appID = creds.id
|
||||
|
||||
// tap 必须用输入节点硬件格式 (Float32, 通常 44.1k 或 48k) —— 不然 installTap 抛 NSException
|
||||
// 转码(浮点→int16 + 下采样)放到 serial queue 上做,绝不在音频线程跑。
|
||||
let input = engine.inputNode
|
||||
hwFmt = input.outputFormat(forBus: 0)
|
||||
|
||||
guard let url = authURL(appID: creds.id, secret: creds.secret, apiKey: creds.key) else {
|
||||
lastErr = "auth URL build failed"
|
||||
log("XUNFEI: \(lastErr!)")
|
||||
return
|
||||
}
|
||||
|
||||
// 用串行 queue 保证 delegate 回调按序触发
|
||||
let delegateQ = OperationQueue()
|
||||
delegateQ.maxConcurrentOperationCount = 1
|
||||
session = URLSession(configuration: .default, delegate: self, delegateQueue: delegateQ)
|
||||
wsTask = session!.webSocketTask(with: url)
|
||||
opened = false
|
||||
wsTask?.resume()
|
||||
|
||||
// tap 回调是音频实时线程,绝对禁止在这里 log / 阻塞 / 分配大对象。
|
||||
// 只做一次 unsafe 指针复制 + 把字节扔到 serial queue。
|
||||
engine.inputNode.removeTap(onBus: 0)
|
||||
engine.inputNode.installTap(onBus: 0, bufferSize: 2048, format: hwFmt!) { [weak self] buf, _ in
|
||||
guard let self = self,
|
||||
let ch = buf.floatChannelData?[0] else { return }
|
||||
let n = Int(buf.frameLength)
|
||||
if n == 0 { return }
|
||||
let raw = UnsafeRawBufferPointer(start: ch, count: n * 4) // Float32 = 4 bytes
|
||||
let copy = Data(raw) // 拷贝出音频线程作用域
|
||||
self.q.async { self.rawBuf.append(copy); self.drainAndConvert() }
|
||||
}
|
||||
do { try engine.start() } catch {
|
||||
lastErr = "engine start failed: \(error)"
|
||||
log("XUNFEI: \(lastErr!)")
|
||||
return
|
||||
}
|
||||
prepared = true
|
||||
log("xunfei: ws connecting + recording (hw=\(hwFmt!.sampleRate)Hz fmt=\(hwFmt!.commonFormat.rawValue))")
|
||||
}
|
||||
|
||||
private func cleanup() {
|
||||
frameTimer?.cancel(); frameTimer = nil
|
||||
wsTask?.cancel(with: .goingAway, reason: nil)
|
||||
wsTask = nil
|
||||
session?.invalidateAndCancel()
|
||||
session = nil
|
||||
engine.inputNode.removeTap(onBus: 0)
|
||||
if engine.isRunning { engine.stop() }
|
||||
pcmBuf = Data(); rawBuf = Data(); resultText = ""
|
||||
firstFrameSent = false
|
||||
hwFmt = nil
|
||||
waitingFinal = false
|
||||
finalSem = DispatchSemaphore(value: 0)
|
||||
openedSem = DispatchSemaphore(value: 0)
|
||||
opened = false
|
||||
}
|
||||
|
||||
private func sendFrames() {
|
||||
q.async {
|
||||
// 先把硬件格式的 rawBuf 转码成 16k int16 mono 进 pcmBuf
|
||||
self.drainAndConvert()
|
||||
// 1280 bytes = 640 samples × 2 bytes/sample = 40ms @ 16kHz
|
||||
while self.pcmBuf.count >= 1280 {
|
||||
guard let ws = self.wsTask, ws.state == .running else { return } // ws 还没握上,等下一帧再试
|
||||
let chunk = self.pcmBuf.prefix(1280)
|
||||
self.pcmBuf.removeFirst(1280)
|
||||
let isFirst = !self.firstFrameSent
|
||||
self.send(status: self.firstFrameSent ? 1 : 0, audio: Data(chunk))
|
||||
self.firstFrameSent = true
|
||||
if isFirst { log("xunfei: sent first chunk (\(chunk.count)B) ws.state=\(ws.state.rawValue)") }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 把 rawBuf(硬件格式 Float32 mono @ hwRate)手动转成 16k int16 mono,入 pcmBuf。
|
||||
// 只在 serial queue 上跑,绝不在音频线程跑。
|
||||
// 解: Float32 -> Int16 (clip); 48k -> 16k 用 3:1 抽取(够 ASR 用,免 AVAudioConverter 崩)
|
||||
private func drainAndConvert() {
|
||||
guard hwFmt != nil else { return }
|
||||
let hwFrameBytes = 4 // Float32 mono = 4 bytes/frame
|
||||
let frameCount = rawBuf.count / hwFrameBytes
|
||||
guard frameCount >= 3 else { return } // 至少一帧 16k 输出
|
||||
|
||||
let ratio = Int((hwFmt!.sampleRate / 16000).rounded())
|
||||
let outFrames = frameCount / max(ratio, 1)
|
||||
guard outFrames > 0 else { return }
|
||||
|
||||
rawBuf.withUnsafeBytes { rawPtr in
|
||||
let floats = rawPtr.bindMemory(to: Float32.self).baseAddress!
|
||||
var out = [Int16](repeating: 0, count: outFrames)
|
||||
for i in 0..<outFrames {
|
||||
let s = floats[i * ratio]
|
||||
let clipped = max(-1.0, min(1.0, s))
|
||||
out[i] = Int16(clipped * 32767)
|
||||
}
|
||||
out.withUnsafeBufferPointer { bp in
|
||||
pcmBuf.append(Data(buffer: bp))
|
||||
}
|
||||
}
|
||||
rawBuf.removeFirst(outFrames * ratio * hwFrameBytes)
|
||||
}
|
||||
|
||||
private func send(status: Int, audio: Data) {
|
||||
guard let ws = wsTask else { return }
|
||||
let payload: [String: Any] = [
|
||||
"common": ["app_id": appID],
|
||||
"business": ["language": "zh_cn", "domain": "iat", "accent": "mandarin", "vad_eos": 10000, "ptt": 1],
|
||||
"data": ["status": status, "format": "audio/L16;rate=16000", "encoding": "raw", "audio": audio.base64EncodedString()]
|
||||
]
|
||||
guard let data = try? JSONSerialization.data(withJSONObject: payload) else { return }
|
||||
log("xunfei: send status=\(status) bytes=\(audio.count) ws=\(ws.state.rawValue)")
|
||||
ws.send(.data(data)) { err in
|
||||
if let err = err { log("xunfei send err: \(err)") }
|
||||
}
|
||||
}
|
||||
|
||||
private func scheduleRecv() {
|
||||
wsTask?.receive { [weak self] result in
|
||||
guard let self = self else { return }
|
||||
switch result {
|
||||
case .success(let msg):
|
||||
self.handle(msg)
|
||||
if self.lastErr == nil { self.scheduleRecv() }
|
||||
case .failure(let err):
|
||||
log("xunfei recv err: \(err)")
|
||||
self.lastErr = err.localizedDescription
|
||||
self.finalSem.signal()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func handle(_ msg: URLSessionWebSocketTask.Message) {
|
||||
let data: Data?
|
||||
switch msg {
|
||||
case .string(let s): data = s.data(using: .utf8)
|
||||
case .data(let d): data = d
|
||||
@unknown default: data = nil
|
||||
}
|
||||
guard let d = data, let j = try? JSONSerialization.jsonObject(with: d) as? [String: Any] else { return }
|
||||
let code = j["code"] as? Int ?? -1
|
||||
let payload = j["data"] as? [String: Any]
|
||||
log("xunfei recv: code=\(code) payload=\(payload ?? [:])")
|
||||
if code != 0 {
|
||||
lastErr = "信令码 \(code) \(j["message"] ?? "")"
|
||||
log("XUNFEI: \(lastErr!)")
|
||||
finalSem.signal()
|
||||
return
|
||||
}
|
||||
guard let payload = j["data"] as? [String: Any] else { return }
|
||||
if let result = payload["result"] as? [String: Any],
|
||||
let wsArr = result["ws"] as? [[String: Any]] {
|
||||
var piece = ""
|
||||
for w in wsArr {
|
||||
if let cw = w["cw"] as? [[String: Any]] {
|
||||
for c in cw {
|
||||
if let ww = c["w"] as? String { piece += ww }
|
||||
}
|
||||
}
|
||||
}
|
||||
// pgs: "rpl" 替换全文;"apd" 追加。"apd" 时将 piece 接到尾部
|
||||
let pgs = result["pgs"] as? String ?? "rpl"
|
||||
if pgs == "apd" { resultText += piece } else { resultText = piece }
|
||||
log("xunfei recv rg: pgs=\(pgs) piece=\"\(piece)\" total=\"\(resultText)\" ls=\(payload["ls"] ?? false)")
|
||||
}
|
||||
if (payload["ls"] as? Bool) == true || waitingFinal {
|
||||
// status=2 之后任何一条回复 = 服务端把最终答案给我了;ls=true 也算
|
||||
finalSem.signal()
|
||||
}
|
||||
}
|
||||
|
||||
func beginTask() {
|
||||
guard prepared, lastErr == nil else {
|
||||
log("XUNFEI: beginTask refused (not prepared: \(lastErr ?? "?"))")
|
||||
return
|
||||
}
|
||||
// 每 40ms 切一帧推上去
|
||||
let t = DispatchSource.makeTimerSource(queue: q)
|
||||
t.schedule(deadline: .now() + 0.04, repeating: 0.04)
|
||||
t.setEventHandler { [weak self] in self?.sendFrames() }
|
||||
t.resume()
|
||||
frameTimer = t
|
||||
scheduleRecv()
|
||||
log("xunfei: begin")
|
||||
}
|
||||
|
||||
func cancel() {
|
||||
cleanup()
|
||||
lastErr = nil
|
||||
prepared = false
|
||||
}
|
||||
|
||||
func flushNow() -> String {
|
||||
// 不发 status=2、不等;直接返回当前 resultText(中间识别累计)
|
||||
let t = resultText
|
||||
cleanup()
|
||||
lastErr = nil
|
||||
prepared = false
|
||||
return t
|
||||
}
|
||||
|
||||
func stop() -> String {
|
||||
frameTimer?.cancel(); frameTimer = nil
|
||||
if prepared, wsTask != nil {
|
||||
// 最后一帧: 先把 rawBuf 转完,再把 pcmBuf 尾巴推上去 + status=2
|
||||
waitingFinal = true // 发完 status=2 后,handle() 收到任何一条回复就 signal
|
||||
q.sync {
|
||||
self.drainAndConvert()
|
||||
let tail = self.pcmBuf
|
||||
self.pcmBuf = Data()
|
||||
self.send(status: 2, audio: tail)
|
||||
}
|
||||
// 等服务端回完最终那条(经验 100~500ms),1s 兜底(waitingFinal + ws close 都会 signal)
|
||||
_ = finalSem.wait(timeout: .now() + 1)
|
||||
}
|
||||
// 必须在 cleanup 前读 resultText —— cleanup 会把它清空
|
||||
let t = resultText
|
||||
cleanup()
|
||||
lastErr = nil
|
||||
prepared = false
|
||||
return t
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- LLM 清洗(可选):OpenAI Chat Completions 兼容 ----------
|
||||
final class LLMCleaner {
|
||||
let cfg: LLMConfig?
|
||||
init(cfg: LLMConfig?) { self.cfg = cfg }
|
||||
|
||||
static let prompt = """
|
||||
你是语音转写后处理助手。把下面这段口述语音转写文字清理成自然书面语中文:
|
||||
- 去掉"嗯/啊/呃/那个/就是/然后/其实/对"等口水词
|
||||
- 加标点(句号、逗号、问号等),分句
|
||||
- 重组零碎短句为通顺长句
|
||||
- 保留全部原意,不增删内容
|
||||
只输出清理后的文字,不要任何解释、标签或前后缀。
|
||||
"""
|
||||
|
||||
func clean(_ raw: String) -> String {
|
||||
// 同步接口(给单引擎调用):1.5s 兜底超时,默认返 raw
|
||||
guard let cfg = cfg, !raw.isEmpty else { return raw }
|
||||
let url = URL(string: cfg.baseURL.trimmingCharacters(in: CharacterSet(charactersIn: "/")) + "/chat/completions")!
|
||||
var req = URLRequest(url: url, timeoutInterval: 1.5)
|
||||
req.httpMethod = "POST"
|
||||
req.setValue("application/json", forHTTPHeaderField: "Content-Type")
|
||||
req.setValue("Bearer \(cfg.key)", forHTTPHeaderField: "Authorization")
|
||||
var body: [String: Any] = [
|
||||
"model": cfg.model,
|
||||
"messages": [
|
||||
["role": "system", "content": Self.prompt],
|
||||
["role": "user", "content": raw],
|
||||
],
|
||||
"temperature": 0.1,
|
||||
]
|
||||
// 强制关推理: GLM-5.3-Flash 默认有 reasoning_content,7s+;关掉 <1s 且内容字段干净。
|
||||
// 不支持 thinking 的代理会静默忽略。
|
||||
body["thinking"] = ["type": "disabled"]
|
||||
req.httpBody = try? JSONSerialization.data(withJSONObject: body)
|
||||
let sem = DispatchSemaphore(value: 0)
|
||||
var result = raw
|
||||
URLSession.shared.dataTask(with: req) { data, _, _ in
|
||||
defer { sem.signal() }
|
||||
guard let data = data,
|
||||
let j = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let choices = j["choices"] as? [[String: Any]],
|
||||
let msg = choices.first?["message"] as? [String: Any],
|
||||
let content = msg["content"] as? String else { return }
|
||||
let cleaned = content.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
if !cleaned.isEmpty { result = cleaned }
|
||||
}.resume()
|
||||
_ = sem.wait(timeout: .now() + 1.5)
|
||||
return result
|
||||
}
|
||||
|
||||
// 异步精修(给 apple-llm / dual 引擎用):长 timeout,回调返 cleaned
|
||||
func refine(_ raw: String, timeout: TimeInterval = 3, _ cb: @escaping (String) -> Void) {
|
||||
guard let cfg = cfg, !raw.isEmpty else { cb(raw); return }
|
||||
let url = URL(string: cfg.baseURL.trimmingCharacters(in: CharacterSet(charactersIn: "/")) + "/chat/completions")!
|
||||
var req = URLRequest(url: url, timeoutInterval: timeout)
|
||||
req.httpMethod = "POST"
|
||||
req.setValue("application/json", forHTTPHeaderField: "Content-Type")
|
||||
req.setValue("Bearer \(cfg.key)", forHTTPHeaderField: "Authorization")
|
||||
var body: [String: Any] = [
|
||||
"model": cfg.model,
|
||||
"messages": [
|
||||
["role": "system", "content": Self.prompt],
|
||||
["role": "user", "content": raw],
|
||||
],
|
||||
"temperature": 0.1,
|
||||
]
|
||||
body["thinking"] = ["type": "disabled"]
|
||||
req.httpBody = try? JSONSerialization.data(withJSONObject: body)
|
||||
let done = DispatchSemaphore(value: 0)
|
||||
URLSession.shared.dataTask(with: req) { data, _, _ in
|
||||
defer { done.signal() }
|
||||
guard let data = data,
|
||||
let j = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let choices = j["choices"] as? [[String: Any]],
|
||||
let msg = choices.first?["message"] as? [String: Any],
|
||||
let content = msg["content"] as? String else {
|
||||
cb(raw); return
|
||||
}
|
||||
let cleaned = content.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
cb(cleaned.isEmpty ? raw : cleaned)
|
||||
}.resume()
|
||||
_ = done.wait(timeout: .now() + timeout + 0.5)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 本地清洗(无) ----------
|
||||
// LLM 之前实验过本地正则去"嗯/啊/那个"等口水词,但中文没空格,
|
||||
// 任何字符级启发式都会误伤"好啊""什么啊"等正常语料 → 已弃用,清洗只走 LLM
|
||||
|
||||
// ---------- Dual 引擎:apple + xunfei 并行,apple 先贴、xunfei 后续覆盖 ----------
|
||||
final class DualEngine: ASREngine {
|
||||
let apple = AppleEngine()
|
||||
let xunfei = XunfeiEngine()
|
||||
let cleaner: LLMCleaner
|
||||
var lastAppleText = "" // 给覆盖阶段用:知道要删多少字
|
||||
var overwriteScheduled = false
|
||||
|
||||
init() {
|
||||
self.cleaner = LLMCleaner(cfg: LLMConfig.load())
|
||||
}
|
||||
|
||||
var readyForBegin: Bool { apple.readyForBegin && xunfei.readyForBegin }
|
||||
|
||||
func prepare() {
|
||||
apple.prepare()
|
||||
xunfei.prepare()
|
||||
}
|
||||
|
||||
func beginTask() {
|
||||
apple.beginTask()
|
||||
xunfei.beginTask()
|
||||
}
|
||||
|
||||
func cancel() {
|
||||
apple.cancel()
|
||||
xunfei.cancel()
|
||||
}
|
||||
|
||||
func flushNow() -> String {
|
||||
return apple.flushNow()
|
||||
}
|
||||
|
||||
func stop() -> String {
|
||||
// 1) 收 apple(~1s) → 立刻贴出原始文本(不本地清洗)
|
||||
let appleRaw = apple.stop()
|
||||
lastAppleText = appleRaw
|
||||
// 2) 异步: xunfei 识别完 → 覆盖 apple;再走 LLM 异步精修 → 若更优再覆盖一次
|
||||
DispatchQueue.global().async { [weak self] in
|
||||
guard let self = self else { return }
|
||||
let xfRaw = self.xunfei.stop()
|
||||
if !xfRaw.isEmpty, xfRaw != appleRaw {
|
||||
log("dual: xunfei overwrite (\(appleRaw.count)→\(xfRaw.count) chars)")
|
||||
self.deleteChars(appleRaw.count)
|
||||
paste(xfRaw)
|
||||
}
|
||||
// 3) LLM 异步精修(可关): 配了 ~/ssw.ini 就跑,返非空且不同时再覆盖一次
|
||||
self.cleaner.refine(xfRaw) { [weak self] refined in
|
||||
guard let self = self else { return }
|
||||
guard !refined.isEmpty, refined != xfRaw else { return }
|
||||
log("dual: llm refine overwrite (\(xfRaw.count)→\(refined.count) chars)")
|
||||
self.deleteChars(xfRaw.count)
|
||||
paste(refined)
|
||||
}
|
||||
}
|
||||
return appleRaw
|
||||
}
|
||||
|
||||
private func deleteChars(_ n: Int) {
|
||||
guard n > 0 else { return }
|
||||
let src = CGEventSource(stateID: .combinedSessionState)
|
||||
// 51 = kVK_Delete(退格)
|
||||
for _ in 0..<n {
|
||||
let kd = CGEvent(keyboardEventSource: src, virtualKey: 51, keyDown: true)
|
||||
kd?.post(tap: .cghidEventTap)
|
||||
let ku = CGEvent(keyboardEventSource: src, virtualKey: 51, keyDown: false)
|
||||
ku?.post(tap: .cghidEventTap)
|
||||
usleep(3_000) // 3ms / 字符,13 字 ≈ 40ms
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
extension XunfeiEngine: URLSessionWebSocketDelegate {
|
||||
func urlSession(_ session: URLSession, webSocketTask: URLSessionWebSocketTask,
|
||||
didOpenWithProtocol p: String?) {
|
||||
opened = true
|
||||
openedSem.signal()
|
||||
log("xunfei: ws open")
|
||||
}
|
||||
func urlSession(_ session: URLSession, webSocketTask: URLSessionWebSocketTask,
|
||||
didCloseWith code: URLSessionWebSocketTask.CloseCode, reason: Data?) {
|
||||
log("xunfei: ws close \(code.rawValue)")
|
||||
// 对端结束也算"最后一帧到了":把 finalSem 抬起来,stop() 别再傻等
|
||||
finalSem.signal()
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 注入:剪贴板 + 合成 Cmd+V ----------
|
||||
let vKey: CGKeyCode = 9 // kVK_ANSI_V
|
||||
func paste(_ s: String) {
|
||||
let pb = NSPasteboard.general
|
||||
let old = pb.string(forType: .string)
|
||||
pb.clearContents()
|
||||
pb.setString(s, forType: .string)
|
||||
let src = CGEventSource(stateID: .combinedSessionState)
|
||||
let kd = CGEvent(keyboardEventSource: src, virtualKey: vKey, keyDown: true)
|
||||
kd?.flags = .maskCommand
|
||||
kd?.post(tap: .cghidEventTap)
|
||||
usleep(60_000)
|
||||
let ku = CGEvent(keyboardEventSource: src, virtualKey: vKey, keyDown: false)
|
||||
ku?.flags = .maskCommand
|
||||
ku?.post(tap: .cghidEventTap)
|
||||
log("pasted \(s.count) chars -> cursor")
|
||||
DispatchQueue.global().asyncAfter(deadline: .now() + 1.5) {
|
||||
if let old = old, pb.changeCount != 0 {
|
||||
pb.clearContents(); pb.setString(old, forType: .string)
|
||||
log("clipboard restored")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func finalize() {
|
||||
suppressDrag.v = false
|
||||
DispatchQueue.global().async {
|
||||
let t = Engine.current.stop()
|
||||
log("transcript: \"\(t)\"")
|
||||
if !t.isEmpty { paste(t) }
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 光标定位 app ----------
|
||||
func appAtCursor(_ pt: CGPoint) -> (pid: pid_t, bundle: String?) {
|
||||
guard let list = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID) as? [[String: Any]] else {
|
||||
return (NSWorkspace.shared.frontmostApplication?.processIdentifier ?? 0,
|
||||
NSWorkspace.shared.frontmostApplication?.bundleIdentifier)
|
||||
}
|
||||
for w in list {
|
||||
guard (w[kCGWindowLayer as String] as? Int) == 0,
|
||||
let b = w[kCGWindowBounds as String] as? [String: CGFloat] else { continue }
|
||||
if CGRect(x: b["X"] ?? 0, y: b["Y"] ?? 0, width: b["Width"] ?? 0, height: b["Height"] ?? 0).contains(pt) {
|
||||
let pid = pid_t(w[kCGWindowOwnerPID as String] as? Int ?? 0)
|
||||
return (pid, NSRunningApplication(processIdentifier: pid)?.bundleIdentifier)
|
||||
}
|
||||
}
|
||||
return (0, nil)
|
||||
}
|
||||
|
||||
// ---------- 微信过滤 + AX 输入框判定 ----------
|
||||
let textRoles: Set<String> = ["AXTextField", "AXTextArea", "AXComboBox", "AXSearchField"]
|
||||
|
||||
func frontmostIsWeChat(_ bundle: String?) -> Bool {
|
||||
bundle?.hasPrefix(weChatPrefix) == true
|
||||
}
|
||||
|
||||
func axCopy(_ el: AXUIElement, _ attr: String, _ tag: String) -> CFTypeRef? {
|
||||
var val: CFTypeRef?
|
||||
let err: AXError = AXUIElementCopyAttributeValue(el, attr as CFString, &val)
|
||||
if err != .success { log(" ax[\(tag)] \(attr) err=\(err.rawValue)") }
|
||||
return err == .success ? val : nil
|
||||
}
|
||||
|
||||
func axSet(_ el: AXUIElement, _ attr: String, _ v: CFTypeRef) {
|
||||
_ = AXUIElementSetAttributeValue(el, attr as CFString, v)
|
||||
}
|
||||
|
||||
func nudgeAX(_ pid: pid_t) {
|
||||
let el = AXUIElementCreateApplication(pid)
|
||||
axSet(el, "AXManualAccessibility", kCFBooleanTrue)
|
||||
axSet(el, "AXEnhancedUserInterface", kCFBooleanTrue)
|
||||
}
|
||||
|
||||
func focusedTextRole(pid: pid_t) -> String? {
|
||||
if let el = axCopy(AXUIElementCreateSystemWide(), kAXFocusedUIElementAttribute, "syswide"),
|
||||
let role = axCopy(el as! AXUIElement, kAXRoleAttribute, "syswide") as? String {
|
||||
return role
|
||||
}
|
||||
if let el = axCopy(AXUIElementCreateApplication(pid), kAXFocusedUIElementAttribute, "perapp"),
|
||||
let role = axCopy(el as! AXUIElement, kAXRoleAttribute, "perapp") as? String {
|
||||
return role
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---------- 长按检测线程 ----------
|
||||
Thread.detachNewThread {
|
||||
while true {
|
||||
let down = CGEventSource.buttonState(.combinedSessionState, button: .left)
|
||||
let loc = CGEvent(source: nil)?.location ?? .zero
|
||||
switch (state.v, down) {
|
||||
|
||||
case (.idle, true):
|
||||
state.v = .press(origin: loc, start: Date())
|
||||
maxMove.v = 0
|
||||
DispatchQueue.global().async { Engine.current.prepare() }
|
||||
|
||||
case (.press(let origin, let start), true):
|
||||
maxMove.v = max(maxMove.v, hypot(loc.x - origin.x, loc.y - origin.y))
|
||||
if Date().timeIntervalSince(start) * 1000 >= holdMs {
|
||||
if maxMove.v <= movePx {
|
||||
let (pid, bundle) = appAtCursor(loc)
|
||||
if frontmostIsWeChat(bundle) {
|
||||
log("skip: WeChat under cursor (\(bundle ?? "?"))")
|
||||
state.v = .dead
|
||||
DispatchQueue.global().async { Engine.current.cancel() }
|
||||
} else if pid != 0 {
|
||||
var role = focusedTextRole(pid: pid)
|
||||
if role == nil {
|
||||
nudgeAX(pid); usleep(250_000)
|
||||
role = focusedTextRole(pid: pid)
|
||||
}
|
||||
func startRec() {
|
||||
log("activate @ \(bundle ?? "?") role=\(role ?? "opaque")")
|
||||
NSSound.beep()
|
||||
if !Engine.ready {
|
||||
log("engine not ready: \(Engine.current is XunfeiEngine ? "xunfei creds/ws?" : "apple init?")")
|
||||
state.v = .dead
|
||||
DispatchQueue.global().async { Engine.current.cancel() }
|
||||
return
|
||||
}
|
||||
suppressDrag.v = true
|
||||
state.v = .recording(origin: origin, start: start)
|
||||
DispatchQueue.global().async { Engine.current.beginTask() }
|
||||
}
|
||||
if let role, textRoles.contains(role) {
|
||||
startRec()
|
||||
} else if let role {
|
||||
log("skip: role=\(role)")
|
||||
state.v = .dead
|
||||
DispatchQueue.global().async { Engine.current.cancel() }
|
||||
} else {
|
||||
log("allow: AX-opaque app \(bundle ?? "?") (fallback)")
|
||||
startRec()
|
||||
}
|
||||
} else {
|
||||
state.v = .dead
|
||||
DispatchQueue.global().async { Engine.current.cancel() }
|
||||
}
|
||||
} else {
|
||||
state.v = .dead
|
||||
DispatchQueue.global().async { Engine.current.cancel() }
|
||||
}
|
||||
}
|
||||
|
||||
case (.recording(_, let start), true):
|
||||
if Date().timeIntervalSince(start) >= maxRecSec {
|
||||
log("max record time, force-stop (paste current)")
|
||||
DispatchQueue.global().async {
|
||||
let t = Engine.current.flushNow()
|
||||
log("force-transcript: \"\(t)\"")
|
||||
if !t.isEmpty { paste(t) }
|
||||
}
|
||||
state.v = .idle
|
||||
}
|
||||
|
||||
case (.dead, true):
|
||||
break
|
||||
|
||||
case (_, false):
|
||||
switch state.v {
|
||||
case .recording: finalize()
|
||||
case .press: DispatchQueue.global().async { Engine.current.cancel() }
|
||||
default: break
|
||||
}
|
||||
state.v = .idle
|
||||
}
|
||||
usleep(pollUs)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 拖选抑制 tap ----------
|
||||
let tapPort = Box<CFMachPort?>(nil)
|
||||
func tapCallback(_ proxy: CGEventTapProxy, _ type: CGEventType, _ ev: CGEvent, _ ud: UnsafeMutableRawPointer?) -> Unmanaged<CGEvent>? {
|
||||
if type == .tapDisabledByTimeout || type == .tapDisabledByUserInput {
|
||||
if let p = tapPort.v { CGEvent.tapEnable(tap: p, enable: true) }
|
||||
return Unmanaged.passUnretained(ev)
|
||||
}
|
||||
return suppressDrag.v ? nil : Unmanaged.passUnretained(ev)
|
||||
}
|
||||
Thread.detachNewThread {
|
||||
let mask = CGEventMask(1 << CGEventType.leftMouseDragged.rawValue)
|
||||
guard let port = CGEvent.tapCreate(tap: .cgSessionEventTap, place: .headInsertEventTap,
|
||||
options: .defaultTap, eventsOfInterest: mask,
|
||||
callback: tapCallback, userInfo: nil) else {
|
||||
log("drag-tap install failed (no Accessibility) — 录音期间移动鼠标会拖选文字")
|
||||
return
|
||||
}
|
||||
let src = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, port, 0)
|
||||
CFRunLoopAddSource(CFRunLoopGetCurrent(), src, .commonModes)
|
||||
tapPort.v = port
|
||||
CFRunLoopRun()
|
||||
}
|
||||
|
||||
// ---------- 启动自检 + 权限请求 ----------
|
||||
freopen("/tmp/voicetype.log", "a", stdout)
|
||||
setbuf(stdout, nil)
|
||||
log("process=\(ProcessInfo.processInfo.processName) bundle=\(Bundle.main.bundleIdentifier ?? "nil")")
|
||||
log("Accessibility trusted: \(AXIsProcessTrusted())")
|
||||
if !AXIsProcessTrusted() {
|
||||
log(">>> 系统设置 → 隐私与安全性 → 辅助功能 → 勾选 VoiceType <<<")
|
||||
}
|
||||
AVCaptureDevice.requestAccess(for: .audio) { ok in log("mic granted: \(ok)") }
|
||||
SFSpeechRecognizer.requestAuthorization { st in log("speech auth: \(st.rawValue) (1=authorized)") }
|
||||
|
||||
dispatchMain()
|
||||
Reference in new issue
Block a user