commit b380eaf25235aefb8157add4bd4fa629dce20473 Author: admin Date: Fri Oct 2 19:23:47 2026 +0800 voicetype: Apple 端侧 + LLM 异步精修,可选 xunfei/dual 新增 apple-llm 引擎(默认):Apple SFSpeechRecognizer → 贴原文 → LLM (~1s, thinking=disabled) → 退格覆盖精修版。LLM 走 ~/ssw.ini [llm-api],无配置 静默回退 Apple 原文。 - 删 localClean(中文无空格,正则误伤'好啊'等正常词) - pickEngine() 默认 apple-llm - LLMCleaner 自动带 thinking: {type: 'disabled'} (GLM 7s+ → ~1s) - build.sh 新增 apple / apple-llm 模式,默认 apple-llm - README + AGENTS.md 同步更新 Co-Authored-By: Claude Opus 5.5 (1M context) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..dc3b426 --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +.DS_Store +*.swp +*.bak +.history/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..f904c89 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,67 @@ +# voicetype / VoiceType + +macOS 工具:**长按鼠标左键 → 说话 → 松开自动粘贴到光标处**(任意 app 输入框;微信窗口内自动放行给微信自己的语音输入)。 + +## 布局 + +``` +src/voicetype.swift 全部源码(单文件探针,~330 行) +src/Info.plist app 包描述(权限声明) +src/build.sh 构建+签名+安装+重启一条龙 +docs/voicetype/ 详细文档(用法/架构/踩坑) +``` + +## 常用命令 + +```bash +src/build.sh # 编译 → 签名 → 装到 /Applications → apple-llm 启动 → 打日志(默认) +src/build.sh apple # 纯端侧(无 LLM 精修) +src/build.sh apple-llm # 端侧 + LLM 异步精修 +src/build.sh xunfei # 同上,但经 xop 注入讯飞凭据 → 讯飞 IAT v2 WebSocket 引擎 +src/build.sh dual # apple + xunfei 并行 + LLM 精修(需 xop 注入) +tail -f /tmp/voicetype.log # 看判定/识别/粘贴日志 +killall VoiceType # 停止 +``` + +## 引擎切换(两个互不重叠的层面) + +1. **持久切换**:`echo apple|apple-llm|xunfei|dual > ~/.voicetype-engine`,任意方式启动都生效。 +2. **启动方式注入**:`src/build.sh xunfei` 用 `xop run --secret ... -- open ...` 临时把讯飞凭据塞进进程环境变量,**只对当次启动有效**,不会写到磁盘/源码。 + +讯飞凭据一次性入库(从 TTY 读值,源码/git/日志都不出现): +```bash +xop set xf_vt_APPID # APPID +xop set xf_vt_APISecret # APISecret +xop set xf_vt_APIKey # APIKey +``` + +LLM 配置(`apple-llm` / `dual` 需要;无 INI 则静默回退 apple 原文): +```ini +# ~/ssw.ini +[llm-api] +key=sk_xxxxxxxxxxxx +base-url=https://your-llm-proxy/v1 +model=GLM-5.3-Flash +``` +容忍全角破折号 `—` 与半角 `-` 等价。引擎请求时强制带 `thinking: {type: "disabled"`,绕过 GLM 推理模式(7s → 1s)。 + +## 红线(违反必翻车) + +1. **签名必须用固定 Apple Development 证书**(`src/build.sh` 里的 `IDENT`)。ad-hoc `-s -` 每次重编译身份都变,TCC 当成新 app,三项授权全作废。 +2. **必须以 .app 形式装在 /Applications 并 LaunchServices 注册**。裸 CLI 的 TCC 归属是宿主终端,行为混乱;/tmp 下的 app 进不了辅助功能列表。 +3. **权限三件套缺一不可**:麦克风(录音)、语音识别(apple 引擎)/ 辅助功能(全模式)。日志 `trusted: true` + `mic granted: true` + `speech auth: 3` 全绿才算就绪。 +4. **长按状态机:一次按下只判定一次**(激活/跳过后进 `.dead`,松开才复位)。打回 idle 会在按住期间循环重启录音引擎(已踩过)。 +5. **讯飞凭据永不落盘**:源码/git/日志/argv/历史都不能出现 APPID/APISecret/APIKey 明文。运行时统一经 `xop run --secret KEY=xf_vt_* -- open ...` 注入到 `ProcessInfo.processInfo.environment["XUNFEI_*"]`。引擎拿到空 env 时**直接报错退出**,不静默回退 apple(避免"以为用了讯飞结果识别差"的误判)。 +6. **LLM 凭据明文存于 `~/ssw.ini`**(用户偏好,不进 xop);`key` 字段不进源码/git/日志。LLM 失败/超时/未配 → 保留 Apple 原文,不静默丢弃。 + +## 核心设计(为什么这么做) + +- **鼠标左键轮询零权限可用**(`CGEventSource.buttonState`),Fn/Globe 键则轮询不可见——所以触发方式选左键而不是 Fn(实验证据见 GOTCHAS §1)。 +- **判定用"光标下最顶层窗口归属"**(CGWindowList z 序),不用 `frontmostApplication`——微信悬浮窗会霸占键盘焦点导致误判(踩坑见 GOTCHAS §5)。 +- **判定三层**:AX 焦点角色严格判断 → `AXManualAccessibility` 强开 Chromium 系 AX 再查 → 完全不透明则 app 级兜底放行。 +- **注入 = 写剪贴板 + 合成 ⌘V**,1.5s 后恢复原剪贴板(与微信同款)。 +- **识别引擎可插拔**:`protocol ASREngine` + `pickEngine()` 读 `~/.voicetype-engine`。默认 **Apple 端侧 SFSpeechRecognizer**(zh-CN,离线流式);备选 **讯飞 IAT v2 流式 WebSocket**(端点 `wss://iat-api.xfyun.cn/v2/iat`,HMAC-SHA256 鉴权,40ms 切片推送,服务端逐条返回拼接)。微信自家 ASR 是私有鉴权协议,外部不可用。 + +详细原理与全部踩坑:`docs/voicetype/README.md`、`docs/voicetype/GOTCHAS.md`。 + +详细原理与全部踩坑:`docs/voicetype/README.md`、`docs/voicetype/GOTCHAS.md`。 diff --git a/docs/voicetype/GOTCHAS.md b/docs/voicetype/GOTCHAS.md new file mode 100644 index 0000000..806c81e --- /dev/null +++ b/docs/voicetype/GOTCHAS.md @@ -0,0 +1,70 @@ +# GOTCHAS — VoiceType 踩坑全记录(2026-10-02 spike) + +## 1. Fn(🌐) 键轮询不可见;鼠标左键零权限可见 + +写了 `CGEventSource.flagsState` 双视图(combinedSessionState + hidSystemState)轮询探针,用户两轮实按 **8ms 采样粒度下一个翻转都抓不到**——Apple Silicon 的 Globe 键在更底层被系统消费,只有 Fn+字母组合时修饰标志才出现。而 `CGEventSource.buttonState(.combinedSessionState, .left)` 一次就干净抓到左键 DOWN/UP,**不需要任何权限**。→ 这就是触发方式选左键不选 Fn 的实验依据;若将来要做 Fn 触发,必须走辅助功能 CGEventTap(flagsChanged/keycode 63)。 + +## 2. 微信的"长按 Fn/鼠标说话"是微信自己的全局劫持 + +- 不是 macOS 系统功能;4.1.7+ 加入,授权「控制你的 Mac」后在微信外生效,结果走**剪贴板渠道**注入,可在微信 设置→快捷键 关闭。 +- 二进制证据(热更新副本 `~/Library/Containers/com.tencent.xinWeChat/Data/Documents/app_data/xplugin/plugins/Magicdy//extracted/wechat.dylib` 的导入符号):`mmui::VoiceInputKeyboardListener / VoiceInputMouseListener / VoiceInputMouseSessionController / VoiceInputFocusListener` + `AXIsProcessTrusted` + `CGEventTapCreateForPid`(按前台 app 建 tap,所以不出现在 CGGetEventTapList 全局列表) + `CGEventSourceButtonState/FlagsState`(轮询兜底) + `NSPasteboard`/`CGEventPost`。ASR 走自家 CGI(`voicetrans`/`uploadvoice`),私有鉴权协议,外部不可用 → 本工具用 Apple 端侧识别。 +- 包内 grep 中文字符串全空:微信 UI 串打包时不可直接 strings,要靠符号表和 `mmui::` 类名。 + +## 3. TCC 权限:裸 CLI 归属混乱,必须打包 .app + 固定签名 + +- 裸 CLI 从终端跑:麦克风 requestAccess 直接 false,授权对象是"责任进程"(宿主终端),行为不可预期。 +- 打包最小 .app(Info.plist 带 NSMicrophoneUsageDescription/NSSpeechRecognitionUsageDescription)+ `open` 启动 → TCC 弹窗正常附着,授权立即可用。 +- **ad-hoc 签名(-s -)每次重编译 cdhash 都变 → TCC 当成新 app,授权作废**。用钥匙串里已有的 Apple Development 证书签名,身份锚定证书,重编译授权保留。 +- /tmp 下的 app 不进「辅助功能」列表(未注册 LaunchServices)→ 装到 /Applications + `lsregister -f`。 + +## 4. AX API 三连坑 + +- 新 SDK 把 `AXUIElementCopyAttributeValue` 导入成 **3 参数版**(没有 error 出参):`(AXUIElement, CFString, UnsafeMutablePointer) -> AXError`。写 4 参报"extra argument in call",用小文件穷举签名确认。 +- 常量 `kAXFocusedUIElementAttribute` 是 String,要 `as CFString` 传。 +- 查询失败码要打出来看:`-25204`(CannotComplete)/`-25212`(NoValue=app 没暴露聚焦元素)是完全不同的病因。 + +## 5. 判定"长按发生在哪个 app":用光标下窗口,别用 frontmostApplication + +用户在备忘录长按,日志却显示 frontmost=微信——微信悬浮窗/语音面板会**霸占键盘焦点**,`NSWorkspace.frontmostApplication` 跟着错。改为 `CGWindowListCopyWindowInfo(optionOnScreenOnly)` 按 z 序找包含光标点的第一个 layer-0 窗口的 owner pid,所见即所得。 + +## 6. Chromium/Electron app 的 AX 树默认不建 + +`kAXFocusedUIElementAttribute` 返回 -25212。对 app 元素设置 `AXManualAccessibility` 和 `AXEnhancedUserInterface` = true,~250ms 后重查,多数 Chromium 系会建树。仍不行的(HexHub 实测)只能 app 级兜底放行。 + +## 7. 状态机 flap 循环 + +按下→判定失败→打回 `.idle`→下一轮询周期又当新按下→重启录音引擎……按住期间每 ~350ms 循环一次,表现为菜单栏录音图标狂闪、永远到不了激活。修法:**一次按下只判定一次**,判定完进 `.dead`,左键抬起才回 `.idle`。 + +## 8. 短句丢头 → 预卷录音 + +v1 在激活(350ms)后才启动录音引擎,用户按下即开口,前几百毫秒语音丢失 → ASR 报 `No speech detected (1110)`。修法:**鼠标按下瞬间**就启动引擎收缓冲,激活时才挂识别任务(`SFSpeechAudioBufferRecognitionRequest` 允许先 append 后建 task),普通点击抬起则丢弃。 + +## 9. 讯飞 IAT v2 鉴权在 URL query 里,不在 header + +`URLSessionWebSocketTask` 用 `webSocketTask(with: URL)` 时**所有参数都得到 URL 上**,没有 header 钩子(Authorization 不能直接挂)。所以讯飞的鉴权模式: +1. RFC1123 GMT 时间串 `date` +2. 拼接 `host: iat-api.xfyun.cn\ndate: \nGET /v2/iat HTTP/1.1` → HMAC-SHA256(secret) → base64 → `signature` +3. 拼 `api_key="...", algorithm="hmac-sha256", headers="host date request-line", signature="..."` → 再 base64 → `authorization` +4. URL = `wss://iat-api.xfyun.cn/v2/iat?authorization=<>&date=<>&host=iat-api.xfyun.cn` + +注意 URL 里的日期**不能跨分钟**:同一连接生命周期里服务端会校验(签名原始串里的 date 与请求 URL 里的 date 一致)。长会话需要周期重连(本工具单次录音 ≤60s,无此问题)。 + +## 10. AVAudioConverter 系统原生 PCM → 16k Int16 mono + +`AVAudioEngine.inputNode.outputFormat(forBus: 0)` 出来通常是 48k 或 44.1k Float32。讯飞要 16k/16bit/mono PCM,所以: +- `AVAudioConverter(from: inputFmt, to: targetFmt)` 直接搭好 +- `convert(to:from:error:)` 一行转完(回调式只用于压缩输入的流式场景,我们用不上) +- 输出 `AVAudioPCMBuffer.int16ChannelData[0]` → `UnsafeRawBufferPointer(start:count:)` → `Data.append(contentsOf:)` +- 注意 `Int16` 指针不能直接喂 `Data.append(_:count:)`,必须先按字节重解释 + +## 11. xop 注入 env 经 `open` 传递到 .app + +`xop run --secret X=xf_vt_X -- open /Applications/VoiceType.app` 时,env 经 `xop → open → LaunchServices → app` 这条链路一直传到底,`ProcessInfo.processInfo.environment["XUNFEI_X"]` 在 app 内能拿到。但有几种场景会丢: +- macOS 14+ 的 `open --new` 在某些沙盒场景下被 launchd 接管,env 被截断 → 备用 `xop run --secret ... -- /Applications/VoiceType.app/Contents/MacOS/VoiceType`(直接执行二进制,TCC 仍生效,因为同一 .app bundle) +- 用户从 Finder 双击 .app:不走 xop,env 没有,引擎立刻报错退出 + +## 复现/验证工具 + +- 事件 tap 全列表:`CGGetEventTapList`(源码曾存 /tmp/taplist.swift,随 /tmp 已失传,可按 GOTCHAS §2 结论重写) +- Fn/左键轮询探针:`CGEventSource.flagsState/.buttonState` + 8ms 循环 +- 日志判据:每次判定一行 `activate/skip/allow`,带 app bundle id + AX 角色/错误码 diff --git a/docs/voicetype/README.md b/docs/voicetype/README.md new file mode 100644 index 0000000..52a3860 --- /dev/null +++ b/docs/voicetype/README.md @@ -0,0 +1,100 @@ +# VoiceType — 长按左键语音输入(原型文档) + +灵感来自微信 Mac 版的"按住说话":在**任意 app 的输入框**长按鼠标左键说话,松开文字自动出现在光标处。本工具把它做成全局的,且微信窗口内自动放行。 + +## 使用 + +1. `src/build.sh` 构建安装(首次需手动开「系统设置 → 隐私与安全性 → 辅助功能 → VoiceType」,麦克风/语音识别弹窗点允许) +2. 点击输入框定位光标 → **按住左键 ≥0.35s**(哔一声开始)→ 说话 → **松开** → 出字 +3. 位移超过 10px 视为拖选,不触发;微信窗口内长按交给微信自己的语音输入 + +日志:`/tmp/voicetype.log`(每次判定都记录走了哪条路径、看到什么角色、识别文本、粘贴结果)。 + +## 引擎选择 + +`~/.voicetype-engine` 文件内容决定用哪个 ASR 引擎(空文件 = `apple-llm`): + +| 值 | 引擎 | 备注 | +|---|---|---| +| `apple-llm` (默认) | Apple 端侧 + LLM 异步精修 | 无凭据;LLM 走 `~/ssw.ini`;无 INI 则回退 Apple 原文 | +| `apple` | SFSpeechRecognizer 端侧 zh-CN | 离线,无凭据,响应快,无 LLM 清洗 | +| `xunfei` | 讯飞 IAT v2 WebSocket 流式 | **需凭据**;首次须 `xop set` 三个 vault 条目,启动走 `src/build.sh xunfei` | +| `dual` | apple + xunfei 并行 + LLM 精修 | 苹果先贴,讯飞覆盖,LLM 再精修 | + +### apple-llm / dual 的 LLM 配置(`~/ssw.ini`) + +LLM 走 OpenAI Chat Completions 兼容协议(任何 zhipu/openai/anthropic-compatible 代理都行)。 + +```ini +[llm-api] +key=sk_xxxxxxxxxxxx +base-url=https://your-llm-proxy/v1 +model=GLM-5.3-Flash +``` + +容忍全角破折号 `—` 与半角 `-` 等价(`[llm—api]` / `base—url` 都能识别)。 + +引擎会自动在请求里带 `thinking: {type: "disabled"`,绕开 GLM 系列推理模式(7s+ → 1s,实测 `thinking=disabled` 后稳定 ~0.9s)。 + +**失败兜底**: 无 INI / key 错 / 网络断 / 超时 → 保留 Apple 原文,绝不静默丢内容。 + +### 一次性:把讯飞凭据放进 xop vault(仅 xunfei/dual 需要) + +凭据**只**从 TTY 读取,不会出现在源码/git/日志/argv/历史里。`xop set ` 然后回车输入值。 + +```bash +xop set xf_vt_APPID # APPID +xop set xf_vt_APISecret # APISecret +xop set xf_vt_APIKey # APIKey +``` + +### 启动对应引擎 + +```bash +src/build.sh # 默认 = apple-llm +src/build.sh apple # 纯端侧(无 LLM 精修) +src/build.sh apple-llm # 端侧 + LLM(默认) +src/build.sh xunfei # 讯飞(需 xop 注入) +src/build.sh dual # 双引擎(需 xop 注入) +``` + +这一步做三件事: +1. 把 `~/.voicetype-engine` 写成对应模式(持久切换) +2. `xunfei` / `dual` 模式会 `xop run --secret XUNFEI_APPID=xf_vt_APPID --secret ... -- open /Applications/VoiceType.app` —— 凭据仅在子进程 env 里存在 +3. 引擎读 `ProcessInfo.processInfo.environment["XUNFEI_*"]`,**任一缺失则报错并丢弃本次录音**(不静默回退 apple,避免你以为用了讯飞但实际是 apple 给你识别差)。 + +## 判定流程(按住 350ms 时触发一次) + +``` +光标下窗口归属(CGWindowList z序) +├─ 微信(com.tencent.xinWeChat*) → 放行,静默 +└─ 其他 app: + ① AX 聚焦元素角色 ∈ {AXTextField, AXTextArea, AXComboBox, AXSearchField} → 激活 + ② 查不到 → 设置 AXManualAccessibility + AXEnhancedUserInterface,250ms 后重查 → 同① + ③ 仍查不到(AX 完全不透明,如 HexHub) → app 级兜底放行 +``` + +## 架构(单文件) + +| 组件 | 职责 | 权限 | +|---|---|---| +| 轮询线程(~125Hz) | `CGEventSource.buttonState` 检测左键,状态机 idle→press→recording/dead | 无 | +| 预卷录音 | **鼠标按下瞬间**就开始收音(`AVAudioEngine` tap),激活时才挂识别任务——短句不丢头 | 麦克风 | +| `AppleEngine` | `SFSpeechRecognizer(zh-CN)` 流式,`requiresOnDeviceRecognition = true`,松开 ~70ms 出结果 | 语音识别 | +| `AppleLLMEngine` | 包 AppleEngine + LLMCleaner;Apple 出文立刻贴,LLM 异步 ~1s 后回来若非空且不同则退格再贴 | 语音识别 + LLM | +| `XunfeiEngine` | `URLSessionWebSocketTask` → `wss://iat-api.xfyun.cn/v2/iat`,HMAC-SHA256 鉴权 URL,系统原生 PCM → 16k/16bit/mono 后 1280 字节/帧推送,逐条收 `data_status` 拼结果 | 麦克风(无语音识别权限要求) | +| `DualEngine` | apple + xunfei 并行,apple 先贴 → xunfei 覆盖 → LLM 异步精修再覆盖(可关) | 语音识别 + 麦克风 + LLM | +| `LLMCleaner` | OpenAI Chat Completions 兼容;强制带 `thinking: disabled`;`refine()` 异步回调,`clean()` 同步 1.5s 兜底 | — | +| `pickEngine()` | 读 `~/.voicetype-engine`,默认 `apple-llm` | — | +| 注入 | 写剪贴板 → 合成 ⌘V → 1.5s 后恢复原剪贴板 | 辅助功能 | +| 拖选抑制 tap | 录音期间吞掉 leftMouseDragged,防拖选文字 | 辅助功能 | +| 聚焦查询 | AX 系统级/per-app `kAXFocusedUIElementAttribute` + 角色 | 辅助功能 | + +## 已知边界(探针范围外) + +- AX 完全不透明的 app 里是 **app 级兜底**:长按它任何位置都触发,不只输入框 +- 剪贴板恢复只管文本(图片/文件内容不还原) +- 录音上限 60s;350ms 内慢速起步(<10px)的拖选理论可能误判 +- LSUIElement 无界面:无菜单栏开关、无开机自启、无常量配置面板(阈值都是源码顶部常量) +- 讯飞 IAT v2 单次请求 60s 上限(服务端);长语音会被截断 +- 讯飞计费按调用次数+音频时长,开发期注意用量 diff --git a/src/Info.plist b/src/Info.plist new file mode 100644 index 0000000..9fa4a24 --- /dev/null +++ b/src/Info.plist @@ -0,0 +1,15 @@ + + + + + CFBundleIdentifiercom.xs.voicetype + CFBundleNameVoiceType + CFBundleExecutableVoiceType + CFBundlePackageTypeAPPL + CFBundleShortVersionString0.1 + LSMinimumSystemVersion14.0 + LSUIElement + NSMicrophoneUsageDescription长按鼠标左键时录音进行语音输入 + NSSpeechRecognitionUsageDescription将录音转成文字 + + diff --git a/src/build.sh b/src/build.sh new file mode 100755 index 0000000..5ccee19 --- /dev/null +++ b/src/build.sh @@ -0,0 +1,86 @@ +#!/bin/bash +# 构建 VoiceType.app 并安装到 /Applications +# 红线: 必须用固定 Apple Development 证书签名 —— ad-hoc(-s -) 每次重编译身份都变, +# TCC 会当成新 app,麦克风/语音/辅助功能授权全部作废 +# +# 用法: +# src/build.sh 编译+签名+安装,默认启动 apple-llm(端侧+LLM 精修) +# src/build.sh apple 纯端侧(无 LLM,最低延迟) +# src/build.sh apple-llm 端侧识别 + LLM 异步精修(默认;需 ~/ssw.ini [llm-api]) +# src/build.sh xunfei 讯飞识别(端点 ~1s) +# src/build.sh dual apple + xunfei 并行 + LLM 精修(xunfei 覆盖后) +# +# 引擎持久切换(两种启动方式之外): +# echo apple > ~/.voicetype-engine +# echo apple-llm > ~/.voicetype-engine +# echo xunfei > ~/.voicetype-engine +# echo dual > ~/.voicetype-engine +# +# 凭据(讯飞,一次性,只需跑一次,xop 从 TTY 读值): +# xop set xf_vt_APPID # APPID +# xop set xf_vt_APISecret # APISecret +# xop set xf_vt_APIKey # APIKey +# +# LLM(apple-llm/dual 需要,否则静默回退到 apple 原文): +# ~/ssw.ini → [llm-api] key=… base-url=… model=… +# (apple-llm 无 ini 也照常工作,只是不精修) +set -e +cd "$(dirname "$0")" + +IDENT="Apple Development: sukyboor@163.com (Y4AJXC7Q9S)" +APP=/Applications/VoiceType.app +BIN=/tmp/voicetype-build + +MODE="${1:-apple-llm}" + +swiftc -O voicetype.swift -o "$BIN" + +mkdir -p "$APP/Contents/MacOS" +cp "$BIN" "$APP/Contents/MacOS/VoiceType" +cp Info.plist "$APP/Contents/Info.plist" +codesign --force --sign "$IDENT" "$APP" + +killall VoiceType 2>/dev/null || true +sleep 0.3 +: > /tmp/voicetype.log + +case "$MODE" in + xunfei) + # 把 ~/.voicetype-engine 也设上,使得日后 `open /Applications/VoiceType.app` + # 直接双击也能继续用 xunfei。凭据永远只走 xop 注入,不落盘不传 argv。 + current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )" + [ "$current" != "xunfei" ] && echo xunfei > ~/.voicetype-engine + xop run \ + --secret XUNFEI_APPID=xf_vt_APPID \ + --secret XUNFEI_API_SECRET=xf_vt_APISecret \ + --secret XUNFEI_API_KEY=xf_vt_APIKey \ + -- open "$APP" + ;; + dual) + current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )" + [ "$current" != "dual" ] && echo dual > ~/.voicetype-engine + xop run \ + --secret XUNFEI_APPID=xf_vt_APPID \ + --secret XUNFEI_API_SECRET=xf_vt_APISecret \ + --secret XUNFEI_API_KEY=xf_vt_APIKey \ + -- open "$APP" + ;; + apple) + current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )" + [ -n "$current" ] && [ "$current" != "apple" ] && echo apple > ~/.voicetype-engine + open "$APP" + ;; + apple-llm|*) + current="$( [ -f ~/.voicetype-engine ] && tr -d '[:space:]' < ~/.voicetype-engine || echo '' )" + [ -n "$current" ] && [ "$current" != "apple-llm" ] && echo apple-llm > ~/.voicetype-engine + [ -z "$current" ] && echo apple-llm > ~/.voicetype-engine + open "$APP" + ;; +esac + +sleep 2 +cat /tmp/voicetype.log +echo +echo "installed: $APP (pid $(pgrep -f VoiceType)) engine=$MODE" +[ "$MODE" = "xunfei" ] || [ "$MODE" = "dual" ] && echo "vault used: xf_vt_APPID / xf_vt_APISecret / xf_vt_APIKey (no values in logs/argv)" +[ "$MODE" = "apple-llm" ] && grep -q '\[llm-api\]' ~/ssw.ini 2>/dev/null && echo "llm: ~/ssw.ini [llm-api] 已配 → LLM 异步精修生效;否则回退 apple 原文" diff --git a/src/voicetype.swift b/src/voicetype.swift new file mode 100644 index 0000000..a7fd2d1 --- /dev/null +++ b/src/voicetype.swift @@ -0,0 +1,910 @@ +// voicetype: 长按鼠标左键 → 说话 → 松开自动粘贴到光标处 +// 引擎可选: apple / xunfei / dual(apple+xunfei 并行,apple 先贴) +// 凭据(仅 xunfei 需要): 经 xop run --secret 注入到环境变量,源码/git/日志绝不出现 +// LLM 清洗(可选): ~/ssw.ini 的 [llm-api] 段,key/base-url/model;无配置则跳过 +// 权限: 麦克风 + 语音识别(apple) / 麦克风 + 辅助功能(全模式) +import AppKit +import ApplicationServices +import AVFoundation +import CoreGraphics +import CryptoKit +import Speech + +// ---------- 配置 ---------- +let holdMs: Double = 350 // 长按激活阈值 +let movePx: CGFloat = 10 // 位移超过此值视为拖选,不激活 +let maxRecSec: Double = 60 // 最长录音(到点直接贴当前已识别部分,不继续等) +let pollUs: UInt32 = 8_000 // 轮询周期 8ms (~125Hz) +let weChatPrefix = "com.tencent.xinWeChat" +let engineConfigPath = NSHomeDirectory() + "/.voicetype-engine" +let llmIniPath = NSHomeDirectory() + "/ssw.ini" + +// ---------- INI 读取(仅支持 section/key=value 简单格式) ---------- +// 容忍 section/key 名里的全角破折号 `—` 与半角 `-` 等价(~/ssw.ini 历史上用 —) +func normINIKey(_ s: String) -> String { + s.replacingOccurrences(of: "—", with: "-").trimmingCharacters(in: .whitespaces) +} + +func loadINI(_ path: String) -> [String: [String: String]] { + guard let raw = try? String(contentsOfFile: path, encoding: .utf8) else { return [:] } + var sections: [String: [String: String]] = [:] + var current: String? + for line in raw.split(whereSeparator: { $0 == "\n" || $0 == "\r" }) { + let s = line.trimmingCharacters(in: .whitespaces) + if s.isEmpty || s.hasPrefix("#") || s.hasPrefix(";") { continue } + if s.hasPrefix("[") && s.hasSuffix("]") { + current = normINIKey(String(s.dropFirst().dropLast())) + sections[current!] = [:] + } else if let cur = current, let eq = s.firstIndex(of: "=") { + let k = normINIKey(String(s[.. LLMConfig? { + let s = loadINI(llmIniPath)["llm-api"] ?? [:] + guard let k = s["key"], !k.isEmpty, + let u = s["base-url"], !u.isEmpty, + let m = s["model"], !m.isEmpty else { + log("llm: ~/ssw.ini [llm-api] 缺失或字段不全,跳过 LLM 清洗") + return nil + } + log("llm: 加载 [llm-api] model=\(m) base=\(u.prefix(30))…") + return LLMConfig(key: k, baseURL: u, model: m) + } +} + +// 讯飞 IAT v2(参考 xfyun 开放平台"语音听写(流式版)WebAPI") +let xunfeiURL = "wss://iat-api.xfyun.cn/v2/iat" +let xunfeiHost = "iat-api.xfyun.cn" + +setbuf(stdout, nil) +func ts() -> String { let f = DateFormatter(); f.dateFormat = "HH:mm:ss.SSS"; return f.string(from: Date()) } +func log(_ s: String) { print("[\(ts())] \(s)") } + +// ---------- 共享状态(仅 poller 线程写) ---------- +final class Box { var v: T; init(_ v: T) { self.v = v } } +enum St { case idle; case press(origin: CGPoint, start: Date); case recording(origin: CGPoint, start: Date); case dead } +let state = Box(.idle) +let maxMove = Box(0) +let suppressDrag = Box(false) + +// ---------- ASR 引擎抽象 ---------- +protocol ASREngine { + func prepare() // 鼠标按下瞬间: 预卷/连WS + func beginTask() // 长按激活: 开始识别 + func cancel() // 普通点击抬起: 丢弃 + func stop() -> String // 录音中抬起: 阻塞等最终结果(dual 模式下返回 apple 的) + func flushNow() -> String // force-stop / maxRecSec: 不等,直接拿当前累计文本 + var readyForBegin: Bool { get } // prepare() 是否成功(凭据检查等) +} + +func pickEngine() -> ASREngine { + let raw = (try? String(contentsOfFile: engineConfigPath, encoding: .utf8))? + .trimmingCharacters(in: .whitespacesAndNewlines) ?? "apple-llm" + switch raw { + case "xunfei": + log("engine: xunfei (require XUNFEI_APPID/API_SECRET/API_KEY in env)") + return XunfeiEngine() + case "dual": + log("engine: dual (apple first, xunfei overwrite later, llm refine)") + return DualEngine() + case "apple-llm", "llm": + log("engine: apple + LLM refine (~/ssw.ini [llm-api] required, no INI → silent fallback to apple)") + return AppleLLMEngine() + default: + log("engine: apple (on-device, no LLM)") + return AppleEngine() + } +} + +final class Engine { + static var current: ASREngine = pickEngine() + static var ready: Bool { current.readyForBegin } +} + +// ---------- Apple 端侧 SFSpeechRecognizer ---------- +final class AppleEngine: ASREngine { + static let q = DispatchQueue(label: "voice.apple.rec") + let engine = AVAudioEngine() + var request: SFSpeechAudioBufferRecognitionRequest? + var task: SFSpeechRecognitionTask? + var recognizer = SFSpeechRecognizer(locale: Locale(identifier: "zh-CN")) + var text = "" + var finalSem = DispatchSemaphore(value: 0) + var prepared = false + + var readyForBegin: Bool { prepared } + + func prepare() { + text = "" + finalSem = DispatchSemaphore(value: 0) + let req = SFSpeechAudioBufferRecognitionRequest() + req.shouldReportPartialResults = true + req.addsPunctuation = true + if recognizer?.supportsOnDeviceRecognition == true { + req.requiresOnDeviceRecognition = true + } else { + log("apple: on-device unavailable → server fallback") + } + let input = engine.inputNode + let fmt = input.outputFormat(forBus: 0) + input.removeTap(onBus: 0) + input.installTap(onBus: 0, bufferSize: 2048, format: fmt) { [req] buf, _ in + req.append(buf) + } + request = req + do { try engine.start() } catch { log("apple engine start failed: \(error)") } + prepared = true + } + + func beginTask() { + guard let req = request, let rec = recognizer, prepared else { return } + task = rec.recognitionTask(with: req) { [weak self] r, err in + guard let self = self else { return } + if let r = r { + let t = r.bestTranscription.formattedString + if !t.isEmpty { self.text = t } + if r.isFinal { self.finalSem.signal() } + } + if err != nil { log("ASR err: \(err!)"); self.finalSem.signal() } + } + log("apple: recording (on-device: \(self.recognizer?.supportsOnDeviceRecognition == true))") + } + + func cancel() { + guard prepared else { return } + engine.inputNode.removeTap(onBus: 0) + engine.stop() + request = nil + task = nil + prepared = false + } + + func flushNow() -> String { + // SFSpeech 不太好中途取结果;返回当前 text(可能部分) + let t = text + cancel() + return t + } + + func stop() -> String { + engine.inputNode.removeTap(onBus: 0) + engine.stop() + request?.endAudio() + _ = finalSem.wait(timeout: .now() + 3) + task?.finish() + let t = text + text = "" + request = nil + task = nil + prepared = false + return t + } +} + +// ---------- Apple 端侧 + LLM 异步精修 ---------- +// 流程: Apple 识别 → 贴出原文 → LLM 异步精修 → 若非空且不同则删原文再贴精修版 +// 失败/超时/未配 INI: 不精修,保留 Apple 原文(不翻车) +final class AppleLLMEngine: ASREngine { + let apple = AppleEngine() + let cleaner: LLMCleaner + var lastRaw = "" // 覆盖阶段用: 知道要删多少字 + + init() { + self.cleaner = LLMCleaner(cfg: LLMConfig.load()) + } + + var readyForBegin: Bool { apple.readyForBegin } + + func prepare() { apple.prepare() } + func beginTask() { apple.beginTask() } + func cancel() { apple.cancel() } + func flushNow() -> String { return apple.flushNow() } + + func stop() -> String { + let raw = apple.stop() + lastRaw = raw + // 异步 LLM 精修: 非空且不同 → 删原文再贴 + DispatchQueue.global().async { [weak self] in + guard let self = self else { return } + self.cleaner.refine(raw) { [weak self] refined in + guard let self = self else { return } + guard !refined.isEmpty, refined != raw else { + log("apple-llm: refine skipped (empty or identical)") + return + } + log("apple-llm: llm refine overwrite (\(raw.count)→\(refined.count) chars)") + self.deleteChars(raw.count) + paste(refined) + } + } + return raw + } + + private func deleteChars(_ n: Int) { + guard n > 0 else { return } + let src = CGEventSource(stateID: .combinedSessionState) + for _ in 0.. (id: String, secret: String, key: String)? { + let env = ProcessInfo.processInfo.environment + guard let id = env["XUNFEI_APPID"], !id.isEmpty, + let s = env["XUNFEI_API_SECRET"], !s.isEmpty, + let k = env["XUNFEI_API_KEY"], !k.isEmpty else { return nil } + return (id, s, k) + } + + private func rfc1123() -> String { + let f = DateFormatter() + f.locale = Locale(identifier: "en_US_POSIX") + f.timeZone = TimeZone(identifier: "GMT") + f.dateFormat = "EEE, dd MMM yyyy HH:mm:ss z" + return f.string(from: Date()) + } + + // HMAC-SHA256 鉴权 URL(同步在 IAT v2 文档) + private func authURL(appID: String, secret: String, apiKey: String) -> URL? { + let date = rfc1123() + let signOrigin = "host: \(xunfeiHost)\ndate: \(date)\nGET /v2/iat HTTP/1.1" + let mac = HMAC.authenticationCode(for: Data(signOrigin.utf8), using: SymmetricKey(data: Data(secret.utf8))) + let sig = Data(mac).base64EncodedString() + let authOrigin = "api_key=\"\(apiKey)\", algorithm=\"hmac-sha256\", headers=\"host date request-line\", signature=\"\(sig)\"" + let auth = Data(authOrigin.utf8).base64EncodedString() + return URL(string: "\(xunfeiURL)?authorization=\(auth)&date=\(date)&host=\(xunfeiHost)") + } + + func prepare() { + // 先收尾旧的(session、wsTask),避免和上一轮并发触发 delegate 回调 + cleanup() + + guard let creds = envCreds() else { + lastErr = "missing env XUNFEI_APPID/API_SECRET/API_KEY — launch via `xop run --secret`" + log("XUNFEI: \(lastErr!)") + return + } + appID = creds.id + + // tap 必须用输入节点硬件格式 (Float32, 通常 44.1k 或 48k) —— 不然 installTap 抛 NSException + // 转码(浮点→int16 + 下采样)放到 serial queue 上做,绝不在音频线程跑。 + let input = engine.inputNode + hwFmt = input.outputFormat(forBus: 0) + + guard let url = authURL(appID: creds.id, secret: creds.secret, apiKey: creds.key) else { + lastErr = "auth URL build failed" + log("XUNFEI: \(lastErr!)") + return + } + + // 用串行 queue 保证 delegate 回调按序触发 + let delegateQ = OperationQueue() + delegateQ.maxConcurrentOperationCount = 1 + session = URLSession(configuration: .default, delegate: self, delegateQueue: delegateQ) + wsTask = session!.webSocketTask(with: url) + opened = false + wsTask?.resume() + + // tap 回调是音频实时线程,绝对禁止在这里 log / 阻塞 / 分配大对象。 + // 只做一次 unsafe 指针复制 + 把字节扔到 serial queue。 + engine.inputNode.removeTap(onBus: 0) + engine.inputNode.installTap(onBus: 0, bufferSize: 2048, format: hwFmt!) { [weak self] buf, _ in + guard let self = self, + let ch = buf.floatChannelData?[0] else { return } + let n = Int(buf.frameLength) + if n == 0 { return } + let raw = UnsafeRawBufferPointer(start: ch, count: n * 4) // Float32 = 4 bytes + let copy = Data(raw) // 拷贝出音频线程作用域 + self.q.async { self.rawBuf.append(copy); self.drainAndConvert() } + } + do { try engine.start() } catch { + lastErr = "engine start failed: \(error)" + log("XUNFEI: \(lastErr!)") + return + } + prepared = true + log("xunfei: ws connecting + recording (hw=\(hwFmt!.sampleRate)Hz fmt=\(hwFmt!.commonFormat.rawValue))") + } + + private func cleanup() { + frameTimer?.cancel(); frameTimer = nil + wsTask?.cancel(with: .goingAway, reason: nil) + wsTask = nil + session?.invalidateAndCancel() + session = nil + engine.inputNode.removeTap(onBus: 0) + if engine.isRunning { engine.stop() } + pcmBuf = Data(); rawBuf = Data(); resultText = "" + firstFrameSent = false + hwFmt = nil + waitingFinal = false + finalSem = DispatchSemaphore(value: 0) + openedSem = DispatchSemaphore(value: 0) + opened = false + } + + private func sendFrames() { + q.async { + // 先把硬件格式的 rawBuf 转码成 16k int16 mono 进 pcmBuf + self.drainAndConvert() + // 1280 bytes = 640 samples × 2 bytes/sample = 40ms @ 16kHz + while self.pcmBuf.count >= 1280 { + guard let ws = self.wsTask, ws.state == .running else { return } // ws 还没握上,等下一帧再试 + let chunk = self.pcmBuf.prefix(1280) + self.pcmBuf.removeFirst(1280) + let isFirst = !self.firstFrameSent + self.send(status: self.firstFrameSent ? 1 : 0, audio: Data(chunk)) + self.firstFrameSent = true + if isFirst { log("xunfei: sent first chunk (\(chunk.count)B) ws.state=\(ws.state.rawValue)") } + } + } + } + + // 把 rawBuf(硬件格式 Float32 mono @ hwRate)手动转成 16k int16 mono,入 pcmBuf。 + // 只在 serial queue 上跑,绝不在音频线程跑。 + // 解: Float32 -> Int16 (clip); 48k -> 16k 用 3:1 抽取(够 ASR 用,免 AVAudioConverter 崩) + private func drainAndConvert() { + guard hwFmt != nil else { return } + let hwFrameBytes = 4 // Float32 mono = 4 bytes/frame + let frameCount = rawBuf.count / hwFrameBytes + guard frameCount >= 3 else { return } // 至少一帧 16k 输出 + + let ratio = Int((hwFmt!.sampleRate / 16000).rounded()) + let outFrames = frameCount / max(ratio, 1) + guard outFrames > 0 else { return } + + rawBuf.withUnsafeBytes { rawPtr in + let floats = rawPtr.bindMemory(to: Float32.self).baseAddress! + var out = [Int16](repeating: 0, count: outFrames) + for i in 0.. String { + // 不发 status=2、不等;直接返回当前 resultText(中间识别累计) + let t = resultText + cleanup() + lastErr = nil + prepared = false + return t + } + + func stop() -> String { + frameTimer?.cancel(); frameTimer = nil + if prepared, wsTask != nil { + // 最后一帧: 先把 rawBuf 转完,再把 pcmBuf 尾巴推上去 + status=2 + waitingFinal = true // 发完 status=2 后,handle() 收到任何一条回复就 signal + q.sync { + self.drainAndConvert() + let tail = self.pcmBuf + self.pcmBuf = Data() + self.send(status: 2, audio: tail) + } + // 等服务端回完最终那条(经验 100~500ms),1s 兜底(waitingFinal + ws close 都会 signal) + _ = finalSem.wait(timeout: .now() + 1) + } + // 必须在 cleanup 前读 resultText —— cleanup 会把它清空 + let t = resultText + cleanup() + lastErr = nil + prepared = false + return t + } +} + +// ---------- LLM 清洗(可选):OpenAI Chat Completions 兼容 ---------- +final class LLMCleaner { + let cfg: LLMConfig? + init(cfg: LLMConfig?) { self.cfg = cfg } + + static let prompt = """ + 你是语音转写后处理助手。把下面这段口述语音转写文字清理成自然书面语中文: + - 去掉"嗯/啊/呃/那个/就是/然后/其实/对"等口水词 + - 加标点(句号、逗号、问号等),分句 + - 重组零碎短句为通顺长句 + - 保留全部原意,不增删内容 + 只输出清理后的文字,不要任何解释、标签或前后缀。 + """ + + func clean(_ raw: String) -> String { + // 同步接口(给单引擎调用):1.5s 兜底超时,默认返 raw + guard let cfg = cfg, !raw.isEmpty else { return raw } + let url = URL(string: cfg.baseURL.trimmingCharacters(in: CharacterSet(charactersIn: "/")) + "/chat/completions")! + var req = URLRequest(url: url, timeoutInterval: 1.5) + req.httpMethod = "POST" + req.setValue("application/json", forHTTPHeaderField: "Content-Type") + req.setValue("Bearer \(cfg.key)", forHTTPHeaderField: "Authorization") + var body: [String: Any] = [ + "model": cfg.model, + "messages": [ + ["role": "system", "content": Self.prompt], + ["role": "user", "content": raw], + ], + "temperature": 0.1, + ] + // 强制关推理: GLM-5.3-Flash 默认有 reasoning_content,7s+;关掉 <1s 且内容字段干净。 + // 不支持 thinking 的代理会静默忽略。 + body["thinking"] = ["type": "disabled"] + req.httpBody = try? JSONSerialization.data(withJSONObject: body) + let sem = DispatchSemaphore(value: 0) + var result = raw + URLSession.shared.dataTask(with: req) { data, _, _ in + defer { sem.signal() } + guard let data = data, + let j = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let choices = j["choices"] as? [[String: Any]], + let msg = choices.first?["message"] as? [String: Any], + let content = msg["content"] as? String else { return } + let cleaned = content.trimmingCharacters(in: .whitespacesAndNewlines) + if !cleaned.isEmpty { result = cleaned } + }.resume() + _ = sem.wait(timeout: .now() + 1.5) + return result + } + + // 异步精修(给 apple-llm / dual 引擎用):长 timeout,回调返 cleaned + func refine(_ raw: String, timeout: TimeInterval = 3, _ cb: @escaping (String) -> Void) { + guard let cfg = cfg, !raw.isEmpty else { cb(raw); return } + let url = URL(string: cfg.baseURL.trimmingCharacters(in: CharacterSet(charactersIn: "/")) + "/chat/completions")! + var req = URLRequest(url: url, timeoutInterval: timeout) + req.httpMethod = "POST" + req.setValue("application/json", forHTTPHeaderField: "Content-Type") + req.setValue("Bearer \(cfg.key)", forHTTPHeaderField: "Authorization") + var body: [String: Any] = [ + "model": cfg.model, + "messages": [ + ["role": "system", "content": Self.prompt], + ["role": "user", "content": raw], + ], + "temperature": 0.1, + ] + body["thinking"] = ["type": "disabled"] + req.httpBody = try? JSONSerialization.data(withJSONObject: body) + let done = DispatchSemaphore(value: 0) + URLSession.shared.dataTask(with: req) { data, _, _ in + defer { done.signal() } + guard let data = data, + let j = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let choices = j["choices"] as? [[String: Any]], + let msg = choices.first?["message"] as? [String: Any], + let content = msg["content"] as? String else { + cb(raw); return + } + let cleaned = content.trimmingCharacters(in: .whitespacesAndNewlines) + cb(cleaned.isEmpty ? raw : cleaned) + }.resume() + _ = done.wait(timeout: .now() + timeout + 0.5) + } +} + +// ---------- 本地清洗(无) ---------- +// LLM 之前实验过本地正则去"嗯/啊/那个"等口水词,但中文没空格, +// 任何字符级启发式都会误伤"好啊""什么啊"等正常语料 → 已弃用,清洗只走 LLM + +// ---------- Dual 引擎:apple + xunfei 并行,apple 先贴、xunfei 后续覆盖 ---------- +final class DualEngine: ASREngine { + let apple = AppleEngine() + let xunfei = XunfeiEngine() + let cleaner: LLMCleaner + var lastAppleText = "" // 给覆盖阶段用:知道要删多少字 + var overwriteScheduled = false + + init() { + self.cleaner = LLMCleaner(cfg: LLMConfig.load()) + } + + var readyForBegin: Bool { apple.readyForBegin && xunfei.readyForBegin } + + func prepare() { + apple.prepare() + xunfei.prepare() + } + + func beginTask() { + apple.beginTask() + xunfei.beginTask() + } + + func cancel() { + apple.cancel() + xunfei.cancel() + } + + func flushNow() -> String { + return apple.flushNow() + } + + func stop() -> String { + // 1) 收 apple(~1s) → 立刻贴出原始文本(不本地清洗) + let appleRaw = apple.stop() + lastAppleText = appleRaw + // 2) 异步: xunfei 识别完 → 覆盖 apple;再走 LLM 异步精修 → 若更优再覆盖一次 + DispatchQueue.global().async { [weak self] in + guard let self = self else { return } + let xfRaw = self.xunfei.stop() + if !xfRaw.isEmpty, xfRaw != appleRaw { + log("dual: xunfei overwrite (\(appleRaw.count)→\(xfRaw.count) chars)") + self.deleteChars(appleRaw.count) + paste(xfRaw) + } + // 3) LLM 异步精修(可关): 配了 ~/ssw.ini 就跑,返非空且不同时再覆盖一次 + self.cleaner.refine(xfRaw) { [weak self] refined in + guard let self = self else { return } + guard !refined.isEmpty, refined != xfRaw else { return } + log("dual: llm refine overwrite (\(xfRaw.count)→\(refined.count) chars)") + self.deleteChars(xfRaw.count) + paste(refined) + } + } + return appleRaw + } + + private func deleteChars(_ n: Int) { + guard n > 0 else { return } + let src = CGEventSource(stateID: .combinedSessionState) + // 51 = kVK_Delete(退格) + for _ in 0.. cursor") + DispatchQueue.global().asyncAfter(deadline: .now() + 1.5) { + if let old = old, pb.changeCount != 0 { + pb.clearContents(); pb.setString(old, forType: .string) + log("clipboard restored") + } + } +} + +func finalize() { + suppressDrag.v = false + DispatchQueue.global().async { + let t = Engine.current.stop() + log("transcript: \"\(t)\"") + if !t.isEmpty { paste(t) } + } +} + +// ---------- 光标定位 app ---------- +func appAtCursor(_ pt: CGPoint) -> (pid: pid_t, bundle: String?) { + guard let list = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID) as? [[String: Any]] else { + return (NSWorkspace.shared.frontmostApplication?.processIdentifier ?? 0, + NSWorkspace.shared.frontmostApplication?.bundleIdentifier) + } + for w in list { + guard (w[kCGWindowLayer as String] as? Int) == 0, + let b = w[kCGWindowBounds as String] as? [String: CGFloat] else { continue } + if CGRect(x: b["X"] ?? 0, y: b["Y"] ?? 0, width: b["Width"] ?? 0, height: b["Height"] ?? 0).contains(pt) { + let pid = pid_t(w[kCGWindowOwnerPID as String] as? Int ?? 0) + return (pid, NSRunningApplication(processIdentifier: pid)?.bundleIdentifier) + } + } + return (0, nil) +} + +// ---------- 微信过滤 + AX 输入框判定 ---------- +let textRoles: Set = ["AXTextField", "AXTextArea", "AXComboBox", "AXSearchField"] + +func frontmostIsWeChat(_ bundle: String?) -> Bool { + bundle?.hasPrefix(weChatPrefix) == true +} + +func axCopy(_ el: AXUIElement, _ attr: String, _ tag: String) -> CFTypeRef? { + var val: CFTypeRef? + let err: AXError = AXUIElementCopyAttributeValue(el, attr as CFString, &val) + if err != .success { log(" ax[\(tag)] \(attr) err=\(err.rawValue)") } + return err == .success ? val : nil +} + +func axSet(_ el: AXUIElement, _ attr: String, _ v: CFTypeRef) { + _ = AXUIElementSetAttributeValue(el, attr as CFString, v) +} + +func nudgeAX(_ pid: pid_t) { + let el = AXUIElementCreateApplication(pid) + axSet(el, "AXManualAccessibility", kCFBooleanTrue) + axSet(el, "AXEnhancedUserInterface", kCFBooleanTrue) +} + +func focusedTextRole(pid: pid_t) -> String? { + if let el = axCopy(AXUIElementCreateSystemWide(), kAXFocusedUIElementAttribute, "syswide"), + let role = axCopy(el as! AXUIElement, kAXRoleAttribute, "syswide") as? String { + return role + } + if let el = axCopy(AXUIElementCreateApplication(pid), kAXFocusedUIElementAttribute, "perapp"), + let role = axCopy(el as! AXUIElement, kAXRoleAttribute, "perapp") as? String { + return role + } + return nil +} + +// ---------- 长按检测线程 ---------- +Thread.detachNewThread { + while true { + let down = CGEventSource.buttonState(.combinedSessionState, button: .left) + let loc = CGEvent(source: nil)?.location ?? .zero + switch (state.v, down) { + + case (.idle, true): + state.v = .press(origin: loc, start: Date()) + maxMove.v = 0 + DispatchQueue.global().async { Engine.current.prepare() } + + case (.press(let origin, let start), true): + maxMove.v = max(maxMove.v, hypot(loc.x - origin.x, loc.y - origin.y)) + if Date().timeIntervalSince(start) * 1000 >= holdMs { + if maxMove.v <= movePx { + let (pid, bundle) = appAtCursor(loc) + if frontmostIsWeChat(bundle) { + log("skip: WeChat under cursor (\(bundle ?? "?"))") + state.v = .dead + DispatchQueue.global().async { Engine.current.cancel() } + } else if pid != 0 { + var role = focusedTextRole(pid: pid) + if role == nil { + nudgeAX(pid); usleep(250_000) + role = focusedTextRole(pid: pid) + } + func startRec() { + log("activate @ \(bundle ?? "?") role=\(role ?? "opaque")") + NSSound.beep() + if !Engine.ready { + log("engine not ready: \(Engine.current is XunfeiEngine ? "xunfei creds/ws?" : "apple init?")") + state.v = .dead + DispatchQueue.global().async { Engine.current.cancel() } + return + } + suppressDrag.v = true + state.v = .recording(origin: origin, start: start) + DispatchQueue.global().async { Engine.current.beginTask() } + } + if let role, textRoles.contains(role) { + startRec() + } else if let role { + log("skip: role=\(role)") + state.v = .dead + DispatchQueue.global().async { Engine.current.cancel() } + } else { + log("allow: AX-opaque app \(bundle ?? "?") (fallback)") + startRec() + } + } else { + state.v = .dead + DispatchQueue.global().async { Engine.current.cancel() } + } + } else { + state.v = .dead + DispatchQueue.global().async { Engine.current.cancel() } + } + } + + case (.recording(_, let start), true): + if Date().timeIntervalSince(start) >= maxRecSec { + log("max record time, force-stop (paste current)") + DispatchQueue.global().async { + let t = Engine.current.flushNow() + log("force-transcript: \"\(t)\"") + if !t.isEmpty { paste(t) } + } + state.v = .idle + } + + case (.dead, true): + break + + case (_, false): + switch state.v { + case .recording: finalize() + case .press: DispatchQueue.global().async { Engine.current.cancel() } + default: break + } + state.v = .idle + } + usleep(pollUs) + } +} + +// ---------- 拖选抑制 tap ---------- +let tapPort = Box(nil) +func tapCallback(_ proxy: CGEventTapProxy, _ type: CGEventType, _ ev: CGEvent, _ ud: UnsafeMutableRawPointer?) -> Unmanaged? { + if type == .tapDisabledByTimeout || type == .tapDisabledByUserInput { + if let p = tapPort.v { CGEvent.tapEnable(tap: p, enable: true) } + return Unmanaged.passUnretained(ev) + } + return suppressDrag.v ? nil : Unmanaged.passUnretained(ev) +} +Thread.detachNewThread { + let mask = CGEventMask(1 << CGEventType.leftMouseDragged.rawValue) + guard let port = CGEvent.tapCreate(tap: .cgSessionEventTap, place: .headInsertEventTap, + options: .defaultTap, eventsOfInterest: mask, + callback: tapCallback, userInfo: nil) else { + log("drag-tap install failed (no Accessibility) — 录音期间移动鼠标会拖选文字") + return + } + let src = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, port, 0) + CFRunLoopAddSource(CFRunLoopGetCurrent(), src, .commonModes) + tapPort.v = port + CFRunLoopRun() +} + +// ---------- 启动自检 + 权限请求 ---------- +freopen("/tmp/voicetype.log", "a", stdout) +setbuf(stdout, nil) +log("process=\(ProcessInfo.processInfo.processName) bundle=\(Bundle.main.bundleIdentifier ?? "nil")") +log("Accessibility trusted: \(AXIsProcessTrusted())") +if !AXIsProcessTrusted() { + log(">>> 系统设置 → 隐私与安全性 → 辅助功能 → 勾选 VoiceType <<<") +} +AVCaptureDevice.requestAccess(for: .audio) { ok in log("mic granted: \(ok)") } +SFSpeechRecognizer.requestAuthorization { st in log("speech auth: \(st.rawValue) (1=authorized)") } + +dispatchMain() \ No newline at end of file