apix 发版后最容易静默失效:location 没挂 / shm 没装载都不报错,只是 404,
而回归打假上游、agent.sh verify 又不发真实请求 —— 之前完全没有覆盖。
新增 tests/apix_check.py:
- 从 /api/v1/me(mdm host + X-API-Key 头)拉能力目录,逐个探 enabled 条目;
- vendor → GET /apix/<slug>?<p>=<占位>;passthrough → GET /apixp/<slug>/health;
- **无效 Key 预检**:证明 location 挂着 + 鉴权生效(与上游是否抽风无关);
- 判定按错误体归属:网关体({"error":{...}} / {"code":..,"msg":..})→ 失败;
上游体({"detail":...} / WAF 页)→ 只告警,不染红发版验证。
实测踩到的三个坑(已写进 README):
1. urllib 默认 UA `Python-urllib/3.x` 被供应商边缘 Cloudflare 拦(403 error code:
1010),看着像 IP 被封,换 curl UA 立刻 200;
2. passthrough tail 是 /apixp/<slug> 之后的全部(上游 /health → /apixp/ocr/health),
usage_hint 里的 /<tail> 要换成 slug,照抄是上游 404;
3. vendor 参数要放白名单值:accel 给 example.com 是 400 target host not allowed
(链路通,属告警),换 raw.githubusercontent 小文件即 200。
live_check.sh 改为三步(LLM 真实流量 / apix 探针 / 环境验收),加 --apix-only。
线上实测:LLM 8/8 + apix 3/3 + 验收全绿。
136 lines
5.4 KiB
Bash
Executable File
136 lines
5.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# ---------------------------------------------------------------------------
|
||
# 上线后验证(live smoke)—— 发版流程的最后一步
|
||
#
|
||
# 本机回归(tests/run.sh)跑的是**假上游**,验不到真实链路;`deploy/agent.sh verify`
|
||
# 又只验管理台页面与后端常量,**一个真实请求都不发**。真正确认「新版在线上能干活」
|
||
# 需要三步合起来:
|
||
#
|
||
# 1. LLM 真实流量:tests/test_models.py —— 4 slot × OpenAI/Anthropic 真实调用
|
||
# + `/v1/models`(下游 Key 鉴权、slot 路由、Key 替换、出站头都在这条链上)
|
||
# 2. apix 探针:tests/apix_check.py —— `/apix/<slug>`(vendor) 与 `/apixp/<slug>`
|
||
# (passthrough) 是否真的转发(**发版后 apix 最容易静默失效**:location 没挂、
|
||
# shm 没装载都不会报错,只是 404)
|
||
# 3. 环境验收:deploy/agent.sh verify —— 登录、页面新标记、subdomain_rule、趋势、
|
||
# 账号级 load(确认跑的确实是新镜像,而不是旧容器没换)
|
||
#
|
||
# 用法:
|
||
# bash tests/live_check.sh # 三步都跑(key 默认 ~/ssw.txt)
|
||
# bash tests/live_check.sh ~/.keys/ipass.txt # 指定 key 文件
|
||
# bash tests/live_check.sh --models-only # 只跑 LLM 真实流量
|
||
# bash tests/live_check.sh --apix-only # 只跑 apix 探针
|
||
# bash tests/live_check.sh --verify-only # 只跑环境验收
|
||
# IPASS_LIVE_KEY_FILE=~/ssw.txt bash tests/live_check.sh
|
||
#
|
||
# Key 文件解析(踩过的坑):文件里常写成 `sk=<key>` 或带 `#` 注释行,
|
||
# **取整行会拿到 `sk=sk_xxx` → 全量 401 invalid_api_key**。这里统一:
|
||
# 跳过空行/注释行 → 有 `=` 取第一个 `=` 之后 → 去引号与空白 → 只认 `sk_` 开头。
|
||
#
|
||
# ⚠️ 全程不打印 Key 原文(子脚本自带 mask,只显示 `sk_2df...2dae`)。
|
||
# ---------------------------------------------------------------------------
|
||
set -uo pipefail
|
||
|
||
HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||
REPO=$(cd "$HERE/.." && pwd)
|
||
|
||
KEY_FILE="${IPASS_LIVE_KEY_FILE:-$HOME/ssw.txt}"
|
||
RUN_MODELS=1
|
||
RUN_APIX=1
|
||
RUN_VERIFY=1
|
||
|
||
while [ $# -gt 0 ]; do
|
||
case "$1" in
|
||
--models-only) RUN_APIX=0; RUN_VERIFY=0; shift ;;
|
||
--apix-only) RUN_MODELS=0; RUN_VERIFY=0; shift ;;
|
||
--verify-only) RUN_MODELS=0; RUN_APIX=0; shift ;;
|
||
-h|--help) sed -n '2,32p' "${BASH_SOURCE[0]}"; exit 0 ;;
|
||
-*) echo "✗ 未知参数:$1(用 -h 看用法)" >&2; exit 2 ;;
|
||
*) KEY_FILE="$1"; shift ;;
|
||
esac
|
||
done
|
||
|
||
TOTAL=$((RUN_MODELS + RUN_APIX + RUN_VERIFY))
|
||
|
||
# python3:优先隔离的托管运行时,其次系统的(与 tests/run.sh 同一套选择逻辑)
|
||
PY=""
|
||
for c in /Users/xs/.workbuddy/binaries/python/versions/3.13.12/bin/python3 python3; do
|
||
if command -v "$c" >/dev/null 2>&1; then PY="$c"; break; fi
|
||
done
|
||
[ -n "$PY" ] || { echo "✗ 找不到 python3" >&2; exit 2; }
|
||
|
||
# 从 key 文件里挑出真正的 Key(见文件头注释:支持 `k=v` / 注释行 / 裸 key)
|
||
pick_key() {
|
||
awk '
|
||
/^[[:space:]]*#/ { next }
|
||
/^[[:space:]]*$/ { next }
|
||
{
|
||
line = $0
|
||
sub(/\r$/, "", line)
|
||
gsub(/^[[:space:]]+|[[:space:]]+$/, "", line)
|
||
if (index(line, "=") > 0) { line = substr(line, index(line, "=") + 1) }
|
||
gsub(/["'"'"']/, "", line)
|
||
if (line ~ /^sk_/) { print line; exit }
|
||
}' "$1"
|
||
}
|
||
|
||
rc=0
|
||
step=0
|
||
KEY=""
|
||
|
||
# 要发真实请求的两步都得先有 Key
|
||
if [ "$RUN_MODELS" -eq 1 ] || [ "$RUN_APIX" -eq 1 ]; then
|
||
if [ ! -f "$KEY_FILE" ]; then
|
||
echo "✗ 找不到 Key 文件:$KEY_FILE"
|
||
echo " 传路径或用 IPASS_LIVE_KEY_FILE 指定;文件内容可以是裸 key 或 sk=<key>"
|
||
exit 1
|
||
fi
|
||
KEY="$(pick_key "$KEY_FILE")"
|
||
if [ -z "$KEY" ]; then
|
||
echo "✗ $KEY_FILE 里没解析到 Key(期望一行 sk_ 开头,或 sk=<key> 形式)"
|
||
exit 1
|
||
fi
|
||
fi
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 1/3 LLM 真实流量
|
||
# ---------------------------------------------------------------------------
|
||
if [ "$RUN_MODELS" -eq 1 ]; then
|
||
step=$((step + 1))
|
||
echo "== $step/$TOTAL LLM 真实流量(test_models.py:4 slot × OpenAI/Anthropic) =="
|
||
"$PY" "$HERE/test_models.py" "$KEY" --api both --concurrent --timeout 60 || rc=1
|
||
echo
|
||
fi
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 2/3 apix 探针
|
||
# ---------------------------------------------------------------------------
|
||
if [ "$RUN_APIX" -eq 1 ]; then
|
||
step=$((step + 1))
|
||
echo "== $step/$TOTAL apix 探针(apix_check.py:vendor + passthrough) =="
|
||
# ⚠️ 判定口径见脚本头:只有「网关侧没生效」才算失败,
|
||
# 上游业务拒绝 / 供应商 WAF(如 Cloudflare 1010)只告警,不染红发版验证。
|
||
"$PY" "$HERE/apix_check.py" "$KEY" --timeout 40 || rc=1
|
||
echo
|
||
fi
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 3/3 正式环境验收
|
||
# ---------------------------------------------------------------------------
|
||
if [ "$RUN_VERIFY" -eq 1 ]; then
|
||
step=$((step + 1))
|
||
echo "== $step/$TOTAL 正式环境验收(deploy/agent.sh verify) =="
|
||
if [ ! -f "$REPO/.env" ]; then
|
||
echo "· 跳过:仓库根没有 .env(控制面凭据只从这里读,没有就跑不了验收)"
|
||
else
|
||
bash "$REPO/deploy/agent.sh" verify || rc=1
|
||
fi
|
||
echo
|
||
fi
|
||
|
||
if [ "$rc" -eq 0 ]; then
|
||
echo "✔ 上线后验证通过"
|
||
else
|
||
echo "✗ 上线后验证有失败项(见上面输出)"
|
||
fi
|
||
exit $rc
|