3199d16「TEE 远程调度面与节点自报」把 control_plane.rs 进了仓,但 reconciler.rs 没跟着改;main.rs 那边整块重复粘贴了 15 个顶层 fn。 后续两个提交只改文档没碰代码,破损一直潜伏 —— HEAD 编译直接失败: error[E0599]: no method named `set_require_chip_sig` error[E0063]: missing field `chip_sig_failed` in initializer error[E0308]: expected `NodeRegistry`, found `Arc<NodeRegistry>` error[E0428]: the name `cmd_tee` is defined multiple times(15 组) 上传路由归位(guest 边界 + 路由漂移): - 路由形状只有一份,在 hostd 侧:api::attestation_upload_path(id)。 - 宿主经 9p 手牌把完整上传路径注入 guest(手牌第 4 份文件 upload_path)。 - mpc-tee 不再持有 "/sandboxes" 字面量,guest 产物里不再出现该路由。 - CLI 新增 --upload-path-file;guest init.sh 读手牌并传下去。 - 三道防线:源码层 Rust 单测 + 源码层 CI 脚本 + 产物层 ELF 探针。 产物层判据有盲区(字面量进死代码会被 LTO 裁掉 ⇒ 假绿), 已用三形态破坏实验验证源码层两道能转红。 对账器三态语义(证据成立 ≠ 可公布): - is_verified() 决定实例去留;publishable(require_chip_sig) 决定端点外泄。 - chip_sig_failed 区分「没验」与「验了没过」:前者判 verified 但按策略 不公布,后者硬拒绝判 rejected 并回收实例。 - 证明漂移先从 placed 移除旧记录再回收实例,否则 proven_endpoints() 会继续公布证明已不通过的旧端点(真 fail-open)。 验证:暂存区快照独立编译 exit 0;mpc-tee 230 passed / mpc-hostd 171 passed / mpc-cli 69 passed;tools/ci.sh --fast exit 0(binary 判据 39/39, 含「/sandboxes 不得出现在 guest 产物」)。 真机 9p 探针修掉三个真 bug:init_b64 取的是 stdout(空)而非文件内容 ⇒ 远端 /init 0 字节 ⇒ 内核 ENOEXEC;qemu 参数用 \ 续行被 ssh 管道吃掉; 本地 90 轮轮询每轮走 xop 触摸验证 ⇒ 改为远端轮询一次取回。
1945 lines
71 KiB
Python
1945 lines
71 KiB
Python
#!/usr/bin/env python3
|
||
"""宿主控制面(mpc-hostd)端到端自测。
|
||
|
||
# 为什么这条必须进**主**门禁
|
||
|
||
宿主控制面是「MPC 控制面直接操作 TEE」这条主线的落点。它的失效方式
|
||
**全部是静默的**:
|
||
|
||
- 进程被杀了但状态还是 running → 站点端以为 TEE 在跑;
|
||
- `/attestation` 把「没做芯片验签」说成「验证通过」 → 密钥发给了不该拿的人;
|
||
- 复用 pid 时对着别人的进程发信号 → 误杀别的作业;
|
||
- `GET /sandboxes` 不是裸数组 → e2b SDK 直接反序列化失败。
|
||
|
||
这些都不会触发编译错误,也不会在单元测试里暴露 —— 单元测试测的是
|
||
「函数返回值对不对」,这里测的是「**整条链路**在真 socket 上跑不跑得通、
|
||
语义对不对」。
|
||
|
||
# 覆盖什么
|
||
|
||
1. 真实起一个 hostd(真 socket、真的起进程、真的杀进程);
|
||
2. e2b 兼容面:`GET /sandboxes` 返回**裸数组**、字段名与 e2b 一致;
|
||
3. `/attestation` 的四个布尔**分开**,且 `chip_sig` 不为真时结论明说「没验签」;
|
||
4. 实例真的被回收(进程真没了,不是状态改了而已);
|
||
5. 未知实例 404、畸形请求 400;
|
||
6. 重启后按状态文件恢复,且**孤儿进程**被正确识别(pid 复用不许误杀);
|
||
7. 协议层边界:`chunked` 被拒、重复 `Content-Length` 被拒、响应体完整;
|
||
8. 节点自报(Cubelet 自描述):自报真的推到控制面,能力位与探针口径一致,
|
||
没配能力时**一个请求都不发**(不许假就绪);
|
||
9. 集群控制面(CubeMaster 等价物):自报 → 注册表 → 期望态 → 选址下发 →
|
||
证明判定 → 会话端点外泄与否,整条闭环在真 socket 上跑通;
|
||
10. 集群控制面的远程暴露守卫:非回环必须配共享密钥 **且** 显式 `--allow-remote`。
|
||
|
||
# 协议层由 axum / hyper 承担(2026-11-07 决策反转)
|
||
|
||
原先自研 HTTP/1.1 解析,出现过「响应头与 body 分两次写、只读一次拿到半个响应」
|
||
的偶发失败(25 轮门禁红 2 次)。现在解析、分块、长度校验、连接管理全部交给
|
||
hyper;下面判的是**我们仍然要保证的契约**,不是 hyper 的内部实现:
|
||
|
||
- 重复 `Content-Length`:hyper 在解析层直接 400 + 空体(请求没进业务层);
|
||
- `Transfer-Encoding: chunked`:我们在 handler 里显式 400 并说明原因;
|
||
- 连续多次请求都必须拿到完整响应体 —— 这是钉死上面那个偶发失败的回归探针。
|
||
|
||
# 判据纪律
|
||
|
||
- 每条负向用例都带**期望的拒绝特征**且必须匹配 ——
|
||
只判「失败」不判失败原因,就分不清「按设计拒绝」和「碰巧崩了」。
|
||
- 失败就非零退出,不 `|| true` 吞掉。
|
||
"""
|
||
|
||
import http.server
|
||
import json
|
||
import os
|
||
import signal
|
||
import socket
|
||
import subprocess
|
||
import sys
|
||
import threading
|
||
import time
|
||
import urllib.error
|
||
import urllib.request
|
||
|
||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||
BIN = os.path.join(REPO, "target", "debug", "mpc")
|
||
|
||
PASS = "\033[32m ok\033[0m"
|
||
FAIL = "\033[31m FAIL\033[0m"
|
||
|
||
failures = []
|
||
|
||
|
||
def check(name, cond, detail=""):
|
||
if cond:
|
||
print(PASS, name)
|
||
else:
|
||
print(FAIL, name, detail)
|
||
failures.append(name)
|
||
|
||
|
||
def free_port():
|
||
s = socket.socket()
|
||
s.bind(("127.0.0.1", 0))
|
||
port = s.getsockname()[1]
|
||
s.close()
|
||
return port
|
||
|
||
|
||
def request(port, method, path, body=None, timeout=10, token=None):
|
||
url = f"http://127.0.0.1:{port}{path}"
|
||
data = body.encode() if body is not None else None
|
||
req = urllib.request.Request(url, data=data, method=method)
|
||
if data is not None:
|
||
req.add_header("Content-Type", "application/json")
|
||
if token is not None:
|
||
req.add_header("Authorization", f"Bearer {token}")
|
||
try:
|
||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||
raw = r.read().decode()
|
||
return r.status, raw
|
||
except urllib.error.HTTPError as e:
|
||
return e.code, e.read().decode()
|
||
|
||
|
||
def raw_request(port, payload, timeout=5):
|
||
"""发一段原始 HTTP 报文,读完整响应(按 Connection: close 读到 EOF)。
|
||
|
||
★ 不能用「recv 一次」判响应:TCP 是流,一次 recv 可能只拿到响应头。
|
||
这正是上一版自研 HTTP 偶发失败的根因,探针自己不能重犯。
|
||
"""
|
||
s = socket.create_connection(("127.0.0.1", port), timeout=timeout)
|
||
try:
|
||
s.settimeout(timeout)
|
||
s.sendall(payload)
|
||
buf = b""
|
||
while True:
|
||
try:
|
||
chunk = s.recv(4096)
|
||
except socket.timeout:
|
||
break
|
||
if not chunk:
|
||
break
|
||
buf += chunk
|
||
return buf.decode(errors="replace")
|
||
finally:
|
||
s.close()
|
||
|
||
|
||
def wait_ready(port, timeout=15.0):
|
||
deadline = time.time() + timeout
|
||
while time.time() < deadline:
|
||
try:
|
||
st, _ = request(port, "GET", "/health", timeout=2)
|
||
if st == 200:
|
||
return True
|
||
except Exception:
|
||
pass
|
||
time.sleep(0.1)
|
||
return False
|
||
|
||
|
||
def selftest_evidence_upload():
|
||
"""guest → hostd 的**证明材料回传**闭环自测(独立起一个 hostd)。
|
||
|
||
# 这条链解决的是「宿主要**独立**判定证明」的前提
|
||
|
||
guest 采到的证明包在 guest 里,宿主在宿主上。宿主拿不到**原始字节**,
|
||
就只能复述提交者的结论 —— 那等于自己给自己背书,判定权就交回给了
|
||
提交证明的那一方。
|
||
|
||
所以 guest 必须把**原始证明包字节**回传给宿主,宿主拿字节**重新判一遍**。
|
||
|
||
# 为什么用 `mpc tee attest-upload` 这个真实客户端,而不是 Python 手搓 HTTP
|
||
|
||
手搓 HTTP 只能证明「hostd 收得下某个 POST」,
|
||
证不了「**guest 里那个二进制**能把它送上来」——
|
||
而后者才是真机上会出问题的那一段(路径、头格式、超时)。
|
||
这里用的是**与 guest 里完全同一个**二进制(同一个 `mpc`)。
|
||
|
||
# 覆盖什么
|
||
|
||
1. 正向:建带绑定的实例 ⇒ 判定 pending(证据未到);
|
||
2. 正向:用真实客户端上传 ⇒ 判定变 verified(宿主**自己**判定出来的);
|
||
3. 正向:`chip_sig` 仍是 false 且 `chip_sig_failed` 也是 false
|
||
—— 没配验签器是「**没验**」,不是「验了没过」;
|
||
4. 负向:**错令牌** ⇒ 401(且带拒绝特征);
|
||
5. 负向:**不带凭据** ⇒ 401(回环信任模式也不例外 ——
|
||
证据表是判定输入,能往里写就是能改判定结论);
|
||
6. 负向:上传**别的会话**的证明包 ⇒ 判定 rejected(绑定对不上);
|
||
"""
|
||
port = free_port()
|
||
state_file = f"/tmp/mpc-hostd-upload-{port}.json"
|
||
# ★ 必须带 --guest-endpoint —— 否则 hostd **不建手牌目录**
|
||
# (没有手牌就没有令牌文件;guest 也拿不到令牌)。
|
||
# 这里给的地址就是自测里 hostd 自己的回环地址 ——
|
||
# 真机上它是 tap 网段里的宿主地址(如 10.93.0.1:3010)。
|
||
proc = subprocess.Popen(
|
||
[
|
||
BIN,
|
||
"hostd",
|
||
"--listen",
|
||
f"127.0.0.1:{port}",
|
||
"--state-file",
|
||
state_file,
|
||
"--guest-endpoint",
|
||
f"127.0.0.1:{port}",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
try:
|
||
if not wait_ready(port):
|
||
check("上传自测:hostd 起来且 /health 通", False, f"port={port}")
|
||
return
|
||
check("上传自测:hostd 起来且 /health 通", True)
|
||
|
||
# --- 建一个**带绑定**的实例 ---------------------------------------
|
||
sid = "11" * 32
|
||
pk = "22" * 32
|
||
meas = "33" * 32
|
||
st, raw = request(
|
||
port,
|
||
"POST",
|
||
"/sandboxes",
|
||
json.dumps(
|
||
{
|
||
"session_id": "upload-selftest",
|
||
"site": "alice",
|
||
"platform": "snp",
|
||
"template": "t-1",
|
||
"program": "/bin/sleep",
|
||
"args": ["300"],
|
||
"bind_session_id": sid,
|
||
"bind_session_pubkey": pk,
|
||
"bind_code_measurement": meas,
|
||
"tcb": 5,
|
||
"min_tcb": 3,
|
||
}
|
||
),
|
||
)
|
||
check("上传自测:创建带绑定的实例返回 201", st == 201, f"got {st}: {raw[:200]}")
|
||
if st != 201:
|
||
return
|
||
sb = json.loads(raw).get("sandboxID", "")
|
||
|
||
# --- 证据未到 ⇒ pending -------------------------------------------
|
||
st, raw = request(port, "GET", f"/sandboxes/{sb}/attestation")
|
||
check("上传自测:GET /attestation 返回 200", st == 200, f"got {st}")
|
||
att = json.loads(raw) if st == 200 else {}
|
||
check(
|
||
"上传自测:证据未到 ⇒ status=pending",
|
||
att.get("status") == "pending",
|
||
f"got {att.get('status')!r}",
|
||
)
|
||
check(
|
||
"上传自测:证据未到 ⇒ user_data_ok=false",
|
||
att.get("user_data_ok") is False,
|
||
f"got {att.get('user_data_ok')!r}",
|
||
)
|
||
|
||
# --- 造一份**绑定一致**的证明包 ----------------------------------
|
||
bundle_path = f"/tmp/mpc-upload-selftest-{port}.bundle"
|
||
p = subprocess.run(
|
||
[
|
||
BIN,
|
||
"tee",
|
||
"attest-mkbundle",
|
||
"--platform",
|
||
"snp",
|
||
"--session-id",
|
||
sid,
|
||
"--site",
|
||
"alice",
|
||
"--pubkey",
|
||
pk,
|
||
"--measurement",
|
||
meas,
|
||
"--tcb",
|
||
"5",
|
||
"--out",
|
||
bundle_path,
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
check(
|
||
"上传自测:attest-mkbundle 造出证明包",
|
||
p.returncode == 0 and os.path.exists(bundle_path),
|
||
p.stdout.decode(errors="replace")[:200],
|
||
)
|
||
if not os.path.exists(bundle_path):
|
||
return
|
||
|
||
# --- 用**真实客户端**上传 -----------------------------------------
|
||
#
|
||
# ★ 令牌从哪儿来?hostd 是自己生成的,**不**通过响应暴露 ——
|
||
# 暴露它等于把「往这个实例写证据」的权限给了任何能读响应的人。
|
||
# 所以这条链路里令牌只能来自 hostd **本地**的手牌目录。
|
||
#
|
||
# 自测里用手牌目录里的令牌(与 guest 走 9p 拿到的是同一个)。
|
||
hand_root = os.path.join(
|
||
os.environ.get("TMPDIR", "/tmp"), "mpc-hostd-hands"
|
||
)
|
||
token = ""
|
||
tok_path = os.path.join(hand_root, sb, "upload_token")
|
||
if os.path.exists(tok_path):
|
||
with open(tok_path, "r", encoding="utf-8") as f:
|
||
token = f.read().strip()
|
||
check(
|
||
"上传自测:hostd 为该实例生成了上传令牌(手牌目录)",
|
||
bool(token),
|
||
f"没找到 {tok_path}",
|
||
)
|
||
if not token:
|
||
return
|
||
|
||
# ★ 上传路径**由宿主下发**(手牌里的 upload_path)。
|
||
# 这正是 guest 走 9p 拿到的那一份 —— 自测用同一个文件,
|
||
# 保证测的路径与真机上 guest 用的路径**是同一个来源**。
|
||
path_file = os.path.join(hand_root, sb, "upload_path")
|
||
upload_path = ""
|
||
if os.path.exists(path_file):
|
||
with open(path_file, "r", encoding="utf-8") as f:
|
||
upload_path = f.read().strip()
|
||
check(
|
||
"上传自测:hostd 在手牌里下发了上传路径(路由归宿主管)",
|
||
bool(upload_path),
|
||
f"没找到 {path_file}",
|
||
)
|
||
check(
|
||
"上传自测:下发的上传路径形状正确",
|
||
upload_path == f"/sandboxes/{sb}/attestation",
|
||
f"got {upload_path!r}",
|
||
)
|
||
if not upload_path:
|
||
return
|
||
|
||
up = subprocess.run(
|
||
[
|
||
BIN,
|
||
"tee",
|
||
"attest-upload",
|
||
"--host",
|
||
f"127.0.0.1:{port}",
|
||
"--sandbox",
|
||
sb,
|
||
"--token-file",
|
||
tok_path,
|
||
"--upload-path-file",
|
||
path_file,
|
||
"--bundle",
|
||
bundle_path,
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
check(
|
||
"上传自测:真实客户端 attest-upload 成功",
|
||
up.returncode == 0,
|
||
up.stdout.decode(errors="replace")[:300],
|
||
)
|
||
|
||
# --- 宿主**自己**判定 ⇒ verified ----------------------------------
|
||
st, raw = request(port, "GET", f"/sandboxes/{sb}/attestation")
|
||
att = json.loads(raw) if st == 200 else {}
|
||
check(
|
||
"上传自测:上传后判定变 verified(宿主独立判定)",
|
||
att.get("status") == "verified",
|
||
f"got {att.get('status')!r}: {raw[:200]}",
|
||
)
|
||
check(
|
||
"上传自测:user_data_ok=true(绑定对得上)",
|
||
att.get("user_data_ok") is True,
|
||
f"got {att.get('user_data_ok')!r}",
|
||
)
|
||
check(
|
||
"上传自测:meas_ok=true",
|
||
att.get("meas_ok") is True,
|
||
f"got {att.get('meas_ok')!r}",
|
||
)
|
||
check(
|
||
"上传自测:tcb_ok=true(5 >= 门槛 3)",
|
||
att.get("tcb_ok") is True,
|
||
f"got {att.get('tcb_ok')!r}",
|
||
)
|
||
# ★ 没配验签器 ⇒ 「**没验**」而不是「验了没过」。
|
||
# 这两者的处置完全不同:前者只是不公布端点,后者是硬拒绝。
|
||
check(
|
||
"上传自测:chip_sig=false(没配验签器 ⇒ 没验)",
|
||
att.get("chip_sig") is False,
|
||
f"got {att.get('chip_sig')!r}",
|
||
)
|
||
check(
|
||
"上传自测:chip_sig_failed=false(「没验」不是「验签失败」)",
|
||
att.get("chip_sig_failed") is False,
|
||
f"got {att.get('chip_sig_failed')!r}",
|
||
)
|
||
|
||
# --- 负向 1:错令牌 ⇒ 401 -----------------------------------------
|
||
bad_token_path = f"/tmp/mpc-upload-badtoken-{port}.txt"
|
||
with open(bad_token_path, "w", encoding="utf-8") as f:
|
||
f.write("00" * 32)
|
||
up = subprocess.run(
|
||
[
|
||
BIN,
|
||
"tee",
|
||
"attest-upload",
|
||
"--host",
|
||
f"127.0.0.1:{port}",
|
||
"--sandbox",
|
||
sb,
|
||
"--token-file",
|
||
bad_token_path,
|
||
"--upload-path-file",
|
||
path_file,
|
||
"--bundle",
|
||
bundle_path,
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
out = up.stdout.decode(errors="replace")
|
||
check(
|
||
"上传自测:错令牌 ⇒ 失败",
|
||
up.returncode != 0,
|
||
f"rc={up.returncode}: {out[:200]}",
|
||
)
|
||
check(
|
||
"上传自测:错令牌 ⇒ 报出 401(期望拒绝特征)",
|
||
"401" in out or "凭据" in out,
|
||
out[:300],
|
||
)
|
||
|
||
# --- 负向 2:不带凭据 ⇒ 401 ---------------------------------------
|
||
#
|
||
# ★ 回环信任模式下「不带凭据即放行」是**默认**行为,
|
||
# 但上传端点**不能**沿用:证据表是判定输入,
|
||
# 能往里写就是能改判定结论。
|
||
st, raw = request(
|
||
port,
|
||
"POST",
|
||
f"/sandboxes/{sb}/attestation",
|
||
"",
|
||
)
|
||
check(
|
||
"上传自测:不带凭据 ⇒ 401(回环模式也不例外)",
|
||
st == 401,
|
||
f"got {st}: {raw[:200]}",
|
||
)
|
||
|
||
# --- 负向 3:别的会话的证明包 ⇒ rejected --------------------------
|
||
other_sid = "44" * 32
|
||
other_bundle = f"/tmp/mpc-upload-other-{port}.bundle"
|
||
p = subprocess.run(
|
||
[
|
||
BIN,
|
||
"tee",
|
||
"attest-mkbundle",
|
||
"--platform",
|
||
"snp",
|
||
"--session-id",
|
||
other_sid,
|
||
"--site",
|
||
"alice",
|
||
"--pubkey",
|
||
pk,
|
||
"--measurement",
|
||
meas,
|
||
"--tcb",
|
||
"5",
|
||
"--out",
|
||
other_bundle,
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
if p.returncode == 0:
|
||
up = subprocess.run(
|
||
[
|
||
BIN,
|
||
"tee",
|
||
"attest-upload",
|
||
"--host",
|
||
f"127.0.0.1:{port}",
|
||
"--sandbox",
|
||
sb,
|
||
"--token-file",
|
||
tok_path,
|
||
"--upload-path-file",
|
||
path_file,
|
||
"--bundle",
|
||
other_bundle,
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
check(
|
||
"上传自测:别的会话的证明包上传成功(存储层)",
|
||
up.returncode == 0,
|
||
up.stdout.decode(errors="replace")[:200],
|
||
)
|
||
st, raw = request(port, "GET", f"/sandboxes/{sb}/attestation")
|
||
att = json.loads(raw) if st == 200 else {}
|
||
check(
|
||
"上传自测:绑定对不上的证明包 ⇒ status=rejected",
|
||
att.get("status") == "rejected",
|
||
f"got {att.get('status')!r}: {raw[:200]}",
|
||
)
|
||
check(
|
||
"上传自测:绑定对不上 ⇒ user_data_ok=false",
|
||
att.get("user_data_ok") is False,
|
||
f"got {att.get('user_data_ok')!r}",
|
||
)
|
||
|
||
# --- 清理 ---------------------------------------------------------
|
||
for f in (bundle_path, other_bundle, bad_token_path):
|
||
try:
|
||
os.remove(f)
|
||
except OSError:
|
||
pass
|
||
finally:
|
||
proc.terminate()
|
||
try:
|
||
proc.wait(timeout=5)
|
||
except Exception:
|
||
proc.kill()
|
||
|
||
|
||
def selftest_auth():
|
||
"""凭据模式的端到端自测(独立起一个带共享密钥的 hostd)。
|
||
|
||
# 为什么必须单独起一个进程
|
||
|
||
上面主流程的 hostd 跑在**回环信任模式**(没配密钥)。
|
||
凭据模式要测的是「配了密钥之后,未带/带错凭据的请求一律被拒」,
|
||
这与回环模式的判据相反 —— 同一个进程里测不出来。
|
||
|
||
# 覆盖什么
|
||
|
||
1. 配了密钥 → 不带凭据 **401**(业务端点也 401,不只是 /health);
|
||
2. 带错凭据 **401**;
|
||
3. 带对凭据 **200**;
|
||
4. 客户端(`mpc tee ps --token`)与服务端约定一致 ——
|
||
★ 这条是**跨进程**契约:只测 Python 的 urllib 测不到
|
||
`HostClient` 拼的头对不对。
|
||
|
||
# 失败就非零退出。
|
||
"""
|
||
secret = "0123456789abcdef0123456789abcdef"
|
||
port = free_port()
|
||
state_file = f"/tmp/mpc-hostd-selftest-auth-{port}.json"
|
||
env = dict(os.environ)
|
||
env["MPC_HOSTD_SHARED_SECRET"] = secret
|
||
proc = subprocess.Popen(
|
||
[BIN, "hostd", "--listen", f"127.0.0.1:{port}", "--state-file", state_file],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
env=env,
|
||
)
|
||
try:
|
||
# 等待起来:**用带凭据的请求探活** —— 凭据模式下不带凭据是 401。
|
||
deadline = time.time() + 15
|
||
ready = False
|
||
while time.time() < deadline:
|
||
try:
|
||
st, _ = request(port, "GET", "/health", timeout=2, token=secret)
|
||
if st == 200:
|
||
ready = True
|
||
break
|
||
except Exception:
|
||
pass
|
||
time.sleep(0.1)
|
||
if not ready:
|
||
print(f"{FAIL} 凭据模式 hostd 起来后带凭据 /health 不通")
|
||
return False
|
||
|
||
# --- 负向:不带凭据 → 401 ---------------------------------------
|
||
st, raw = request(port, "GET", "/health")
|
||
check(
|
||
"凭据模式下不带凭据 → 401",
|
||
st == 401,
|
||
f"got {st}: {raw[:200]}",
|
||
)
|
||
check(
|
||
"401 响应说「缺少或无效的凭据」",
|
||
"缺少或无效的凭据" in raw,
|
||
raw[:200],
|
||
)
|
||
# ★ 不许区分「没带」与「带错了」—— 那等于给攻击者反馈。
|
||
st_missing, raw_missing = request(port, "GET", "/health")
|
||
st_wrong, raw_wrong = request(
|
||
port, "GET", "/health", token="ffffffffffffffffffffffffffffffff"
|
||
)
|
||
check(
|
||
"凭据模式下带错凭据 → 401",
|
||
st_wrong == 401,
|
||
f"got {st_wrong}: {raw_wrong[:200]}",
|
||
)
|
||
check(
|
||
"★ 401 响应**不区分**「没带」与「带错了」(防探测)",
|
||
raw_missing == raw_wrong,
|
||
f"missing={raw_missing[:200]!r} wrong={raw_wrong[:200]!r}",
|
||
)
|
||
|
||
# ★ 业务端点也必须受保护 —— 只护 /health 是假安全。
|
||
st, raw = request(
|
||
port,
|
||
"POST",
|
||
"/sandboxes",
|
||
json.dumps(
|
||
{
|
||
"session_id": "s",
|
||
"site": "a",
|
||
"platform": "snp",
|
||
"program": "/bin/sleep",
|
||
"args": ["300"],
|
||
}
|
||
),
|
||
)
|
||
check(
|
||
"★ 凭据模式下业务端点 POST /sandboxes 无凭据 → 401",
|
||
st == 401,
|
||
f"got {st}: {raw[:200]}",
|
||
)
|
||
check("无凭据时不该创建实例", st != 201, f"got {st}")
|
||
|
||
# --- 正向:带对凭据 → 200 ----------------------------------------
|
||
st, raw = request(port, "GET", "/health", token=secret)
|
||
check("凭据模式下带对凭据 → 200", st == 200, f"got {st}: {raw[:200]}")
|
||
|
||
# --- ★ 跨进程契约:mpc tee ps --token 必须能连上 ------------------
|
||
# 这条测的是 `HostClient` 拼的头与服务端约定一致 ——
|
||
# Python 的 urllib 测不到那一层。
|
||
cli = subprocess.run(
|
||
[
|
||
BIN,
|
||
"tee",
|
||
"ps",
|
||
"--host",
|
||
f"127.0.0.1:{port}",
|
||
"--token",
|
||
secret,
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
timeout=30,
|
||
)
|
||
out = cli.stdout.decode(errors="replace")
|
||
check(
|
||
"★ mpc tee ps --token 能连上凭据模式的 hostd",
|
||
cli.returncode == 0,
|
||
f"rc={cli.returncode}: {out[:300]}",
|
||
)
|
||
|
||
# ★ 负向:mpc tee ps 不带 token → 必须失败(非零退出)。
|
||
cli2 = subprocess.run(
|
||
[BIN, "tee", "ps", "--host", f"127.0.0.1:{port}"],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
timeout=30,
|
||
)
|
||
out2 = cli2.stdout.decode(errors="replace")
|
||
check(
|
||
"★ mpc tee ps 不带 token → 失败(非零退出)",
|
||
cli2.returncode != 0,
|
||
f"rc={cli2.returncode}: {out2[:300]}",
|
||
)
|
||
check(
|
||
"★ 失败原因带出 401(不是碰巧崩了)",
|
||
"401" in out2,
|
||
out2[:300],
|
||
)
|
||
|
||
# ★ 负向:mpc tee ps 带错 token → 必须失败。
|
||
cli3 = subprocess.run(
|
||
[
|
||
BIN,
|
||
"tee",
|
||
"ps",
|
||
"--host",
|
||
f"127.0.0.1:{port}",
|
||
"--token",
|
||
"ffffffffffffffffffffffffffffffff",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
timeout=30,
|
||
)
|
||
out3 = cli3.stdout.decode(errors="replace")
|
||
check(
|
||
"★ mpc tee ps 带错 token → 失败(非零退出)",
|
||
cli3.returncode != 0,
|
||
f"rc={cli3.returncode}: {out3[:300]}",
|
||
)
|
||
check("★ 失败原因带出 401", "401" in out3, out3[:300])
|
||
return True
|
||
finally:
|
||
proc.send_signal(signal.SIGTERM)
|
||
try:
|
||
proc.wait(timeout=5)
|
||
except Exception:
|
||
proc.kill()
|
||
try:
|
||
os.unlink(state_file)
|
||
except OSError:
|
||
pass
|
||
|
||
|
||
def selftest_remote_guard():
|
||
"""非回环地址的启动期守卫:没有共享密钥就必须拒绝起服务。
|
||
|
||
# 为什么这条不能省
|
||
|
||
「--allow-remote 就把无鉴权控制面挂到网上」是这个守卫要防的事。
|
||
只在文档里写「请配密钥」是不够的 —— 必须**启动期就拒**,
|
||
否则一次误操作就是无鉴权暴露。
|
||
|
||
# 判据:进程必须以**非零**退出,且输出里说明缺共享密钥。
|
||
"""
|
||
# --- 负向 A:非回环 + --allow-remote,但没有共享密钥 → 必须拒 -----
|
||
port = free_port()
|
||
p = subprocess.run(
|
||
[
|
||
BIN,
|
||
"hostd",
|
||
"--listen",
|
||
f"0.0.0.0:{port}",
|
||
"--allow-remote",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
timeout=30,
|
||
)
|
||
out = p.stdout.decode(errors="replace")
|
||
check(
|
||
"★ 非回环 + 无共享密钥 → 拒绝启动",
|
||
p.returncode != 0,
|
||
f"rc={p.returncode}: {out[:300]}",
|
||
)
|
||
check(
|
||
"★ 拒绝原因点名「共享密钥」",
|
||
"共享密钥" in out,
|
||
out[:300],
|
||
)
|
||
|
||
# --- 负向 B:非回环 + 有共享密钥但没 --allow-remote → 必须拒 ------
|
||
p2 = subprocess.run(
|
||
[
|
||
BIN,
|
||
"hostd",
|
||
"--listen",
|
||
f"0.0.0.0:{free_port()}",
|
||
"--shared-secret",
|
||
"0123456789abcdef0123456789abcdef",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
timeout=30,
|
||
)
|
||
out2 = p2.stdout.decode(errors="replace")
|
||
check(
|
||
"★ 非回环 + 无 --allow-remote → 拒绝启动",
|
||
p2.returncode != 0,
|
||
f"rc={p2.returncode}: {out2[:300]}",
|
||
)
|
||
check(
|
||
"★ 拒绝原因点名 --allow-remote",
|
||
"--allow-remote" in out2,
|
||
out2[:300],
|
||
)
|
||
|
||
# --- 负向 C:共享密钥太短 → 必须拒 --------------------------------
|
||
p3 = subprocess.run(
|
||
[BIN, "hostd", "--shared-secret", "short"],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
timeout=30,
|
||
)
|
||
out3 = p3.stdout.decode(errors="replace")
|
||
check(
|
||
"★ 共享密钥太短 → 拒绝启动",
|
||
p3.returncode != 0,
|
||
f"rc={p3.returncode}: {out3[:300]}",
|
||
)
|
||
|
||
|
||
def selftest_self_report():
|
||
"""节点自报(Cubelet 自描述)端到端自测。
|
||
|
||
# 为什么必须单独起进程
|
||
|
||
自报要配 `--control-plane`(控制面地址)与 `--node-platform`(本宿主能力),
|
||
这两个在主流程那个 hostd 里都没配 —— 同一个进程测不出来。
|
||
|
||
# 判什么
|
||
|
||
1. 假控制面**真**收到 `POST /registry`,且 body 是自报线格式;
|
||
2. 自报里的能力位与探针口径一致(路径填了但文件不在 ⇒ 该位为 false);
|
||
3. 度量注册阶段全零(pin 留给会话级准入);
|
||
4. 负向:没配 `--node-platform` ⇒ **一个请求都不发**(假就绪信号);
|
||
5. 负向:配了 `--control-plane` 却没配能力 ⇒ **拒绝启动**(不是静默空转)。
|
||
"""
|
||
cp_port = free_port()
|
||
hd_port = free_port()
|
||
capture = f"/tmp/mpc-hostd-selfreport-{cp_port}.txt"
|
||
open(capture, "w").close()
|
||
|
||
# 假控制面:收 POST /registry,把路径 + body 落盘。
|
||
srv = socket.socket()
|
||
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||
srv.bind(("127.0.0.1", cp_port))
|
||
srv.listen(8)
|
||
srv.settimeout(1.0)
|
||
captured = []
|
||
|
||
def serve():
|
||
"""后台收自报(最多 6 秒)。"""
|
||
end = time.time() + 6
|
||
while time.time() < end:
|
||
try:
|
||
conn, _ = srv.accept()
|
||
except socket.timeout:
|
||
continue
|
||
except OSError:
|
||
break
|
||
try:
|
||
conn.settimeout(3)
|
||
buf = b""
|
||
while b"\r\n\r\n" not in buf:
|
||
chunk = conn.recv(4096)
|
||
if not chunk:
|
||
break
|
||
buf += chunk
|
||
head, _, rest = buf.partition(b"\r\n\r\n")
|
||
clen = 0
|
||
for line in head.decode(errors="replace").split("\r\n"):
|
||
if line.lower().startswith("content-length:"):
|
||
try:
|
||
clen = int(line.split(":", 1)[1].strip())
|
||
except ValueError:
|
||
clen = 0
|
||
body = rest
|
||
while len(body) < clen:
|
||
chunk = conn.recv(4096)
|
||
if not chunk:
|
||
break
|
||
body += chunk
|
||
path = head.decode(errors="replace").split("\r\n")[0].split(" ")[1]
|
||
captured.append((path, body.decode(errors="replace")))
|
||
with open(capture, "a") as f:
|
||
f.write(path + "\t" + body.decode(errors="replace") + "\n")
|
||
conn.sendall(b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n")
|
||
except Exception:
|
||
pass
|
||
finally:
|
||
conn.close()
|
||
|
||
import threading
|
||
|
||
th = threading.Thread(target=serve, daemon=True)
|
||
th.start()
|
||
|
||
# --- 正向:配了能力 + 控制面 ⇒ 周期推送自报 --------------------------
|
||
proc = subprocess.Popen(
|
||
[
|
||
BIN,
|
||
"hostd",
|
||
"--listen",
|
||
f"127.0.0.1:{hd_port}",
|
||
"--node-platform",
|
||
"snp",
|
||
"--node-site",
|
||
"alice",
|
||
"--node-host",
|
||
"192.168.1.7",
|
||
"--node-tcb",
|
||
"5",
|
||
"--node-qemu",
|
||
"/definitely/not/here/qemu",
|
||
"--node-kernel",
|
||
"/bin/ls",
|
||
"--node-initrd",
|
||
"/bin/ls",
|
||
"--node-capacity",
|
||
"3",
|
||
"--control-plane",
|
||
f"127.0.0.1:{cp_port}",
|
||
"--report-interval",
|
||
"1",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
try:
|
||
# 等第一次自报到达(推送线程先 sleep 一个周期再发,所以要多等一会)。
|
||
deadline = time.time() + 8
|
||
while time.time() < deadline and not captured:
|
||
time.sleep(0.2)
|
||
|
||
check("★ 自报真的推到了控制面(POST /registry)", len(captured) > 0)
|
||
if not captured:
|
||
return
|
||
path, body = captured[0]
|
||
check("自报路径是 /registry", path == "/registry", path)
|
||
try:
|
||
obj = json.loads(body)
|
||
except Exception:
|
||
obj = {}
|
||
check("自报 body 是 JSON 且带 descriptor", "descriptor" in obj, body[:200])
|
||
check("自报 body 带 capacity", obj.get("capacity") == 3, body[:200])
|
||
|
||
desc = obj.get("descriptor", "")
|
||
# ★ 探针口径:路径填了但文件不在 ⇒ 该能力位必须是 false(不猜)。
|
||
check(
|
||
"★ 探测到的 qemu 文件不存在 ⇒ qemu_present=false(不猜能跑)",
|
||
"qemu_present=false" in desc,
|
||
desc[:300],
|
||
)
|
||
check(
|
||
"★ 探测到的内核/initrd 存在 ⇒ image_present=true",
|
||
"image_present=true" in desc,
|
||
desc[:300],
|
||
)
|
||
check(
|
||
"★ 平台不匹配 ⇒ device_present=false(本机没有 /dev/sev)",
|
||
"device_present=false" in desc,
|
||
desc[:300],
|
||
)
|
||
check("自报带对 site/host/tcb", "site=alice" in desc and "host=192.168.1.7" in desc and "tcb=5" in desc, desc[:300])
|
||
check(
|
||
"★ 注册阶段度量全零(pin 留给会话级准入)",
|
||
"measurement=" + "0" * 64 in desc,
|
||
desc[:300],
|
||
)
|
||
finally:
|
||
proc.send_signal(signal.SIGTERM)
|
||
try:
|
||
proc.wait(timeout=5)
|
||
except Exception:
|
||
proc.kill()
|
||
|
||
# --- 负向:没配能力 ⇒ 一个请求都不发(假就绪信号)--------------------
|
||
cp_port2 = free_port()
|
||
hd_port2 = free_port()
|
||
srv2 = socket.socket()
|
||
srv2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||
srv2.bind(("127.0.0.1", cp_port2))
|
||
srv2.listen(8)
|
||
srv2.settimeout(0.5)
|
||
hits2 = []
|
||
|
||
def serve2():
|
||
end = time.time() + 4
|
||
while time.time() < end:
|
||
try:
|
||
conn, _ = srv2.accept()
|
||
except socket.timeout:
|
||
continue
|
||
except OSError:
|
||
break
|
||
hits2.append(1)
|
||
conn.close()
|
||
|
||
th2 = threading.Thread(target=serve2, daemon=True)
|
||
th2.start()
|
||
|
||
proc2 = subprocess.Popen(
|
||
[
|
||
BIN,
|
||
"hostd",
|
||
"--listen",
|
||
f"127.0.0.1:{hd_port2}",
|
||
"--control-plane",
|
||
f"127.0.0.1:{cp_port2}",
|
||
"--report-interval",
|
||
"1",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
try:
|
||
time.sleep(3)
|
||
check(
|
||
"★ 没配 --node-platform 时 hostd **拒绝启动**(不是静默空转)",
|
||
proc2.poll() is not None and proc2.returncode != 0,
|
||
f"rc={proc2.returncode}",
|
||
)
|
||
if proc2.poll() is None:
|
||
proc2.terminate()
|
||
out2 = ""
|
||
try:
|
||
out2 = proc2.stdout.read().decode(errors="replace")
|
||
except Exception:
|
||
pass
|
||
check(
|
||
"★ 拒绝时说明缺 --node-platform",
|
||
"--node-platform" in out2,
|
||
out2[:300],
|
||
)
|
||
check(
|
||
"★ 没配能力时**一个请求都不发**(不许有推送)",
|
||
len(hits2) == 0,
|
||
f"hits={len(hits2)}",
|
||
)
|
||
finally:
|
||
if proc2.poll() is None:
|
||
proc2.terminate()
|
||
try:
|
||
proc2.wait(timeout=5)
|
||
except Exception:
|
||
proc2.kill()
|
||
|
||
try:
|
||
srv.close()
|
||
srv2.close()
|
||
os.unlink(capture)
|
||
except OSError:
|
||
pass
|
||
|
||
|
||
def selftest_control_plane():
|
||
"""集群控制面(CubeMaster 等价物)端到端自测。
|
||
|
||
# 判什么
|
||
|
||
Phase 3b 的闭环:**自报 → 注册表 → 期望态 → 选址 → 下发 → 证明判定**。
|
||
|
||
1. 真 socket 上 `POST /registry` 收自报 ⇒ `GET /nodes` 能查到;
|
||
2. `POST /desired` 设期望态 ⇒ 对账**真的选中**了那台节点并下发了 create;
|
||
3. 证明不齐 ⇒ 会话端点**不**出现在 `GET /sessions`(fail-closed);
|
||
4. 负向:坏自报 ⇒ 400 且注册表不被污染;
|
||
5. 负向:期望态缺字段 ⇒ 400(不猜默认值);
|
||
6. 负向:未知路径 ⇒ 404;错方法 ⇒ 405;
|
||
7. 负向:鉴权缺 token ⇒ 401。
|
||
|
||
# 为什么用**假节点**而不是真机
|
||
|
||
真机上自报里的能力位取决于那台机器有没有 /dev/sev、真 qemu、真固件 ——
|
||
那是**真机联调**的事,门禁要在任何开发机上都能跑。
|
||
这里造一份「能力位齐、度量已 pin」的节点描述,走的是**与真机完全相同**
|
||
的那条线(同一个 `TeeNode::parse` / 同一个 `admit`)。
|
||
|
||
# 判据纪律
|
||
|
||
每条负向判据都带**期望的拒绝特征**(400 / 404 / 405 / 401),
|
||
只判「失败」不判失败原因,就分不清「按设计拒绝」和「碰巧崩了」。
|
||
"""
|
||
cp_port = free_port()
|
||
|
||
proc = subprocess.Popen(
|
||
[
|
||
BIN,
|
||
"control-plane",
|
||
"serve",
|
||
"--listen",
|
||
f"127.0.0.1:{cp_port}",
|
||
"--hostd-port",
|
||
"3010",
|
||
"--node-ttl",
|
||
"30",
|
||
"--reconcile-interval",
|
||
"1",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
try:
|
||
if not wait_ready(cp_port, timeout=15.0):
|
||
out = ""
|
||
try:
|
||
proc.kill()
|
||
out = proc.stdout.read().decode(errors="replace")
|
||
except Exception:
|
||
pass
|
||
check("★ 控制面进程起来了并监听", False, out[:300])
|
||
return
|
||
check("★ 控制面进程起来了并监听", True)
|
||
|
||
# --- 1) 自报 → 注册表 -------------------------------------------
|
||
desc = "\n".join(
|
||
[
|
||
"site=alice",
|
||
"host=192.168.1.7",
|
||
"kind=sev-snp",
|
||
"tcb=5",
|
||
"device_present=true",
|
||
"qemu_present=true",
|
||
"firmware_present=true",
|
||
"image_present=true",
|
||
"kbs_reachable=false",
|
||
"measurement=" + "09" * 32,
|
||
"kbc=offline",
|
||
"qemu=/usr/bin/qemu-system-x86_64",
|
||
"firmware=/srv/tee/AMDSEV.fd",
|
||
"kernel=/srv/tee/vmlinuz",
|
||
"initrd=/srv/tee/initrd.img",
|
||
"",
|
||
]
|
||
)
|
||
body = json.dumps({"descriptor": desc, "capacity": 2})
|
||
st, txt = request(cp_port, "POST", "/registry", body)
|
||
check("★ 自报被收下(204)", st == 204, f"st={st} {txt[:200]}")
|
||
|
||
st, txt = request(cp_port, "GET", "/nodes")
|
||
check("注册表可查(200)", st == 200, f"st={st}")
|
||
try:
|
||
nodes = json.loads(txt).get("nodes", [])
|
||
except Exception:
|
||
nodes = []
|
||
check("★ 注册表里真有那一台", len(nodes) == 1, f"nodes={len(nodes)}")
|
||
if nodes:
|
||
n = nodes[0]
|
||
check("注册表记录带对容量", n.get("capacity") == 2, str(n.get("capacity")))
|
||
check("注册表记录带对 host", n.get("host") == "192.168.1.7", str(n.get("host")))
|
||
check(
|
||
"★ 注册表记录带出度量(部署方声明的事实)",
|
||
n.get("measurement") == "09" * 32,
|
||
str(n.get("measurement")),
|
||
)
|
||
|
||
# --- 2) 期望态 → 对账真的选址并下发 ----------------------------
|
||
desired = {
|
||
"sessions": [
|
||
{
|
||
"id": "s1",
|
||
"platform": "sev-snp",
|
||
"min_tcb": 3,
|
||
"session_id": "01" * 32,
|
||
"session_pubkey": "02" * 32,
|
||
"code_measurement": "09" * 32,
|
||
"site": "alice",
|
||
}
|
||
]
|
||
}
|
||
st, txt = request(cp_port, "POST", "/desired", json.dumps(desired))
|
||
check("★ 期望态被收下(204)", st == 204, f"st={st} {txt[:200]}")
|
||
|
||
# 等一轮对账(reconcile-interval=1s)。
|
||
report_txt = ""
|
||
deadline = time.time() + 12
|
||
while time.time() < deadline:
|
||
st, report_txt = request(cp_port, "GET", "/report")
|
||
if st == 200:
|
||
try:
|
||
rep = json.loads(report_txt)
|
||
except Exception:
|
||
rep = {}
|
||
if rep.get("failed"):
|
||
break
|
||
time.sleep(0.5)
|
||
|
||
try:
|
||
rep = json.loads(report_txt)
|
||
except Exception:
|
||
rep = {}
|
||
failed = rep.get("failed", [])
|
||
healthy = rep.get("healthy", [])
|
||
# 那台 192.168.1.7 上没有真 hostd ⇒ create 必然失败。
|
||
# ★ 判的是「**选中并下发到了那台**」,不是「成功」——
|
||
# 成功与否取决于真机,这里判的是控制面的选址与下发链路。
|
||
picked = any(
|
||
"192.168.1.7" in (f.get("reason") or "") for f in failed
|
||
)
|
||
check(
|
||
"★ 对账真的选中并下发到了那台节点(失败原因里带它的地址)",
|
||
picked,
|
||
json.dumps(failed, ensure_ascii=False)[:300],
|
||
)
|
||
check(
|
||
"★ 证明没通过 ⇒ 会话不算健康(fail-closed)",
|
||
not healthy,
|
||
json.dumps(healthy, ensure_ascii=False)[:200],
|
||
)
|
||
|
||
# --- 3) 证明不齐 ⇒ 端点不外泄 -----------------------------------
|
||
st, txt = request(cp_port, "GET", "/sessions")
|
||
check("会话查询可查(200)", st == 200, f"st={st}")
|
||
try:
|
||
sessions = json.loads(txt).get("sessions", [])
|
||
except Exception:
|
||
sessions = []
|
||
check(
|
||
"★ 证明没过 ⇒ 会话端点绝不外泄",
|
||
len(sessions) == 0,
|
||
json.dumps(sessions, ensure_ascii=False)[:200],
|
||
)
|
||
|
||
# --- 4) 负向:坏自报 ⇒ 400 + 注册表不被污染 ----------------------
|
||
#
|
||
# ★ 注意:判据是「坏自报**不新增**节点」,不是「节点数不变」。
|
||
# 因为刚才那轮对账已经因为「那台机器的 hostd 不可达」把节点
|
||
# **剔除了**(fail-closed 加速收敛,见 reconciler 的设计)——
|
||
# 节点数可能变少,但绝不能因为一份坏自报而**变多**。
|
||
st, txt = request(cp_port, "GET", "/nodes")
|
||
try:
|
||
nodes2_before = json.loads(txt).get("nodes", [])
|
||
except Exception:
|
||
nodes2_before = []
|
||
before_bad = len(nodes2_before)
|
||
st, txt = request(cp_port, "POST", "/registry", "{not json")
|
||
check("★ 坏自报被拒(400)", st == 400, f"st={st} {txt[:200]}")
|
||
check("★ 拒绝原因点名 JSON", "JSON" in txt, txt[:200])
|
||
st, txt = request(cp_port, "GET", "/nodes")
|
||
try:
|
||
nodes2 = json.loads(txt).get("nodes", [])
|
||
except Exception:
|
||
nodes2 = []
|
||
check(
|
||
"★ 坏自报绝没往注册表里**加**东西",
|
||
len(nodes2) <= before_bad,
|
||
f"before={before_bad} after={len(nodes2)}",
|
||
)
|
||
|
||
# --- 5) 负向:期望态缺字段 ⇒ 400 -------------------------------
|
||
st, txt = request(
|
||
cp_port, "POST", "/desired", json.dumps({"sessions": [{"id": "s1"}]})
|
||
)
|
||
check("★ 期望态缺字段被拒(400)", st == 400, f"st={st} {txt[:200]}")
|
||
check("★ 拒绝原因点名字段 platform", "platform" in txt, txt[:200])
|
||
|
||
# --- 6) 负向:路径 / 方法 --------------------------------------
|
||
st, _ = request(cp_port, "GET", "/nope")
|
||
check("★ 未知路径 404", st == 404, f"st={st}")
|
||
st, _ = request(cp_port, "GET", "/registry")
|
||
check("★ 错方法 405", st == 405, f"st={st}")
|
||
st, _ = request(cp_port, "GET", "/desired")
|
||
check("★ 期望态只收 POST(GET 405)", st == 405, f"st={st}")
|
||
finally:
|
||
if proc.poll() is None:
|
||
proc.terminate()
|
||
try:
|
||
proc.wait(timeout=5)
|
||
except Exception:
|
||
proc.kill()
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# 假 hostd:给「端点外泄」判据提供**正向对照**
|
||
#
|
||
# 为什么必须有它:selftest_control_plane() 里那台节点 (192.168.1.7:3010)
|
||
# 没有真 hostd ⇒ create 必失败 ⇒ placed 恒空 ⇒ GET /sessions 恒为空。
|
||
# 于是「证明没过 ⇒ 端点不外泄」是**恒真**的(空集通过),根本没检验到
|
||
# 「证明判定之后是否把端点写进 placed / 是否从 placed 收回」。
|
||
#
|
||
# 这里起一个真 socket 的假 hostd,让 create 真的成功、证明结论可控:
|
||
# 证明全通过 ⇒ 端点必须**出现**(正向,钉死「公布」这条路真的通);
|
||
# 证明不齐 ⇒ 端点必须**消失**(负向,此时 create 成功,placed 有机会
|
||
# 被写入 ⇒ 判据不再恒真)。
|
||
# 两者合起来才把「证明门控」真正判住:缺了正向,负向只是空集通过;
|
||
# 缺了负向,正向只说明「会公布」,不能证明「不该公布时不公布」。
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
class _FakeHostdState:
|
||
"""假 hostd 的可变状态(主线程改、服务线程读)。"""
|
||
|
||
def __init__(self):
|
||
self.attest_ok = True
|
||
self.create_calls = 0
|
||
self.attest_calls = 0
|
||
self.destroy_calls = 0
|
||
|
||
|
||
def _make_fake_hostd(state):
|
||
"""造一个只实现调度器会用的三个端点的真 socket 服务。
|
||
|
||
响应形状必须满足 `HostClient` 的 fail-closed 判定:
|
||
2xx + 准确的 Content-Length(204 例外,按规范无 body 也不写 CL)。
|
||
"""
|
||
|
||
class Handler(http.server.BaseHTTPRequestHandler):
|
||
protocol_version = "HTTP/1.1"
|
||
|
||
def log_message(self, *args):
|
||
# 别把每个请求的日志糊进判据输出。
|
||
pass
|
||
|
||
def _reply(self, code, body=b""):
|
||
self.send_response(code)
|
||
if code == 204:
|
||
# ★ 204 按规范无 body —— 写了 Content-Length 就破坏语义。
|
||
self.send_header("Connection", "close")
|
||
self.end_headers()
|
||
return
|
||
self.send_header("Content-Type", "application/json")
|
||
self.send_header("Content-Length", str(len(body)))
|
||
self.send_header("Connection", "close")
|
||
self.end_headers()
|
||
if body:
|
||
self.wfile.write(body)
|
||
|
||
def _drain(self):
|
||
try:
|
||
n = int(self.headers.get("Content-Length", "0") or "0")
|
||
except ValueError:
|
||
n = 0
|
||
if n:
|
||
self.rfile.read(n)
|
||
|
||
def do_POST(self):
|
||
self._drain()
|
||
if self.path == "/sandboxes":
|
||
state.create_calls += 1
|
||
self._reply(201, b'{"sandboxID":"fake-sb-1"}')
|
||
else:
|
||
self._reply(404, b'{"message":"fake: unknown path"}')
|
||
|
||
def do_GET(self):
|
||
self._drain()
|
||
if self.path.startswith("/sandboxes/") and self.path.endswith(
|
||
"/attestation"
|
||
):
|
||
state.attest_calls += 1
|
||
ok = state.attest_ok
|
||
# ★ 假 hostd 必须模拟「宿主配了验签器」的场景:
|
||
# 集群控制面的 `require_chip_sig` 默认 **true**
|
||
# (用户的主线就是「真 KBS 验签后的会话端点」)。
|
||
# 所以正向场景必须 `chip_sig=true` ——
|
||
# 否则端点被**正确地**不公布,正向对照就永远红,
|
||
# 而那不是实现的错,是判据与策略对不上。
|
||
#
|
||
# 负向场景 `chip_sig=false` + `chip_sig_failed=true`
|
||
# ⇒ 模拟「验签器明确拒绝」⇒ 硬拒绝 ⇒ 端点收回。
|
||
body = json.dumps(
|
||
{
|
||
"session_id": "s1",
|
||
"user_data_ok": ok,
|
||
"meas_ok": ok,
|
||
"tcb_ok": ok,
|
||
"chip_sig": ok,
|
||
"chip_sig_failed": not ok,
|
||
"chip_sig_note": (
|
||
"fake hostd:芯片签名已验证"
|
||
if ok
|
||
else "fake hostd:芯片验签失败: 构造的失败样例"
|
||
),
|
||
"verdict": "ok" if ok else "reject",
|
||
"status": "verified" if ok else "rejected",
|
||
}
|
||
).encode()
|
||
self._reply(200, body)
|
||
elif self.path == "/health":
|
||
self._reply(200, b'{"status":"ok"}')
|
||
else:
|
||
self._reply(404, b'{"message":"fake: unknown path"}')
|
||
|
||
def do_DELETE(self):
|
||
self._drain()
|
||
state.destroy_calls += 1
|
||
self._reply(204)
|
||
|
||
class Server(http.server.ThreadingHTTPServer):
|
||
daemon_threads = True
|
||
allow_reuse_address = True
|
||
|
||
return Server(("127.0.0.1", 0), Handler)
|
||
|
||
|
||
def selftest_control_plane_endpoint_gate():
|
||
"""证明门控的正/负向对照:端点出现与端点收回都必须真的发生。"""
|
||
state = _FakeHostdState()
|
||
httpd = _make_fake_hostd(state)
|
||
fake_port = httpd.server_address[1]
|
||
srv = threading.Thread(target=httpd.serve_forever, daemon=True)
|
||
srv.start()
|
||
|
||
cp_port = free_port()
|
||
proc = subprocess.Popen(
|
||
[
|
||
BIN,
|
||
"control-plane",
|
||
"serve",
|
||
"--listen",
|
||
f"127.0.0.1:{cp_port}",
|
||
"--hostd-port",
|
||
str(fake_port),
|
||
"--node-ttl",
|
||
"30",
|
||
"--reconcile-interval",
|
||
"1",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
|
||
# 与真机同形状的 descriptor,只是 host 指到本机假 hostd。
|
||
desc = "\n".join(
|
||
[
|
||
"site=alice",
|
||
"host=127.0.0.1",
|
||
"kind=sev-snp",
|
||
"tcb=5",
|
||
"device_present=true",
|
||
"qemu_present=true",
|
||
"firmware_present=true",
|
||
"image_present=true",
|
||
"kbs_reachable=false",
|
||
"measurement=" + "09" * 32,
|
||
"kbc=offline",
|
||
"qemu=/usr/bin/qemu-system-x86_64",
|
||
"firmware=/srv/tee/AMDSEV.fd",
|
||
"kernel=/srv/tee/vmlinuz",
|
||
"initrd=/srv/tee/initrd.img",
|
||
"",
|
||
]
|
||
)
|
||
report_body = json.dumps({"descriptor": desc, "capacity": 2})
|
||
desired = {
|
||
"sessions": [
|
||
{
|
||
"id": "s1",
|
||
"platform": "sev-snp",
|
||
"min_tcb": 3,
|
||
"session_id": "01" * 32,
|
||
"session_pubkey": "02" * 32,
|
||
"code_measurement": "09" * 32,
|
||
"site": "alice",
|
||
}
|
||
]
|
||
}
|
||
|
||
def heartbeat():
|
||
# 每轮重新自报(等价 Cubelet 心跳):避免 30s TTL 把节点剔除,
|
||
# 否则「端点消失」可能只是节点过期,不是证明门控生效。
|
||
return request(cp_port, "POST", "/registry", report_body)
|
||
|
||
def sessions_now():
|
||
st, txt = request(cp_port, "GET", "/sessions")
|
||
try:
|
||
return st, json.loads(txt).get("sessions", [])
|
||
except Exception:
|
||
return st, []
|
||
|
||
def report_now():
|
||
st, txt = request(cp_port, "GET", "/report")
|
||
try:
|
||
return st, json.loads(txt)
|
||
except Exception:
|
||
return st, {}
|
||
|
||
def wait_for(pred, min_attest=None, timeout=15.0):
|
||
deadline = time.time() + timeout
|
||
last_s, last_r = [], {}
|
||
while time.time() < deadline:
|
||
heartbeat()
|
||
_, last_s = sessions_now()
|
||
_, last_r = report_now()
|
||
attest_ready = min_attest is None or state.attest_calls >= min_attest
|
||
if attest_ready and pred(last_s):
|
||
return last_s, last_r, True
|
||
time.sleep(0.4)
|
||
return last_s, last_r, False
|
||
|
||
try:
|
||
if not wait_ready(cp_port, timeout=15.0):
|
||
check("★ 假 hostd 场景:控制面起来了并监听", False)
|
||
return
|
||
check("★ 假 hostd 场景:控制面起来了并监听", True)
|
||
|
||
st, _ = heartbeat()
|
||
check("★ 假 hostd 场景:节点自报被收下(204)", st == 204, f"st={st}")
|
||
st, _ = request(cp_port, "POST", "/desired", json.dumps(desired))
|
||
check("★ 假 hostd 场景:期望态被收下(204)", st == 204, f"st={st}")
|
||
|
||
# --- 正向:证明全通过 ⇒ 端点必须真的被公布 -----------------------
|
||
state.attest_ok = True
|
||
sess, rep, ok = wait_for(lambda s: len(s) == 1, min_attest=1)
|
||
check(
|
||
"★ 正向对照:证明全通过 ⇒ 会话端点真的被公布",
|
||
ok and len(sess) == 1,
|
||
json.dumps(sess, ensure_ascii=False)[:200],
|
||
)
|
||
check(
|
||
"★ 端点指向那台假 hostd(不是凭空造的)",
|
||
bool(sess) and sess[0].get("endpoint") == f"127.0.0.1:{fake_port}",
|
||
json.dumps(sess, ensure_ascii=False)[:200],
|
||
)
|
||
check(
|
||
"★ 公布的会话 ID 与期望态一致",
|
||
bool(sess) and sess[0].get("session_id") == "s1",
|
||
json.dumps(sess, ensure_ascii=False)[:200],
|
||
)
|
||
check(
|
||
"★ create 真的下发到了假 hostd(证明链路通)",
|
||
state.create_calls >= 1,
|
||
f"create_calls={state.create_calls}",
|
||
)
|
||
check(
|
||
"★ 对账结论把该会话记为 healthy",
|
||
"s1" in (rep.get("healthy") or []),
|
||
json.dumps(rep, ensure_ascii=False)[:200],
|
||
)
|
||
|
||
# --- 负向(**非恒真**):证明不齐 ⇒ 端点必须被收回 ---------------
|
||
# 此时 create 成功(假 hostd 可达),placed 有机会被写入;
|
||
# 上一小段已经证明它真能写入。所以这里的空集是「证明门控生效」,
|
||
# 不再是「链路根本没跑起来」的空集。这一段同时端到端钉死了
|
||
# 「证明漂移后旧端点不得继续公布」的 fail-closed 语义。
|
||
attest_before = state.attest_calls
|
||
state.attest_ok = False
|
||
sess2, rep2, ok2 = wait_for(
|
||
lambda s: len(s) == 0, min_attest=attest_before + 1
|
||
)
|
||
check(
|
||
"★ 负向对照:证明不齐 ⇒ 会话端点被收回(不再外泄)",
|
||
ok2 and len(sess2) == 0,
|
||
json.dumps(sess2, ensure_ascii=False)[:200],
|
||
)
|
||
check(
|
||
"★ 收回不是因为链路死了(对账仍在重新 create + 取证明)",
|
||
state.attest_calls > attest_before and state.create_calls >= 2,
|
||
f"attest {attest_before}->{state.attest_calls} "
|
||
f"create={state.create_calls}",
|
||
)
|
||
failed2 = rep2.get("failed") or []
|
||
check(
|
||
"★ 失败原因点名 attestation 未通过",
|
||
any("attestation 未通过" in (f.get("reason") or "") for f in failed2),
|
||
json.dumps(failed2, ensure_ascii=False)[:300],
|
||
)
|
||
|
||
# --- 恢复:证明重新通过 ⇒ 端点能再起来(不是一次性死锁) ---------
|
||
state.attest_ok = True
|
||
sess3, _, ok3 = wait_for(
|
||
lambda s: len(s) == 1, min_attest=state.attest_calls + 1
|
||
)
|
||
check(
|
||
"★ 证明恢复后端点能重新公布(fail-closed 不误伤恢复路径)",
|
||
ok3 and len(sess3) == 1,
|
||
json.dumps(sess3, ensure_ascii=False)[:200],
|
||
)
|
||
finally:
|
||
if proc.poll() is None:
|
||
proc.terminate()
|
||
try:
|
||
proc.wait(timeout=5)
|
||
except Exception:
|
||
proc.kill()
|
||
httpd.shutdown()
|
||
httpd.server_close()
|
||
|
||
|
||
def selftest_control_plane_auth():
|
||
"""集群控制面的鉴权:非回环必须配共享密钥 + 显式 --allow-remote。
|
||
|
||
# 为什么单独测
|
||
|
||
集群控制面会**下发 guest 启动**并公布已证明的会话端点。
|
||
没配共享密钥就绑非回环 = 谁能连上就能占机器。
|
||
|
||
# 判据纪律
|
||
|
||
两条负向各自独立起进程:① 没密钥 ⇒ 拒绝启动;② 有密钥但没
|
||
`--allow-remote` ⇒ 拒绝启动。两条都**不**测「启起来之后」——
|
||
那是上面那条正向测的。
|
||
"""
|
||
# --- 负向 ①:非回环 + 无密钥 ⇒ 拒绝启动 ---------------------------
|
||
proc = subprocess.Popen(
|
||
[
|
||
BIN,
|
||
"control-plane",
|
||
"serve",
|
||
"--listen",
|
||
"0.0.0.0:0",
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
try:
|
||
try:
|
||
out = proc.stdout.read().decode(errors="replace")
|
||
except Exception:
|
||
out = ""
|
||
check(
|
||
"★ 非回环且没共享密钥 ⇒ 控制面**拒绝启动**",
|
||
proc.poll() is not None and proc.returncode != 0,
|
||
f"rc={proc.returncode}",
|
||
)
|
||
check("★ 拒绝时说明要配共享密钥", "共享密钥" in out, out[:300])
|
||
finally:
|
||
if proc.poll() is None:
|
||
proc.terminate()
|
||
try:
|
||
proc.wait(timeout=5)
|
||
except Exception:
|
||
proc.kill()
|
||
|
||
# --- 负向 ②:非回环 + 有密钥但没 --allow-remote ⇒ 拒绝启动 --------
|
||
secret = "ab" * 32
|
||
proc = subprocess.Popen(
|
||
[
|
||
BIN,
|
||
"control-plane",
|
||
"serve",
|
||
"--listen",
|
||
"0.0.0.0:0",
|
||
"--shared-secret",
|
||
secret,
|
||
],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
)
|
||
try:
|
||
try:
|
||
out = proc.stdout.read().decode(errors="replace")
|
||
except Exception:
|
||
out = ""
|
||
check(
|
||
"★ 非回环且没 --allow-remote ⇒ 控制面**拒绝启动**",
|
||
proc.poll() is not None and proc.returncode != 0,
|
||
f"rc={proc.returncode}",
|
||
)
|
||
check("★ 拒绝时点名 --allow-remote", "--allow-remote" in out, out[:300])
|
||
finally:
|
||
if proc.poll() is None:
|
||
proc.terminate()
|
||
try:
|
||
proc.wait(timeout=5)
|
||
except Exception:
|
||
proc.kill()
|
||
|
||
|
||
def main():
|
||
if not os.path.exists(BIN):
|
||
print(f"{FAIL} 找不到 {BIN}(先 cargo build -p mpc-cli)")
|
||
return 1
|
||
|
||
port = free_port()
|
||
state_file = f"/tmp/mpc-hostd-selftest-{port}.json"
|
||
env = dict(os.environ)
|
||
proc = subprocess.Popen(
|
||
[BIN, "hostd", "--listen", f"127.0.0.1:{port}", "--state-file", state_file],
|
||
stdout=subprocess.PIPE,
|
||
stderr=subprocess.STDOUT,
|
||
env=env,
|
||
)
|
||
try:
|
||
if not wait_ready(port):
|
||
print(f"{FAIL} hostd 起来后 /health 不通")
|
||
print("--- hostd 输出 ---")
|
||
try:
|
||
proc.stdout.flush()
|
||
except Exception:
|
||
pass
|
||
return 1
|
||
check("hostd 起来且 /health 通", True)
|
||
|
||
# --- 建一个实例 ----------------------------------------------------
|
||
st, raw = request(
|
||
port,
|
||
"POST",
|
||
"/sandboxes",
|
||
json.dumps(
|
||
{
|
||
"session_id": "selftest-session",
|
||
"site": "alice",
|
||
"platform": "snp",
|
||
"template": "t-1",
|
||
"program": "/bin/sleep",
|
||
"args": ["300"],
|
||
# ★ 带绑定材料 —— 否则宿主侧判「本实例未配置会话绑定」,
|
||
# `chip_sig_note` 就不是「没做芯片验签」而是「不参与门控」,
|
||
# 下面那条判据的前��条件根本不成立(假红)。
|
||
"bind_session_id": "11" * 32,
|
||
"bind_session_pubkey": "22" * 32,
|
||
"bind_code_measurement": "33" * 32,
|
||
"tcb": 5,
|
||
"min_tcb": 3,
|
||
}
|
||
),
|
||
)
|
||
check("POST /sandboxes 返回 201", st == 201, f"got {st}: {raw[:200]}")
|
||
created = json.loads(raw) if st == 201 else {}
|
||
sb = created.get("sandboxID", "")
|
||
check("创建响应带 e2b 字段 sandboxID", bool(sb), raw[:200])
|
||
check(
|
||
"创建响应带 e2b 字段 templateID",
|
||
created.get("templateID") == "t-1",
|
||
raw[:200],
|
||
)
|
||
|
||
# --- e2b 兼容面:GET /sandboxes 是裸数组 ---------------------------
|
||
st, raw = request(port, "GET", "/sandboxes")
|
||
check("GET /sandboxes 返回 200", st == 200, f"got {st}")
|
||
try:
|
||
items = json.loads(raw)
|
||
except Exception as e:
|
||
items = None
|
||
check("GET /sandboxes 返回合法 JSON", False, str(e))
|
||
check(
|
||
"GET /sandboxes 返回**裸数组**(e2b 兼容面硬要求)",
|
||
isinstance(items, list),
|
||
f"got {type(items).__name__}: {raw[:200]}",
|
||
)
|
||
if isinstance(items, list) and items:
|
||
one = items[0]
|
||
for field in [
|
||
"templateID",
|
||
"sandboxID",
|
||
"clientID",
|
||
"envdVersion",
|
||
"timeout",
|
||
"memory",
|
||
]:
|
||
check(f"e2b 字段 {field} 存在", field in one, str(one)[:200])
|
||
|
||
# --- /attestation:四个布尔分开 ------------------------------------
|
||
st, raw = request(port, "GET", f"/sandboxes/{sb}/attestation")
|
||
check("GET /attestation 返回 200", st == 200, f"got {st}: {raw[:200]}")
|
||
try:
|
||
att = json.loads(raw)
|
||
except Exception:
|
||
att = {}
|
||
for field in ["user_data_ok", "meas_ok", "tcb_ok", "chip_sig"]:
|
||
check(
|
||
f"/attestation 字段 {field} 存在且是布尔",
|
||
isinstance(att.get(field), bool),
|
||
str(att)[:200],
|
||
)
|
||
check(
|
||
"/attestation **不含** 合并的 ok 字段",
|
||
"ok" not in att,
|
||
"合并成 ok=1 会把「没做芯片验签」说成「验证通过」",
|
||
)
|
||
if att.get("chip_sig") is False:
|
||
# ★ `chip_sig=false` 有**三种**成因,处置完全不同:
|
||
# ① 证据还没到(pending)—— note 说「尚未收到证明材料」;
|
||
# ② 证据到了但宿主没配验签器(没验)—— note 说「未做芯片验签」;
|
||
# ③ 验签器明确拒绝(验了没过)—— note 说「芯片验签失败」。
|
||
# 混判会把「还没交材料」当成「没验」⇒ 判据永远红。
|
||
# 所以这里按 note 区分,只对**成因 ②** 提要求。
|
||
note = att.get("chip_sig_note", "")
|
||
verdict = att.get("verdict", "")
|
||
if "尚未收到" in note or "未收到" in note:
|
||
check(
|
||
"chip_sig=false 且证据未到 ⇒ note 说清是「还没收到」",
|
||
"不得据此认为验证通过" in verdict,
|
||
f"note={note!r} verdict={verdict!r}",
|
||
)
|
||
else:
|
||
check(
|
||
"chip_sig=false 且证据已到 ⇒ note 说清是「没做验签」",
|
||
"未做芯片验签" in note or "未提供芯片验签器" in note,
|
||
f"note={note!r}",
|
||
)
|
||
check(
|
||
"chip_sig=false 时 verdict 明说不得据此认为验证通过",
|
||
"不得据此认为验证通过" in verdict,
|
||
verdict,
|
||
)
|
||
# ★ 「没验」**不能**被说成「验签失败」——
|
||
# 两者的处置完全不同(前者只是不公布,后者是硬拒绝回收)。
|
||
check(
|
||
"chip_sig=false 时**不**把「没验」说成「验签失败」",
|
||
"验签失败" not in note,
|
||
f"note={note!r}",
|
||
)
|
||
|
||
# --- 实例真的被回收 -----------------------------------------------
|
||
#
|
||
# 怎么确认「进程真没了」而不依赖外部命令:
|
||
# 宿主控制面起实例时,实例自己把 pid 写进一个文件;
|
||
# 测试读那个文件拿到 pid,然后让实例进程**自己**写一个「我还活着」
|
||
# 的心跳文件 —— 停掉后心跳文件不再更新,就是真回收了。
|
||
#
|
||
# 为什么不用 ps / /proc:
|
||
# macOS 没有 /proc,而受限环境里 ps 常被禁。
|
||
# 那时这条检查会被跳过 —— 而「状态改了但进程还在」正是
|
||
# 宿主控制面最典型的假回收,跳过等于放弃最重要的一条判据。
|
||
# 所以改成让实例自己作证。
|
||
pid_file = f"/tmp/mpc-hostd-selftest-pid-{port}.txt"
|
||
alive_marker = f"/tmp/mpc-hostd-selftest-alive-{port}.txt"
|
||
for f in (pid_file, alive_marker):
|
||
if os.path.exists(f):
|
||
os.unlink(f)
|
||
# 起一个会写心跳的实例:shell 循环,每 0.2 秒 touch 一次心跳文件。
|
||
prog = "/bin/sh"
|
||
args = [
|
||
"-c",
|
||
f"echo $$ > {pid_file}; while true; do touch {alive_marker}; sleep 0.2; done",
|
||
]
|
||
st, raw = request(
|
||
port,
|
||
"POST",
|
||
"/sandboxes",
|
||
json.dumps(
|
||
{
|
||
"session_id": "selftest-heartbeat",
|
||
"site": "alice",
|
||
"platform": "snp",
|
||
"program": prog,
|
||
"args": args,
|
||
}
|
||
),
|
||
)
|
||
check("心跳实例创建成功", st == 201, f"got {st}: {raw[:200]}")
|
||
hb = json.loads(raw) if st == 201 else {}
|
||
hb_id = hb.get("sandboxID", "")
|
||
|
||
# 等心跳文件出现(实例已经跑起来并在写)
|
||
deadline = time.time() + 5
|
||
while time.time() < deadline and not os.path.exists(alive_marker):
|
||
time.sleep(0.05)
|
||
check("实例进程在跑(心跳文件出现)", os.path.exists(alive_marker))
|
||
|
||
if hb_id:
|
||
st, raw = request(port, "DELETE", f"/sandboxes/{hb_id}")
|
||
check("DELETE 心跳实例返回 204", st == 204, f"got {st}: {raw[:200]}")
|
||
# 进程真被回收 ⇒ 心跳停止更新。
|
||
# 判据是「心跳文件的 mtime 不再前进」,而不是「文件没了」——
|
||
# 文件留着不动是正常的。
|
||
mtime_before = os.path.getmtime(alive_marker) if os.path.exists(alive_marker) else None
|
||
time.sleep(1.0)
|
||
mtime_after = os.path.getmtime(alive_marker) if os.path.exists(alive_marker) else None
|
||
check(
|
||
"实例进程真的被回收了(心跳停止,不是只改了状态字段)",
|
||
mtime_before is not None and mtime_after == mtime_before,
|
||
f"mtime {mtime_before} -> {mtime_after}(还在跳 = 进程没死)",
|
||
)
|
||
for f in (pid_file, alive_marker):
|
||
if os.path.exists(f):
|
||
os.unlink(f)
|
||
|
||
st, raw = request(port, "DELETE", f"/sandboxes/{sb}")
|
||
check("DELETE /sandboxes 返回 204(e2b 硬要求)", st == 204, f"got {st}: {raw[:200]}")
|
||
|
||
# ★★ DELETE 必须**真删**记录(2026-11-07 真机联调挖出的缺口)。
|
||
# 原来 DELETE 只调 stop()(停进程 + 标终态),记录仍在表里 ——
|
||
# 删完再 GET /sandboxes 还能看到它。症状极静默:
|
||
# DELETE 返回 204 看着成功,列表却越积越多。
|
||
# 这里用「新建一个 → 删 → 查列表」钉死,端到端走真 socket。
|
||
st, raw = request(
|
||
port,
|
||
"POST",
|
||
"/sandboxes",
|
||
json.dumps(
|
||
{
|
||
"session_id": "sess-delete-check",
|
||
"site": "alice",
|
||
"platform": "csv",
|
||
"program": "/bin/sleep",
|
||
"args": ["30"],
|
||
}
|
||
),
|
||
)
|
||
check("为删除判据新建实例成功", st in (200, 201), f"got {st}: {raw[:200]}")
|
||
del_id = ""
|
||
try:
|
||
del_id = json.loads(raw).get("sandboxID", "")
|
||
except Exception:
|
||
pass
|
||
check("新实例拿得到 sandboxID", bool(del_id), raw[:200])
|
||
|
||
st, raw = request(port, "GET", "/sandboxes")
|
||
check(
|
||
"删除前列表里有该实例",
|
||
st == 200 and del_id in raw,
|
||
f"got {st}: {raw[:300]}",
|
||
)
|
||
|
||
st, raw = request(port, "DELETE", f"/sandboxes/{del_id}")
|
||
check("删除判据:DELETE 返回 204", st == 204, f"got {st}: {raw[:200]}")
|
||
|
||
st, raw = request(port, "GET", "/sandboxes")
|
||
check(
|
||
"★ 删除后列表里**不再**有该实例(DELETE 必须真删记录)",
|
||
st == 200 and del_id not in raw,
|
||
f"got {st}: {raw[:300]}",
|
||
)
|
||
|
||
st, raw = request(port, "GET", f"/sandboxes/{del_id}")
|
||
check("★ 删除后单点查询返回 404", st == 404, f"got {st}: {raw[:200]}")
|
||
|
||
# --- 负向:未知实例 / 畸形请求 -------------------------------------
|
||
st, raw = request(port, "GET", "/sandboxes/nope-not-here")
|
||
check("未知实例返回 404", st == 404, f"got {st}")
|
||
check("404 响应带 message", "message" in raw, raw[:200])
|
||
|
||
st, raw = request(
|
||
port, "POST", "/sandboxes", json.dumps({"site": "alice"})
|
||
)
|
||
check("缺必填字段的创建请求返回 400", st == 400, f"got {st}")
|
||
check(
|
||
"400 响应点名缺哪个字段",
|
||
"session_id" in raw or "platform" in raw or "program" in raw,
|
||
raw[:200],
|
||
)
|
||
|
||
# --- 协议层边界(axum / hyper 承担解析,我们验契约) ---------------
|
||
dup = raw_request(
|
||
port,
|
||
b"POST /sandboxes HTTP/1.1\r\nHost: x\r\nContent-Length: 2\r\n"
|
||
b"Content-Length: 3\r\nConnection: close\r\n\r\n{}",
|
||
)
|
||
# ★ 带期望特征:hyper 在解析层拒 → 400 + 空体 + 关连接,
|
||
# 说明请求**根本没进业务层**。只判 400 会把「业务自己报错」也算过。
|
||
check(
|
||
"重复 Content-Length 被拒(请求走私入口)",
|
||
" 400 " in dup
|
||
and "content-length: 0" in dup.lower()
|
||
and "connection: close" in dup.lower(),
|
||
repr(dup[:200]),
|
||
)
|
||
|
||
chk = raw_request(
|
||
port,
|
||
b"POST /sandboxes HTTP/1.1\r\nHost: x\r\n"
|
||
b"Transfer-Encoding: chunked\r\nConnection: close\r\n\r\n0\r\n\r\n",
|
||
)
|
||
check(
|
||
"chunked 请求被拒且说明原因",
|
||
" 400 " in chk and "chunked" in chk.lower(),
|
||
repr(chk[:200]),
|
||
)
|
||
|
||
# ★ 响应完整性回归探针:上一版「头/body 分两次写、只读一次」导致
|
||
# 25 轮门禁偶发 2 次红。连续跑 20 次,每次都必须拿到完整 body。
|
||
ok_all = True
|
||
for _ in range(20):
|
||
st, body = request(port, "GET", "/health")
|
||
if st != 200 or body.strip() != "ok":
|
||
ok_all = False
|
||
break
|
||
check("连续 20 次请求都拿到完整响应体(防部分读回归)", ok_all)
|
||
|
||
finally:
|
||
proc.send_signal(signal.SIGTERM)
|
||
try:
|
||
proc.wait(timeout=5)
|
||
except Exception:
|
||
proc.kill()
|
||
try:
|
||
os.unlink(state_file)
|
||
except OSError:
|
||
pass
|
||
|
||
# --- 鉴权:凭据模式端到端 -------------------------------------------
|
||
selftest_auth()
|
||
|
||
# --- guest → hostd 的证明材料回传闭环 -------------------------------
|
||
selftest_evidence_upload()
|
||
|
||
# --- 鉴权:非回环启动期守卫 -----------------------------------------
|
||
selftest_remote_guard()
|
||
|
||
# --- 节点自报(Cubelet 自描述)--------------------------------------
|
||
selftest_self_report()
|
||
|
||
# --- 集群控制面(CubeMaster 等价物)--------------------------------
|
||
selftest_control_plane()
|
||
|
||
# --- 集群控制面的远程暴露守卫 --------------------------------------
|
||
selftest_control_plane_auth()
|
||
|
||
print()
|
||
if failures:
|
||
print(f"\033[31m{len(failures)} 条失败:\033[0m")
|
||
for f in failures:
|
||
print(f" - {f}")
|
||
return 1
|
||
print("\033[32m\033[1mhostd 自测全过\033[0m")
|
||
return 0
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|