From 00e4fc322b1df2e5c01572203bcf6e0ba892b1b0 Mon Sep 17 00:00:00 2001 From: rong Date: Fri, 30 Oct 2026 12:14:26 +0800 Subject: [PATCH] =?UTF-8?q?test(ray):=201.0.1=20=E4=B8=8E=202.59=20?= =?UTF-8?q?=E5=8F=8C=E5=90=91=E4=B8=8D=E5=85=BC=E5=AE=B9=EF=BC=9B=E6=94=B9?= =?UTF-8?q?=E5=81=9A=20x86+armv7=20=E6=B7=B7=E5=90=88=201.0.1=20=E9=9B=86?= =?UTF-8?q?=E7=BE=A4=E5=B9=B6=E8=B7=91=E9=80=9A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 兼容性(双向都测了): - 1.0.1 raylet → 2.59 head:6379 报 Protocol error, got "\x00" (2.59 的 6379 是 gRPC/HTTP2,1.0.1 说 RESP;Ray 2.0 移除了 Redis GCS 后端) - 2.59 ray start --address=<1.0.1 gcs:6402> 报 RpcError - 结论:没有中间路线,只能整集群同版本 混合架构集群(x86_64 原生 + armv7 qemu,同为 1.0.1): - x86 raylet ID f85f933f…(cpu=4 / nm:6403) - armv7 raylet ID 9f3e5ae1…(cpu=2 / nm:6413) - 两侧节点通知是同一对 ID,均 IsAlive=1;armv7 收到 x86 心跳 - GCS 分片表 4 条:2×NODE_RESOURCE + 2×CLIENT, CLIENT socket 分别 raylet_a.sock(armv7) / raylet_x.sock(x86) 新增 ray/armv7/{mixed_cluster.sh,compat_probe.sh,decode_gcs.py}; docs/RAY_ARMV7.md 增 §6 兼容性、§7 混合集群、§8 复现命令。 --- docs/RAY_ARMV7.md | 97 ++++++++++++++++++++++++++++++++++++-- ray/armv7/compat_probe.sh | 37 +++++++++++++++ ray/armv7/decode_gcs.py | 44 +++++++++++++++++ ray/armv7/mixed_cluster.sh | 95 +++++++++++++++++++++++++++++++++++++ 4 files changed, 270 insertions(+), 3 deletions(-) create mode 100644 ray/armv7/compat_probe.sh create mode 100644 ray/armv7/decode_gcs.py create mode 100644 ray/armv7/mixed_cluster.sh diff --git a/docs/RAY_ARMV7.md b/docs/RAY_ARMV7.md index 06610a0..1779c3f 100644 --- a/docs/RAY_ARMV7.md +++ b/docs/RAY_ARMV7.md @@ -116,7 +116,95 @@ raylet 存活 ✅ **当前选择:C 作为默认,A 作为「确实需要 Ray 生态」时的备选。** B 不建议——为一个我们只用「放置」的能力去背 38 个依赖的 32 位交叉编译。 -## 6. 复现 +## 6. 跨版本兼容性实测:1.0.1 ⟷ 2.59 双向都不通 + +结论:**不兼容,且不是「协商失败」,是传输层就不是一个协议。** + +### 1.0.1 节点想加入 2.59 集群 + +``` +$ redis-cli -h 169.254.68.6 -p 6379 ping +Error: Protocol error, got "\x00" as reply type byte + +$ # 1.0.1 armv7 raylet --redis_address=<2.59 head> --redis_port=6379 +F redis_client.cc:114] Check failed: _s.ok() + Bad status: RedisError: Protocol error, got "\x00" as reply type byte +``` + +原因:Ray 2.0 把 Redis 从 GCS 后端彻底移除,head 的 6379 从 RESP 换成了 gRPC(HTTP/2)。 +1.0.1 的 raylet 一上来就发 RESP 的 `PING`,收到的第一个字节是 HTTP/2 的 `\x00`。 + +### 2.59 节点想加入 1.0.1 集群 + +``` +$ ray start --address=127.0.0.1:6402 # 1.0.1 的 gcs_server +RuntimeError: Failed to connect to GCS. Last connection error: + ray.exceptions.RpcError: RPC error: +``` + +gRPC 这一层通了,但 GCS 的 service/protobuf 定义在 1.x → 2.x 之间已经改过, +握手拿不到可用的服务。 + +### 所以 + +跨版本混部**没有中间路线**。要么整集群 2.x(那么 armv7 节点进不来, +因为 2.59 没有 armv7 构建),要么整集群 1.0.1(那么 armv7 节点能进来, +但要接受 2020 年的代码)。这就是「对成」的意思——把两端都放在 1.0.1 上。 + +## 7. 混合架构集群实测:x86_64 原生 + armv7,同为 1.0.1 + +`ray/armv7/mixed_cluster.sh`:x86_64 用 1.0.1 的 `manylinux1_x86_64` wheel 里的 +`redis-server` / `gcs_server` / `raylet` / `plasma_store_server` 原生跑; +armv7 那一侧用 qemu-arm 跑同版本的二进制。两端同源同版本,只差指令集。 + +实测: + +``` +=== 4) raylet x86_64(cpu=4) +I node_manager.cc:178] Initializing NodeManager with ID f85f933f3a48916f22e2e5b32636793c58faee04 +I grpc_server.cc:74] NodeManager server started, listening on port 6403. + +=== 5) raylet armv7(cpu=2,qemu) +I node_manager.cc:178] Initializing NodeManager with ID 9f3e5ae101ab5da23e975b27f8f09d0c9ca00f72 +I grpc_server.cc:74] NodeManager server started, listening on port 6413. + +--- 存活 + raylet x86 ✅ + raylet armv7 ✅ +``` + +**互相看得见**(两侧日志里的节点通知是同一对 ID): + +x86 raylet 看到: +``` +9f3e5ae101ab5da23e975b27f8f09d0c9ca00f72, IsAlive = 1 +f85f933f3a48916f22e2e5b32636793c58faee04, IsAlive = 1 +``` +armv7 raylet 看到: +``` +9f3e5ae101ab5da23e975b27f8f09d0c9ca00f72, IsAlive = 1 +f85f933f3a48916f22e2e5b32636793c58faee04, IsAlive = 1 +I node_manager.cc:949] [HeartbeatAdded]: received heartbeat from unknown client id f85f933f3a48… +``` + +GCS 表里正好 4 条(2 个节点 × `NODE_RESOURCE` + `CLIENT`), +`ray/armv7/decode_gcs.py` 直接读 redis 分片解码: + +``` +shard 键总数: 4 + NODE_RESOURCE node=9f3e5ae101ab5da23e975b27… value=49B → 1000000000 | 127.0.0.1 + NODE_RESOURCE node=f85f933f3a48916f22e2e5b3… value=49B → 1000000000 | 127.0.0.1 + CLIENT node=9f3e5ae101ab5da23e975b27… value=93B → /tmp/raylet_a.sock | /tmp/plasma_a.sock + CLIENT node=f85f933f3a48916f22e2e5b3… value=93B → /tmp/raylet_x.sock | /tmp/plasma_x.sock +``` + +CLIENT 条目里的 socket 路径一个 `_a`(armv7)一个 `_x`(x86),是两条独立的注册记录, +不是同一条被覆盖。 + +**这一步的意义**:armv7 节点不是「能启动」,而是真的作为一个成员挂在集群里, +和 x86 节点交换心跳、被 GCS 记账。跨指令集组集群这件事在 Ray 1.0.1 上是通的。 + +## 8. 复现 ```bash # 拿包(GitHub 直连,注意 zip 可能被代理截断,用 -C - 续传) @@ -132,6 +220,9 @@ apt-get update && apt-get install -y libc6:armhf libstdc++6:armhf libgcc-s1:armh # 解包 + 起节点 python3 -c "import zipfile;zipfile.ZipFile('ray_armv7.zip').extractall('.')" python3 -c "import zipfile,glob;zipfile.ZipFile(glob.glob('*.whl')[0]).extractall('/tmp/r17')" -bash ray/armv7/run_node.sh # 完整节点 -bash ray/armv7/run_gcs_only.sh # 只起控制面 +bash ray/armv7/run_node.sh # armv7 单节点(redis+gcs+plasma+raylet) +bash ray/armv7/run_gcs_only.sh # 只起 armv7 控制面 +bash ray/armv7/compat_probe.sh # 探测 1.0.1 ⟷ 2.59 不兼容 +bash ray/armv7/mixed_cluster.sh # x86_64 原生 + armv7 混合集群(同为 1.0.1) +python3 ray/armv7/decode_gcs.py # 解码 GCS 节点表(需集群在跑) ``` diff --git a/ray/armv7/compat_probe.sh b/ray/armv7/compat_probe.sh new file mode 100644 index 0000000..e2b3744 --- /dev/null +++ b/ray/armv7/compat_probe.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# 兼容性探测:ray 1.0.1 的 raylet 能不能接进 2.59 的 head? +# 1.0.1 的 raylet 通过 redis 协议读 GCS 表;2.59 的 6379 是 gRPC GCS。 +GCS=169.254.68.6 +Q="qemu-arm-static -L /usr/lib/arm-linux-gnueabihf" +D=/tmp/r17/ray-1.0.1.data/purelib/ray/core/src/ray + +echo "=== ① 2.59 的 6379 说的是什么协议(试 redis PING)" +timeout 8 redis-cli -h $GCS -p 6379 ping 2>&1 | head -3 +echo "--- 直接发 redis 命令,看返回" +timeout 8 bash -c "printf 'PING\r\n' | timeout 5 nc $GCS 6379" 2>&1 | head -c 200 | od -c | head -3 + +echo +echo "=== ② 1.0.1(armv7) raylet 试图加入 2.59 GCS" +rm -f /tmp/plasma_a.sock /tmp/raylet_a.sock +pkill -f 'qemu-arm-static.*plasma_store_server' 2>/dev/null +$Q $D/../plasma/plasma_store_server -s /tmp/plasma_a.sock -m 20000000 -d /tmp/plasmadir_a > /tmp/plasma_a.log 2>&1 & +sleep 2 +$Q $D/raylet/raylet \ + --raylet_socket_name=/tmp/raylet_a.sock \ + --store_socket_name=/tmp/plasma_a.sock \ + --object_manager_port=0 --min_worker_port=0 --max_worker_port=0 \ + --node_manager_port=6413 --node_ip_address=127.0.0.1 \ + --redis_address=$GCS --redis_port=6379 \ + --num_initial_workers=0 --maximum_startup_concurrency=1 \ + --static_resource_list=127.0.0.1,2,1000000000,20000000 \ + --config_list= --python_worker_command= --java_worker_command= \ + --cpp_worker_command=/bin/true --redis_password= \ + --temp_dir=/tmp/r7temp_a --session_dir=/tmp/r7sess_a \ + --metrics-agent-port=6414 --metrics_export_port=6415 > /tmp/r7raylet_compat.log 2>&1 & +LP=$! +sleep 25 +echo "--- 日志" +grep -vE 'metric_exporter' /tmp/r7raylet_compat.log | head -12 +kill -0 $LP 2>/dev/null && { echo "--- raylet 存活(意外);两秒后杀"; sleep 2; kill -9 $LP; } || echo "--- raylet 已退出" +pkill -f 'qemu-arm-static.*plasma_store_server' 2>/dev/null +echo "=== done" diff --git a/ray/armv7/decode_gcs.py b/ray/armv7/decode_gcs.py new file mode 100644 index 0000000..af947b3 --- /dev/null +++ b/ray/armv7/decode_gcs.py @@ -0,0 +1,44 @@ +import socket, sys + +def cmd(f, *args): + out = b"*%d\r\n" % len(args) + for a in args: + if isinstance(a, str): + a = a.encode() + out += b"$%d\r\n%s\r\n" % (len(a), a) + f.write(out); f.flush() + +def read(f): + line = f.readline() + t, rest = line[:1], line[1:-2] + if t == b'+': return rest + if t == b'-': raise RuntimeError(rest) + if t == b':': return int(rest) + if t == b'$': + n = int(rest) + if n < 0: return None + d = f.read(n + 2)[:-2] + return d + if t == b'*': + n = int(rest) + return [read(f) for _ in range(n)] + raise RuntimeError('bad type ' + repr(t)) + +s = socket.create_connection(('127.0.0.1', 6401)) +f = s.makefile('rwb') +cmd(f, 'DBSIZE'); print("shard 键总数:", read(f)) + +cmd(f, 'KEYS', '*') +keys = read(f) +print("条目:") +for k in keys: + cmd(f, 'GET', k) + v = read(f) + kind = k.split(b':')[0].decode('ascii', 'replace') + nid = k.split(b':', 1)[1] if b':' in k else b'' + print(f" {kind:14s} node={nid.hex()[:24]}… value={len(v or b'')}B") + # 把 value 里的可读片段抽出来(资源表是 protobuf,字符串字段能直接看见) + printable = ''.join(chr(c) if 32 <= c < 127 else ' ' for c in (v or b'')) + runs = [t for t in printable.split() if len(t) >= 3] + if runs: + print(f" 可读片段: {' | '.join(runs[:8])}") diff --git a/ray/armv7/mixed_cluster.sh b/ray/armv7/mixed_cluster.sh new file mode 100644 index 0000000..02f5e61 --- /dev/null +++ b/ray/armv7/mixed_cluster.sh @@ -0,0 +1,95 @@ +#!/bin/bash +# 混合架构 ray 1.0.1 集群:x86_64 原生 head + armv7(qemu) 节点,两端同为 1.0.1 +Q="qemu-arm-static -L /usr/lib/arm-linux-gnueabihf" +X=/tmp/r17x/ray-1.0.1.data/purelib/ray +A=/tmp/r17/ray-1.0.1.data/purelib/ray +XD=$X/core/src/ray +AD=$A/core/src/ray +MODX=$XD/gcs/redis_module/libray_redis_module.so +MODA=$AD/gcs/redis_module/libray_redis_module.so + +pkill -f qemu-arm-static 2>/dev/null; pkill -f 'r17x.*raylet' 2>/dev/null; sleep 1 +rm -rf /tmp/plasmadir_x /tmp/plasmadir_a /tmp/plasma_x.sock /tmp/plasma_a.sock \ + /tmp/raylet_x.sock /tmp/raylet_a.sock /tmp/mc_sess_x /tmp/mc_sess_a +mkdir -p /tmp/plasmadir_x /tmp/plasmadir_a /tmp/mc_sess_x /tmp/mc_sess_a +chmod +x $XD/gcs/gcs_server $XD/raylet/raylet $XD/thirdparty/redis/src/redis-server $MODX 2>/dev/null + +echo "=== 1) redis 主/分片(x86_64 原生,1.0.1 自带)" +$XD/thirdparty/redis/src/redis-server --port 6400 --save '' --appendonly no \ + --protected-mode no --loadmodule $MODX > /tmp/mc_redis.log 2>&1 & +$XD/thirdparty/redis/src/redis-server --port 6401 --save '' --appendonly no \ + --protected-mode no --loadmodule $MODX > /tmp/mc_redis_b.log 2>&1 & +for i in $(seq 1 30); do timeout 2 bash -c "echo > /dev/tcp/127.0.0.1/6400" 2>/dev/null && break; sleep 0.4; done +for i in $(seq 1 30); do timeout 2 bash -c "echo > /dev/tcp/127.0.0.1/6401" 2>/dev/null && break; sleep 0.4; done +grep -o "Module 'ray' loaded.*" /tmp/mc_redis.log | head -1 +redis-cli -p 6400 set NumRedisShards 1 >/dev/null +redis-cli -p 6400 set RedirectOutput 0 >/dev/null +redis-cli -p 6400 set JobCounter 0 >/dev/null +redis-cli -p 6400 set ray_version 1.0.1 >/dev/null +redis-cli -p 6400 set ray_commit matched >/dev/null +redis-cli -p 6400 rpush RedisShards 127.0.0.1:6401 >/dev/null +echo " redis ok" + +echo "=== 2) gcs_server :6402(x86_64 原生)" +$XD/gcs/gcs_server --redis_address 127.0.0.1 --redis_port 6400 \ + --node_ip_address 127.0.0.1 --gcs_server_port 6402 > /tmp/mc_gcs.log 2>&1 & +for i in $(seq 1 40); do timeout 2 bash -c "echo > /dev/tcp/127.0.0.1/6402" 2>/dev/null && break; sleep 0.5; done +grep -E 'listening on port|Gcs server address' /tmp/mc_gcs.log | head -2 + +echo "=== 3) plasma × 2" +$X/core/src/plasma/plasma_store_server -s /tmp/plasma_x.sock -m 50000000 -d /tmp/plasmadir_x > /tmp/mc_plasma_x.log 2>&1 & +$Q $A/core/src/plasma/plasma_store_server -s /tmp/plasma_a.sock -m 20000000 -d /tmp/plasmadir_a > /tmp/mc_plasma_a.log 2>&1 & +for i in $(seq 1 30); do [ -S /tmp/plasma_x.sock ] && break; sleep 0.4; done +for i in $(seq 1 30); do [ -S /tmp/plasma_a.sock ] && break; sleep 0.4; done +[ -S /tmp/plasma_x.sock ] && echo " plasma x86 ok"; [ -S /tmp/plasma_a.sock ] && echo " plasma armv7 ok" + +echo "=== 4) raylet x86_64(cpu=4)" +$XD/raylet/raylet \ + --raylet_socket_name=/tmp/raylet_x.sock --store_socket_name=/tmp/plasma_x.sock \ + --object_manager_port=0 --min_worker_port=0 --max_worker_port=0 \ + --node_manager_port=6403 --node_ip_address=127.0.0.1 \ + --redis_address=127.0.0.1 --redis_port=6400 \ + --num_initial_workers=0 --maximum_startup_concurrency=1 \ + --static_resource_list=127.0.0.1,4,1000000000,50000000 \ + --config_list= --python_worker_command= --java_worker_command= \ + --cpp_worker_command=/bin/true --redis_password= \ + --temp_dir=/tmp/mc_sess_x/tmp --session_dir=/tmp/mc_sess_x \ + --metrics-agent-port=6404 --metrics_export_port=6405 > /tmp/mc_raylet_x.log 2>&1 & +XP=$! +for i in $(seq 1 40); do grep -q 'NodeManager server started' /tmp/mc_raylet_x.log && break; sleep 0.5; done +grep -E 'Initializing NodeManager with ID|NodeManager server started' /tmp/mc_raylet_x.log | head -2 + +echo "=== 5) raylet armv7(cpu=2,qemu)" +$Q $AD/raylet/raylet \ + --raylet_socket_name=/tmp/raylet_a.sock --store_socket_name=/tmp/plasma_a.sock \ + --object_manager_port=0 --min_worker_port=0 --max_worker_port=0 \ + --node_manager_port=6413 --node_ip_address=127.0.0.1 \ + --redis_address=127.0.0.1 --redis_port=6400 \ + --num_initial_workers=0 --maximum_startup_concurrency=1 \ + --static_resource_list=127.0.0.1,2,1000000000,20000000 \ + --config_list= --python_worker_command= --java_worker_command= \ + --cpp_worker_command=/bin/true --redis_password= \ + --temp_dir=/tmp/mc_sess_a/tmp --session_dir=/tmp/mc_sess_a \ + --metrics-agent-port=6414 --metrics_export_port=6415 > /tmp/mc_raylet_a.log 2>&1 & +AP=$! +for i in $(seq 1 60); do grep -q 'NodeManager server started' /tmp/mc_raylet_a.log && break; sleep 0.5; done +grep -E 'Initializing NodeManager with ID|NodeManager server started' /tmp/mc_raylet_a.log | head -2 + +sleep 12 +echo +echo "=== 6) GCS 里的节点表" +(ss -lntp 2>/dev/null || netstat -lntp 2>/dev/null) | grep -E '6400|6402|6403|6413' | awk '{print $4}' | sort | head +echo "--- redis 键" +redis-cli -p 6400 keys '*' | sort | head -20 +echo "--- NODE 表里的节点" +if redis-cli -p 6400 exists NODE | grep -q 1; then + redis-cli -p 6400 hkeys NODE +else + for k in $(redis-cli -p 6400 keys '*NODE*'); do echo "key=$k type=$(redis-cli -p 6400 type $k)"; done +fi +echo "--- 存活" +kill -0 $XP 2>/dev/null && echo " raylet x86 ✅" || echo " raylet x86 ❌" +kill -0 $AP 2>/dev/null && echo " raylet armv7 ✅" || echo " raylet armv7 ❌" +echo "--- armv7 节点日志尾部" +grep -vE 'metric_exporter' /tmp/mc_raylet_a.log | tail -5 +echo "=== done(进程留着不杀,便于继续查)"