test(ray): 1.0.1 与 2.59 双向不兼容;改做 x86+armv7 混合 1.0.1 集群并跑通
兼容性(双向都测了):
- 1.0.1 raylet → 2.59 head:6379 报 Protocol error, got "\x00"
(2.59 的 6379 是 gRPC/HTTP2,1.0.1 说 RESP;Ray 2.0 移除了 Redis GCS 后端)
- 2.59 ray start --address=<1.0.1 gcs:6402> 报 RpcError
- 结论:没有中间路线,只能整集群同版本
混合架构集群(x86_64 原生 + armv7 qemu,同为 1.0.1):
- x86 raylet ID f85f933f…(cpu=4 / nm:6403)
- armv7 raylet ID 9f3e5ae1…(cpu=2 / nm:6413)
- 两侧节点通知是同一对 ID,均 IsAlive=1;armv7 收到 x86 心跳
- GCS 分片表 4 条:2×NODE_RESOURCE + 2×CLIENT,
CLIENT socket 分别 raylet_a.sock(armv7) / raylet_x.sock(x86)
新增 ray/armv7/{mixed_cluster.sh,compat_probe.sh,decode_gcs.py};
docs/RAY_ARMV7.md 增 §6 兼容性、§7 混合集群、§8 复现命令。
This commit is contained in:
1 parent
d60fcf40ca
commit
00e4fc322b
4 files changed
+270
-3
No files matched your search
+94
-3
@@ -116,7 +116,95 @@ raylet 存活 ✅
|
||||
**当前选择:C 作为默认,A 作为「确实需要 Ray 生态」时的备选。**
|
||||
B 不建议——为一个我们只用「放置」的能力去背 38 个依赖的 32 位交叉编译。
|
||||
|
||||
## 6. 复现
|
||||
## 6. 跨版本兼容性实测:1.0.1 ⟷ 2.59 双向都不通
|
||||
|
||||
结论:**不兼容,且不是「协商失败」,是传输层就不是一个协议。**
|
||||
|
||||
### 1.0.1 节点想加入 2.59 集群
|
||||
|
||||
```
|
||||
$ redis-cli -h 169.254.68.6 -p 6379 ping
|
||||
Error: Protocol error, got "\x00" as reply type byte
|
||||
|
||||
$ # 1.0.1 armv7 raylet --redis_address=<2.59 head> --redis_port=6379
|
||||
F redis_client.cc:114] Check failed: _s.ok()
|
||||
Bad status: RedisError: Protocol error, got "\x00" as reply type byte
|
||||
```
|
||||
|
||||
原因:Ray 2.0 把 Redis 从 GCS 后端彻底移除,head 的 6379 从 RESP 换成了 gRPC(HTTP/2)。
|
||||
1.0.1 的 raylet 一上来就发 RESP 的 `PING`,收到的第一个字节是 HTTP/2 的 `\x00`。
|
||||
|
||||
### 2.59 节点想加入 1.0.1 集群
|
||||
|
||||
```
|
||||
$ ray start --address=127.0.0.1:6402 # 1.0.1 的 gcs_server
|
||||
RuntimeError: Failed to connect to GCS. Last connection error:
|
||||
ray.exceptions.RpcError: RPC error:
|
||||
```
|
||||
|
||||
gRPC 这一层通了,但 GCS 的 service/protobuf 定义在 1.x → 2.x 之间已经改过,
|
||||
握手拿不到可用的服务。
|
||||
|
||||
### 所以
|
||||
|
||||
跨版本混部**没有中间路线**。要么整集群 2.x(那么 armv7 节点进不来,
|
||||
因为 2.59 没有 armv7 构建),要么整集群 1.0.1(那么 armv7 节点能进来,
|
||||
但要接受 2020 年的代码)。这就是「对成」的意思——把两端都放在 1.0.1 上。
|
||||
|
||||
## 7. 混合架构集群实测:x86_64 原生 + armv7,同为 1.0.1
|
||||
|
||||
`ray/armv7/mixed_cluster.sh`:x86_64 用 1.0.1 的 `manylinux1_x86_64` wheel 里的
|
||||
`redis-server` / `gcs_server` / `raylet` / `plasma_store_server` 原生跑;
|
||||
armv7 那一侧用 qemu-arm 跑同版本的二进制。两端同源同版本,只差指令集。
|
||||
|
||||
实测:
|
||||
|
||||
```
|
||||
=== 4) raylet x86_64(cpu=4)
|
||||
I node_manager.cc:178] Initializing NodeManager with ID f85f933f3a48916f22e2e5b32636793c58faee04
|
||||
I grpc_server.cc:74] NodeManager server started, listening on port 6403.
|
||||
|
||||
=== 5) raylet armv7(cpu=2,qemu)
|
||||
I node_manager.cc:178] Initializing NodeManager with ID 9f3e5ae101ab5da23e975b27f8f09d0c9ca00f72
|
||||
I grpc_server.cc:74] NodeManager server started, listening on port 6413.
|
||||
|
||||
--- 存活
|
||||
raylet x86 ✅
|
||||
raylet armv7 ✅
|
||||
```
|
||||
|
||||
**互相看得见**(两侧日志里的节点通知是同一对 ID):
|
||||
|
||||
x86 raylet 看到:
|
||||
```
|
||||
9f3e5ae101ab5da23e975b27f8f09d0c9ca00f72, IsAlive = 1
|
||||
f85f933f3a48916f22e2e5b32636793c58faee04, IsAlive = 1
|
||||
```
|
||||
armv7 raylet 看到:
|
||||
```
|
||||
9f3e5ae101ab5da23e975b27f8f09d0c9ca00f72, IsAlive = 1
|
||||
f85f933f3a48916f22e2e5b32636793c58faee04, IsAlive = 1
|
||||
I node_manager.cc:949] [HeartbeatAdded]: received heartbeat from unknown client id f85f933f3a48…
|
||||
```
|
||||
|
||||
GCS 表里正好 4 条(2 个节点 × `NODE_RESOURCE` + `CLIENT`),
|
||||
`ray/armv7/decode_gcs.py` 直接读 redis 分片解码:
|
||||
|
||||
```
|
||||
shard 键总数: 4
|
||||
NODE_RESOURCE node=9f3e5ae101ab5da23e975b27… value=49B → 1000000000 | 127.0.0.1
|
||||
NODE_RESOURCE node=f85f933f3a48916f22e2e5b3… value=49B → 1000000000 | 127.0.0.1
|
||||
CLIENT node=9f3e5ae101ab5da23e975b27… value=93B → /tmp/raylet_a.sock | /tmp/plasma_a.sock
|
||||
CLIENT node=f85f933f3a48916f22e2e5b3… value=93B → /tmp/raylet_x.sock | /tmp/plasma_x.sock
|
||||
```
|
||||
|
||||
CLIENT 条目里的 socket 路径一个 `_a`(armv7)一个 `_x`(x86),是两条独立的注册记录,
|
||||
不是同一条被覆盖。
|
||||
|
||||
**这一步的意义**:armv7 节点不是「能启动」,而是真的作为一个成员挂在集群里,
|
||||
和 x86 节点交换心跳、被 GCS 记账。跨指令集组集群这件事在 Ray 1.0.1 上是通的。
|
||||
|
||||
## 8. 复现
|
||||
|
||||
```bash
|
||||
# 拿包(GitHub 直连,注意 zip 可能被代理截断,用 -C - 续传)
|
||||
@@ -132,6 +220,9 @@ apt-get update && apt-get install -y libc6:armhf libstdc++6:armhf libgcc-s1:armh
|
||||
# 解包 + 起节点
|
||||
python3 -c "import zipfile;zipfile.ZipFile('ray_armv7.zip').extractall('.')"
|
||||
python3 -c "import zipfile,glob;zipfile.ZipFile(glob.glob('*.whl')[0]).extractall('/tmp/r17')"
|
||||
bash ray/armv7/run_node.sh # 完整节点
|
||||
bash ray/armv7/run_gcs_only.sh # 只起控制面
|
||||
bash ray/armv7/run_node.sh # armv7 单节点(redis+gcs+plasma+raylet)
|
||||
bash ray/armv7/run_gcs_only.sh # 只起 armv7 控制面
|
||||
bash ray/armv7/compat_probe.sh # 探测 1.0.1 ⟷ 2.59 不兼容
|
||||
bash ray/armv7/mixed_cluster.sh # x86_64 原生 + armv7 混合集群(同为 1.0.1)
|
||||
python3 ray/armv7/decode_gcs.py # 解码 GCS 节点表(需集群在跑)
|
||||
```
|
||||
@@ -0,0 +1,37 @@
|
||||
#!/bin/bash
|
||||
# 兼容性探测:ray 1.0.1 的 raylet 能不能接进 2.59 的 head?
|
||||
# 1.0.1 的 raylet 通过 redis 协议读 GCS 表;2.59 的 6379 是 gRPC GCS。
|
||||
GCS=169.254.68.6
|
||||
Q="qemu-arm-static -L /usr/lib/arm-linux-gnueabihf"
|
||||
D=/tmp/r17/ray-1.0.1.data/purelib/ray/core/src/ray
|
||||
|
||||
echo "=== ① 2.59 的 6379 说的是什么协议(试 redis PING)"
|
||||
timeout 8 redis-cli -h $GCS -p 6379 ping 2>&1 | head -3
|
||||
echo "--- 直接发 redis 命令,看返回"
|
||||
timeout 8 bash -c "printf 'PING\r\n' | timeout 5 nc $GCS 6379" 2>&1 | head -c 200 | od -c | head -3
|
||||
|
||||
echo
|
||||
echo "=== ② 1.0.1(armv7) raylet 试图加入 2.59 GCS"
|
||||
rm -f /tmp/plasma_a.sock /tmp/raylet_a.sock
|
||||
pkill -f 'qemu-arm-static.*plasma_store_server' 2>/dev/null
|
||||
$Q $D/../plasma/plasma_store_server -s /tmp/plasma_a.sock -m 20000000 -d /tmp/plasmadir_a > /tmp/plasma_a.log 2>&1 &
|
||||
sleep 2
|
||||
$Q $D/raylet/raylet \
|
||||
--raylet_socket_name=/tmp/raylet_a.sock \
|
||||
--store_socket_name=/tmp/plasma_a.sock \
|
||||
--object_manager_port=0 --min_worker_port=0 --max_worker_port=0 \
|
||||
--node_manager_port=6413 --node_ip_address=127.0.0.1 \
|
||||
--redis_address=$GCS --redis_port=6379 \
|
||||
--num_initial_workers=0 --maximum_startup_concurrency=1 \
|
||||
--static_resource_list=127.0.0.1,2,1000000000,20000000 \
|
||||
--config_list= --python_worker_command= --java_worker_command= \
|
||||
--cpp_worker_command=/bin/true --redis_password= \
|
||||
--temp_dir=/tmp/r7temp_a --session_dir=/tmp/r7sess_a \
|
||||
--metrics-agent-port=6414 --metrics_export_port=6415 > /tmp/r7raylet_compat.log 2>&1 &
|
||||
LP=$!
|
||||
sleep 25
|
||||
echo "--- 日志"
|
||||
grep -vE 'metric_exporter' /tmp/r7raylet_compat.log | head -12
|
||||
kill -0 $LP 2>/dev/null && { echo "--- raylet 存活(意外);两秒后杀"; sleep 2; kill -9 $LP; } || echo "--- raylet 已退出"
|
||||
pkill -f 'qemu-arm-static.*plasma_store_server' 2>/dev/null
|
||||
echo "=== done"
|
||||
@@ -0,0 +1,44 @@
|
||||
import socket, sys
|
||||
|
||||
def cmd(f, *args):
|
||||
out = b"*%d\r\n" % len(args)
|
||||
for a in args:
|
||||
if isinstance(a, str):
|
||||
a = a.encode()
|
||||
out += b"$%d\r\n%s\r\n" % (len(a), a)
|
||||
f.write(out); f.flush()
|
||||
|
||||
def read(f):
|
||||
line = f.readline()
|
||||
t, rest = line[:1], line[1:-2]
|
||||
if t == b'+': return rest
|
||||
if t == b'-': raise RuntimeError(rest)
|
||||
if t == b':': return int(rest)
|
||||
if t == b'$':
|
||||
n = int(rest)
|
||||
if n < 0: return None
|
||||
d = f.read(n + 2)[:-2]
|
||||
return d
|
||||
if t == b'*':
|
||||
n = int(rest)
|
||||
return [read(f) for _ in range(n)]
|
||||
raise RuntimeError('bad type ' + repr(t))
|
||||
|
||||
s = socket.create_connection(('127.0.0.1', 6401))
|
||||
f = s.makefile('rwb')
|
||||
cmd(f, 'DBSIZE'); print("shard 键总数:", read(f))
|
||||
|
||||
cmd(f, 'KEYS', '*')
|
||||
keys = read(f)
|
||||
print("条目:")
|
||||
for k in keys:
|
||||
cmd(f, 'GET', k)
|
||||
v = read(f)
|
||||
kind = k.split(b':')[0].decode('ascii', 'replace')
|
||||
nid = k.split(b':', 1)[1] if b':' in k else b''
|
||||
print(f" {kind:14s} node={nid.hex()[:24]}… value={len(v or b'')}B")
|
||||
# 把 value 里的可读片段抽出来(资源表是 protobuf,字符串字段能直接看见)
|
||||
printable = ''.join(chr(c) if 32 <= c < 127 else ' ' for c in (v or b''))
|
||||
runs = [t for t in printable.split() if len(t) >= 3]
|
||||
if runs:
|
||||
print(f" 可读片段: {' | '.join(runs[:8])}")
|
||||
@@ -0,0 +1,95 @@
|
||||
#!/bin/bash
|
||||
# 混合架构 ray 1.0.1 集群:x86_64 原生 head + armv7(qemu) 节点,两端同为 1.0.1
|
||||
Q="qemu-arm-static -L /usr/lib/arm-linux-gnueabihf"
|
||||
X=/tmp/r17x/ray-1.0.1.data/purelib/ray
|
||||
A=/tmp/r17/ray-1.0.1.data/purelib/ray
|
||||
XD=$X/core/src/ray
|
||||
AD=$A/core/src/ray
|
||||
MODX=$XD/gcs/redis_module/libray_redis_module.so
|
||||
MODA=$AD/gcs/redis_module/libray_redis_module.so
|
||||
|
||||
pkill -f qemu-arm-static 2>/dev/null; pkill -f 'r17x.*raylet' 2>/dev/null; sleep 1
|
||||
rm -rf /tmp/plasmadir_x /tmp/plasmadir_a /tmp/plasma_x.sock /tmp/plasma_a.sock \
|
||||
/tmp/raylet_x.sock /tmp/raylet_a.sock /tmp/mc_sess_x /tmp/mc_sess_a
|
||||
mkdir -p /tmp/plasmadir_x /tmp/plasmadir_a /tmp/mc_sess_x /tmp/mc_sess_a
|
||||
chmod +x $XD/gcs/gcs_server $XD/raylet/raylet $XD/thirdparty/redis/src/redis-server $MODX 2>/dev/null
|
||||
|
||||
echo "=== 1) redis 主/分片(x86_64 原生,1.0.1 自带)"
|
||||
$XD/thirdparty/redis/src/redis-server --port 6400 --save '' --appendonly no \
|
||||
--protected-mode no --loadmodule $MODX > /tmp/mc_redis.log 2>&1 &
|
||||
$XD/thirdparty/redis/src/redis-server --port 6401 --save '' --appendonly no \
|
||||
--protected-mode no --loadmodule $MODX > /tmp/mc_redis_b.log 2>&1 &
|
||||
for i in $(seq 1 30); do timeout 2 bash -c "echo > /dev/tcp/127.0.0.1/6400" 2>/dev/null && break; sleep 0.4; done
|
||||
for i in $(seq 1 30); do timeout 2 bash -c "echo > /dev/tcp/127.0.0.1/6401" 2>/dev/null && break; sleep 0.4; done
|
||||
grep -o "Module 'ray' loaded.*" /tmp/mc_redis.log | head -1
|
||||
redis-cli -p 6400 set NumRedisShards 1 >/dev/null
|
||||
redis-cli -p 6400 set RedirectOutput 0 >/dev/null
|
||||
redis-cli -p 6400 set JobCounter 0 >/dev/null
|
||||
redis-cli -p 6400 set ray_version 1.0.1 >/dev/null
|
||||
redis-cli -p 6400 set ray_commit matched >/dev/null
|
||||
redis-cli -p 6400 rpush RedisShards 127.0.0.1:6401 >/dev/null
|
||||
echo " redis ok"
|
||||
|
||||
echo "=== 2) gcs_server :6402(x86_64 原生)"
|
||||
$XD/gcs/gcs_server --redis_address 127.0.0.1 --redis_port 6400 \
|
||||
--node_ip_address 127.0.0.1 --gcs_server_port 6402 > /tmp/mc_gcs.log 2>&1 &
|
||||
for i in $(seq 1 40); do timeout 2 bash -c "echo > /dev/tcp/127.0.0.1/6402" 2>/dev/null && break; sleep 0.5; done
|
||||
grep -E 'listening on port|Gcs server address' /tmp/mc_gcs.log | head -2
|
||||
|
||||
echo "=== 3) plasma × 2"
|
||||
$X/core/src/plasma/plasma_store_server -s /tmp/plasma_x.sock -m 50000000 -d /tmp/plasmadir_x > /tmp/mc_plasma_x.log 2>&1 &
|
||||
$Q $A/core/src/plasma/plasma_store_server -s /tmp/plasma_a.sock -m 20000000 -d /tmp/plasmadir_a > /tmp/mc_plasma_a.log 2>&1 &
|
||||
for i in $(seq 1 30); do [ -S /tmp/plasma_x.sock ] && break; sleep 0.4; done
|
||||
for i in $(seq 1 30); do [ -S /tmp/plasma_a.sock ] && break; sleep 0.4; done
|
||||
[ -S /tmp/plasma_x.sock ] && echo " plasma x86 ok"; [ -S /tmp/plasma_a.sock ] && echo " plasma armv7 ok"
|
||||
|
||||
echo "=== 4) raylet x86_64(cpu=4)"
|
||||
$XD/raylet/raylet \
|
||||
--raylet_socket_name=/tmp/raylet_x.sock --store_socket_name=/tmp/plasma_x.sock \
|
||||
--object_manager_port=0 --min_worker_port=0 --max_worker_port=0 \
|
||||
--node_manager_port=6403 --node_ip_address=127.0.0.1 \
|
||||
--redis_address=127.0.0.1 --redis_port=6400 \
|
||||
--num_initial_workers=0 --maximum_startup_concurrency=1 \
|
||||
--static_resource_list=127.0.0.1,4,1000000000,50000000 \
|
||||
--config_list= --python_worker_command= --java_worker_command= \
|
||||
--cpp_worker_command=/bin/true --redis_password= \
|
||||
--temp_dir=/tmp/mc_sess_x/tmp --session_dir=/tmp/mc_sess_x \
|
||||
--metrics-agent-port=6404 --metrics_export_port=6405 > /tmp/mc_raylet_x.log 2>&1 &
|
||||
XP=$!
|
||||
for i in $(seq 1 40); do grep -q 'NodeManager server started' /tmp/mc_raylet_x.log && break; sleep 0.5; done
|
||||
grep -E 'Initializing NodeManager with ID|NodeManager server started' /tmp/mc_raylet_x.log | head -2
|
||||
|
||||
echo "=== 5) raylet armv7(cpu=2,qemu)"
|
||||
$Q $AD/raylet/raylet \
|
||||
--raylet_socket_name=/tmp/raylet_a.sock --store_socket_name=/tmp/plasma_a.sock \
|
||||
--object_manager_port=0 --min_worker_port=0 --max_worker_port=0 \
|
||||
--node_manager_port=6413 --node_ip_address=127.0.0.1 \
|
||||
--redis_address=127.0.0.1 --redis_port=6400 \
|
||||
--num_initial_workers=0 --maximum_startup_concurrency=1 \
|
||||
--static_resource_list=127.0.0.1,2,1000000000,20000000 \
|
||||
--config_list= --python_worker_command= --java_worker_command= \
|
||||
--cpp_worker_command=/bin/true --redis_password= \
|
||||
--temp_dir=/tmp/mc_sess_a/tmp --session_dir=/tmp/mc_sess_a \
|
||||
--metrics-agent-port=6414 --metrics_export_port=6415 > /tmp/mc_raylet_a.log 2>&1 &
|
||||
AP=$!
|
||||
for i in $(seq 1 60); do grep -q 'NodeManager server started' /tmp/mc_raylet_a.log && break; sleep 0.5; done
|
||||
grep -E 'Initializing NodeManager with ID|NodeManager server started' /tmp/mc_raylet_a.log | head -2
|
||||
|
||||
sleep 12
|
||||
echo
|
||||
echo "=== 6) GCS 里的节点表"
|
||||
(ss -lntp 2>/dev/null || netstat -lntp 2>/dev/null) | grep -E '6400|6402|6403|6413' | awk '{print $4}' | sort | head
|
||||
echo "--- redis 键"
|
||||
redis-cli -p 6400 keys '*' | sort | head -20
|
||||
echo "--- NODE 表里的节点"
|
||||
if redis-cli -p 6400 exists NODE | grep -q 1; then
|
||||
redis-cli -p 6400 hkeys NODE
|
||||
else
|
||||
for k in $(redis-cli -p 6400 keys '*NODE*'); do echo "key=$k type=$(redis-cli -p 6400 type $k)"; done
|
||||
fi
|
||||
echo "--- 存活"
|
||||
kill -0 $XP 2>/dev/null && echo " raylet x86 ✅" || echo " raylet x86 ❌"
|
||||
kill -0 $AP 2>/dev/null && echo " raylet armv7 ✅" || echo " raylet armv7 ❌"
|
||||
echo "--- armv7 节点日志尾部"
|
||||
grep -vE 'metric_exporter' /tmp/mc_raylet_a.log | tail -5
|
||||
echo "=== done(进程留着不杀,便于继续查)"
|
||||
Reference in new issue
Block a user