Files
ssw-mpc/ray/build101.sh
T
rong 45bc626256 feat(ray): 对齐 Ray 1.0.1,nax 作为 driver 跑通跨架构 MPC
用户要求先用 1.0.1 对齐两端再把 nax MPC 跑通。闭环达成:

  x86_64  Ray node 73bfdc0a…  nax site=b  MPC 35
  armv7   Ray node 235afb22…  nax site=a  MPC 35

同一进程既是 Ray driver 又是 MPC 参与方;node_id 由 raylet 握手分配,
两端不同即「真的挂在两个节点上」的证据。35 = (1·4+1)+(2·5+1)+(3·6+1)。

1.0.1 shim(ray/nax_ray_shim_101.cc)与 2.59 的六个接口差异:
CoreWorkerProcess 无独立头、TaskExecutionCallback 只有 8 参数、
GcsClientOptions 位置参数、无 GetNodeToConnectForDriver/GetNextJobID、
无 ray::core 命名空间、GcsNodeInfo 无 node_manager_port。

两个必踩的坑:
- options.num_workers 必须显式设 1(1.0.1 有 RAY_CHECK(>0),默认 0 abort)
- gcs_options 要指向 Redis(6400)而非 gcs_server(6402),指错报
  Protocol error got "\x00",看着像协议问题其实是端口给错

同一版本两个架构的 string ABI 不同(关键):
x86_64 _raylet.so 是旧 ABI(0 个 cxx11),armv7 是新 ABI(423 个)。
搞错的表现是运行期 basic_string::_M_create 异常,错误信息不提 ABI。

编译补丁自动打在解出来的头上:client_call.h 的 vector<CompletionQueue> 换 deque
(grpc 1.51 删了移动构造)、影子 arrow/buffer.h、-Dgrpc_impl=grpc。
Python 符号 x86 缺 8 个 / armv7 缺 2 个,shim 零值占位(不执行 Python 代码)。

nax-cli 加可选 feature ray:设 NAX_RAY_RAYLET 即注册 driver,默认构建仍是
纯静态单二进制。rpath 放在 nax-cli 的 build.rs(link-arg-bins),放 nax-ray
里传不到依赖方的 bin。

新增 docs/RAY_CORE_101.md、ray/{nax_ray_shim_101.cc,build101.sh,
build101_armv7.sh,test101.py,test101_arm.c,verify101.sh}、ray/armv7/mpc_ray.sh;
nax-ray 重写为 1.0.1 签名;README 状态表更新。
2026-10-30 13:44:00 +08:00

92 lines
3.3 KiB
Bash
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/bin/bash
# 把 Ray 1.0.1 的 C++ core 编成 libnax_ray.so(x86_64 原生版)。
# 与 2.59 版(ray/build.sh)同一思路:不碰 bazel,直接链 wheel 里的 _raylet.so。
set -o pipefail
set -u
CACHE=/tmp
OUT_DIR=${OUT_DIR:-/tmp}
RAYSO=${RAYSO:-/tmp/r17x/ray-1.0.1.data/purelib/ray/_raylet.so}
HDR=/tmp/r101hdr
GEN=/tmp/gen101
SHIM=/tmp/shiminc101
echo "== ray 1.0.1 -> $OUT_DIR/libnax_ray.so"
echo " _raylet.so: $RAYSO"
[ -f "$RAYSO" ] || { echo "缺 _raylet.so"; exit 1; }
rm -rf "$HDR" "$GEN" "$SHIM"
mkdir -p "$HDR" "$GEN" "$SHIM"
tar xzf /tmp/r101hdr.tgz -C "$HDR"
echo " 头文件: $(find "$HDR" -name '*.h' | wc -l) 个"
cd "$HDR"
# reporter.proto 依赖 opencensus-proto(bazel 里才有),nax 用不到
PROTOS=$(find src/ray/protobuf -name '*.proto' | grep -v reporter.proto)
protoc -I. --cpp_out="$GEN" --grpc_out="$GEN" \
--plugin=protoc-gen-grpc="$(which grpc_cpp_plugin)" $PROTOS 2>&1 | grep -v "unused" | tail -3
echo " pb.h: $(ls "$GEN"/src/ray/protobuf/*.pb.h 2>/dev/null | wc -l) 个"
for f in $(find src -name '*.fbs'); do
flatc --cpp --scoped-enums -o "$(dirname "$f")" "$f" || exit 1
done
echo " flatbuffers: $(find src -name '*_generated.h' | wc -l) 个"
# --- 补丁:grpc 1.51 把 CompletionQueue 的移动构造删了,ray 1.0.1 用
# std::vector<CompletionQueue>+reserve 才能 emplace_back。换成 deque 即可原地构造。
python3 - <<'PYEOF'
import re
p = "/tmp/r101hdr/src/ray/rpc/client_call.h"
s = open(p).read()
s = s.replace(" cqs_.reserve(num_threads_);\n", "")
s = s.replace("std::vector<grpc::CompletionQueue> cqs_;",
"std::deque<grpc::CompletionQueue> cqs_;")
if "#include <deque>" not in s:
s = s.replace("#include <vector>", "#include <deque>\n#include <vector>", 1)
open(p, "w").write(s)
print(" patched client_call.h:", "std::deque<grpc::CompletionQueue> cqs_;" in s)
PYEOF
# 影子头 (a):arrow/buffer.h —— 1.0.1 里只有 PlasmaBuffer 用 std::shared_ptr<arrow::Buffer>,
# nax 不实例化 PlasmaBuffer(任务体不在 Ray 里跑),给个最小声明即可。
mkdir -p "$SHIM/arrow"
cat > "$SHIM/arrow/buffer.h" <<'EOF'
#pragma once
#include <cstddef>
#include <cstdint>
namespace arrow {
class Buffer {
public:
Buffer() = default;
virtual ~Buffer() = default;
virtual const uint8_t *data() const { return nullptr; }
virtual int64_t size() const { return 0; }
virtual bool is_mutable() const { return false; }
};
}
EOF
# 关键宏:
# _GLIBCXX_USE_CXX11_ABI=0 ray 1.0.1 也是旧 ABI(符号里没有 B5cxx11)
# ABSL_* absl::optional 走 std::optional
rm -f "$OUT_DIR/libnax_ray.so"
g++ -std=c++17 -O2 -fPIC -shared \
-D_GLIBCXX_USE_CXX11_ABI=0 \
-DABSL_USES_STD_OPTIONAL=1 -DABSL_OPTION_USE_STD_OPTIONAL=1 \
-Dgrpc_impl=grpc \
-o "$OUT_DIR/libnax_ray.so" /tmp/naxray/nax_ray_shim_101.cc \
-I"$SHIM" -I"$HDR" -I"$HDR/src" -I"$GEN" \
"$RAYSO" \
-labsl_synchronization -labsl_time -labsl_base -labsl_spinlock_wait -labsl_raw_logging_internal \
-Wl,--no-as-needed -L/usr/local/lib -lpython3.12 -Wl,-rpath,/usr/local/lib \
2>/tmp/b101.log
echo " errors: $(grep -c 'error:' /tmp/b101.log)"
if [ -f "$OUT_DIR/libnax_ray.so" ]; then
echo "== 成功: $OUT_DIR/libnax_ray.so ($(du -h "$OUT_DIR/libnax_ray.so" | cut -f1))"
else
echo "== 失败"; exit 1
fi