相关链接
- 特性清单: https://gitcode.com/openFuyao/release-management/blob/main/openFuyao-26.03/release-plan.md
- 安装部署前置环境校验工具使用指导: https://gitcode.com/openFuyao/sig-installation/blob/master/docs/zh/user_guide/cluster_installation_deployment/environment_pre_check_tool_guide.md
测试环境
- CPU:
Kunpeng-920 - OS:
openEuler 24.03 LTS SP3 aarch64 - Fuyao Version:
v26.03 rc3 - docker:
2:18.09.0-346.oe2403sp3
测试特性
- [ ] 在线部署；
- [ ] 离线包制备；
- [ ] 离线部署；
- [ ] 安装部署前置检查工具；
- [ ] NPU Operator；
[ ] AI推理套件；
建议优化点
- 环境检测工具，检查 iptables 默认策略是否放行，若未放行可能在部署成功后无法访问；默认防火墙策略为 FORWARD DROP ，对集群运行和访问带来的潜在问题；
- 运行 cli 前检查是否存在命令并及时抛出错误；检查
tar/unzip是否安装，安装过程有很多地方会用到，而且出错时不会得到明显的解压失败报错，难以定位问题。 安装命令变化，考虑上下兼容性？
场景记录
离线部署管理面和业务面集群
- CPU:
Kunpeng-920 - OS:
openEuler 24.03 LTS SP3 aarch64 - Fuyao Version:
v26.03 rc3 docker:
2:18.09.0-346.oe2403sp3arm64 环境下构建离线制品包为什么会执行 amd64 的 bin
[bke][2026-03-26 07:48:49][INFO] The bke binary file version is . sh: line 1: /root/fuyao-26-03/packages/usr/bin/bkeadm_linux_amd64: cannot execute binary file: Exec format error
完整日志
[root@master1 fuyao-26-03]# cat build-offline-package.log | grep -v sha256
2026-03-26T06:49:56.467+0800 info infrastructure/infrastructure.go:53 The docker client is ready.
[bke][2026-03-26 06:49:56][step.1] Configuration file check
[bke][2026-03-26 06:49:56][step.2] Creates a workspace in the current directory
[bke][2026-03-26 06:49:56][step.5] Collect the required image files
[bke][2026-03-26 06:49:56][INFO] Try pulling away the mirror image cr.openfuyao.cn/openfuyao/registry:2.8.1
[bke][2026-03-26 06:49:56][step.3] Collect host dependency packages and package files
[bke][2026-03-26 06:49:56][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/kubernetes/kubernetes/releases/download/1.34.3-of.1/bin/linux/arm64/kubectl to /root/fuyao-26-03/packages/tmp/packages/files/kubectl-v1.34.3-of.1-arm64
[bke][2026-03-26 06:50:07][INFO] Try pulling away the mirror image cr.openfuyao.cn/openfuyao/registry:2.8.1
[bke][2026-03-26 06:50:18][step.6] Collect images from the source repository to the target repository
[bke][2026-03-26 06:50:18][INFO] Remove the image repository
2026-03-26T06:50:18.026+0800 info infrastructure/infrastructure.go:53 The docker client is ready.
2026-03-26T06:50:18.028+0800 info infrastructure/infrastructure.go:53 The docker client is ready.
[bke][2026-03-26 06:50:20][WARN] Image cr.openfuyao.cn/openfuyao/registry:2.8.1 inspect failed: Error response from daemon: no such image: cr.openfuyao.cn/openfuyao/registry:2.8.1: No such image: cr.openfuyao.cn/openfuyao/registry:2.8.1, retrying (1/3)...
[bke][2026-03-26 06:50:21][WARN] Image cr.openfuyao.cn/openfuyao/registry:2.8.1 inspect failed: Error response from daemon: no such image: cr.openfuyao.cn/openfuyao/registry:2.8.1: No such image: cr.openfuyao.cn/openfuyao/registry:2.8.1, retrying (2/3)...
[bke][2026-03-26 06:50:22][WARN] Image cr.openfuyao.cn/openfuyao/registry:2.8.1 inspect failed: Error response from daemon: no such image: cr.openfuyao.cn/openfuyao/registry:2.8.1: No such image: cr.openfuyao.cn/openfuyao/registry:2.8.1, retrying (3/3)...
[bke][2026-03-26 06:50:23][WARN] Get image cr.openfuyao.cn/openfuyao/registry:2.8.1 inspect failed: failed to inspect image cr.openfuyao.cn/openfuyao/registry:2.8.1 after 3 attempts: Error response from daemon: no such image: cr.openfuyao.cn/openfuyao/registry:2.8.1: No such image: cr.openfuyao.cn/openfuyao/registry:2.8.1
[bke][2026-03-26 06:50:23][INFO] Image cr.openfuyao.cn/openfuyao/registry:2.8.1 is downloading
[bke][2026-03-26 06:50:37][INFO] Wait for the container mirroring service to start...
[bke][2026-03-26 06:50:42][INFO] The container mirroring service is started.
Getting image list signatures
Copying 6 images generated from 6 images in list
Getting image source signatures
[bke][2026-03-26 06:50:56][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/kubernetes/kubernetes/releases/download/1.34.3-of.1/bin/linux/arm64/kubelet to /root/fuyao-26-03/packages/tmp/packages/files/kubelet-v1.34.3-of.1-arm64
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 7 images generated from 7 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:52:10][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/kubernetes/kubernetes/releases/download/1.34.3-of.1/bin/linux/amd64/kubectl to /root/fuyao-26-03/packages/tmp/packages/files/kubectl-v1.34.3-of.1-amd64
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:53:23][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/kubernetes/kubernetes/releases/download/1.34.3-of.1/bin/linux/amd64/kubelet to /root/fuyao-26-03/packages/tmp/packages/files/kubelet-v1.34.3-of.1-amd64
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:54:34][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/containerd/containerd/releases/download/v2.1.1-origin/containerd-v2.1.1-linux-amd64.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/containerd-v2.1.1-linux-amd64.tar.gz
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 3 images generated from 3 images in list
Getting image source signatures
[bke][2026-03-26 06:55:53][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/containerd/containerd/releases/download/v2.1.1-origin/containerd-v2.1.1-linux-arm64.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/containerd-v2.1.1-linux-arm64.tar.gz
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:57:07][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/containernetworking/plugins/releases/download/v1.4.1/cni-plugins-linux-amd64-v1.4.1.tgz to /root/fuyao-26-03/packages/tmp/packages/files/cni-plugins-linux-amd64-v1.4.1.tgz
[bke][2026-03-26 06:58:00][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/containernetworking/plugins/releases/download/v1.4.1/cni-plugins-linux-arm64-v1.4.1.tgz to /root/fuyao-26-03/packages/tmp/packages/files/cni-plugins-linux-arm64-v1.4.1.tgz
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:58:49][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/helm/releases/download/v3.14.2/helm-v3.14.2-linux-amd64.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/helm-v3.14.2-linux-amd64.tar.gz
[bke][2026-03-26 06:59:08][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/helm/releases/download/v3.14.2/helm-v3.14.2-linux-arm64.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/helm-v3.14.2-linux-arm64.tar.gz
[bke][2026-03-26 06:59:24][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/mikefarah/yq/releases/download/v4.43.1/yq_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/yq_linux_arm64
[bke][2026-03-26 06:59:35][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/mikefarah/yq/releases/download/v4.43.1/yq_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/yq_linux_amd64
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 10 images generated from 10 images in list
Getting image source signatures
[bke][2026-03-26 06:59:44][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/jqlang/jq/releases/download/v1.7.1/jq-linux-arm64 to /root/fuyao-26-03/packages/tmp/packages/files/jq-linux-arm64
[bke][2026-03-26 06:59:46][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/jqlang/jq/releases/download/v1.7.1/jq-linux-amd64 to /root/fuyao-26-03/packages/tmp/packages/files/jq-linux-amd64
[bke][2026-03-26 06:59:49][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssl-certinfo_1.6.4_linux_arm64
[bke][2026-03-26 07:00:00][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssl-certinfo_1.6.4_linux_amd64
[bke][2026-03-26 07:00:11][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssl_1.6.4_linux_arm64
[bke][2026-03-26 07:00:24][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssl_1.6.4_linux_amd64
[bke][2026-03-26 07:00:38][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssljson_1.6.4_linux_arm64
[bke][2026-03-26 07:00:46][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssljson_1.6.4_linux_amd64
[bke][2026-03-26 07:00:55][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/opencontainers/runc/releases/download/v1.1.12/runc-arm64 to /root/fuyao-26-03/packages/tmp/packages/files/runc-arm64
[bke][2026-03-26 07:01:07][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/opencontainers/runc/releases/download/v1.1.12/runc-amd64 to /root/fuyao-26-03/packages/tmp/packages/files/runc-amd64
[bke][2026-03-26 07:01:19][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/etcd-io/etcd/releases/download/v3.5.6/etcdctl-v3.5.6-linux-amd64 to /root/fuyao-26-03/packages/tmp/packages/files/etcdctl-v3.5.6-linux-amd64
[bke][2026-03-26 07:01:40][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/etcd-io/etcd/releases/download/v3.5.6/etcdctl-v3.5.6-linux-arm64 to /root/fuyao-26-03/packages/tmp/packages/files/etcdctl-v3.5.6-linux-arm64
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 07:01:58][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/rpm/releases/download/v0.0.1/rpm.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/rpm.tar.gz
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 7 images generated from 7 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 8 images generated from 8 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 07:45:30][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/nfs/nfsshare.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/nfsshare.tar.gz
[bke][2026-03-26 07:45:30][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/bkeadm/releases/download/1.2.1/bkeadm_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/bkeadm_linux_amd64
[bke][2026-03-26 07:46:48][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/bkeadm/releases/download/1.2.1/bkeadm_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/bkeadm_linux_arm64
[bke][2026-03-26 07:48:03][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/oauth-webhook-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/oauth-webhook-1.0.2.tgz
Writing manifest to image destination
[bke][2026-03-26 07:48:03][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/oauth-server-1.0.3.tgz to /root/fuyao-26-03/packages/tmp/charts/oauth-server-1.0.3.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/console-website-1.0.4.tgz to /root/fuyao-26-03/packages/tmp/charts/console-website-1.0.4.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/monitoring-service-1.0.4.tgz to /root/fuyao-26-03/packages/tmp/charts/monitoring-service-1.0.4.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/console-service-1.0.4.tgz to /root/fuyao-26-03/packages/tmp/charts/console-service-1.0.4.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/marketplace-service-1.0.3.tgz to /root/fuyao-26-03/packages/tmp/charts/marketplace-service-1.0.3.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/application-management-service-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/application-management-service-1.0.2.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/plugin-management-service-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/plugin-management-service-1.0.2.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/user-management-operator-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/user-management-operator-1.0.2.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/web-terminal-service-1.0.3.tgz to /root/fuyao-26-03/packages/tmp/charts/web-terminal-service-1.0.3.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/installer-service-1.2.1.tgz to /root/fuyao-26-03/packages/tmp/charts/installer-service-1.2.1.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/installer-website-1.2.1.tgz to /root/fuyao-26-03/packages/tmp/charts/installer-website-1.2.1.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/harbor-1.11.4.tgz to /root/fuyao-26-03/packages/tmp/charts/harbor-1.11.4.tgz
Getting image source signatures
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/bke-console-website-1.0.3.tgz to /root/fuyao-26-03/packages/tmp/charts/bke-console-website-1.0.3.tgz
[bke][2026-03-26 07:48:05][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/bke-console-service-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/bke-console-service-1.0.2.tgz
[bke][2026-03-26 07:48:49][step.4] Collect the bke binary file
[bke][2026-03-26 07:48:49][INFO] The bke binary file version is . sh: line 1: /root/fuyao-26-03/packages/usr/bin/bkeadm_linux_amd64: cannot execute binary file: Exec format error
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 17 images generated from 17 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 16 images generated from 16 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 10 images generated from 10 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
[bke][2026-03-26 08:41:56][INFO] The system starts to pack the image file.
[bke][2026-03-26 08:43:49][INFO] Remove the image repository
2026-03-26T08:43:49.940+0800 info infrastructure/infrastructure.go:53 The docker client is ready.
[bke][2026-03-26 08:43:51][step.7] Build the bke package, please wait for the larger package...
[bke][2026-03-26 08:46:22][step.8] Packaging complete bke.tar.gz
openEuler 24.03 lts 部署 npu operator 有 pod 无法启动：
Every 1.0s: kubectl get pod -A master1: Fri Mar 27 10:33:04 2026
NAMESPACE NAME READY STATUS RESTARTS AGE
cluster-system bkeagent-deployer-fcnfd 1/1 Running 0 29m
default mindio-acp-g67n9 1/1 Running 0 7m56s
default mindio-tft-bk8mj 1/1 Running 0 7m57s
ingress-nginx ingress-nginx-admission-create-fh4rv 0/1 Completed 0 28m
ingress-nginx ingress-nginx-admission-patch-nn79l 0/1 Completed 2 28m
ingress-nginx ingress-nginx-controller-jdb5s 1/1 Running 0 28m
kube-system ascend-device-plugin-5hmhf 0/1 CrashLoopBackOff 6 (33s ago) 7m59s
kube-system calico-kube-controllers-9c987b475-db86h 1/1 Running 0 29m
kube-system calico-node-sfvjm 1/1 Running 0 29m
kube-system coredns-6bd9b76578-2mqzh 1/1 Running 2 (29m ago) 29m
kube-system coredns-6bd9b76578-94pmp 1/1 Running 2 (21m ago) 29m
kube-system etcd-master1 1/1 Running 0 28m
kube-system kube-apiserver-master1 1/1 Running 0 27m
kube-system kube-controller-manager-master1 1/1 Running 1 (27m ago) 29m
kube-system kube-proxy-rwpzc 1/1 Running 0 29m
kube-system kube-scheduler-master1 1/1 Running 1 (27m ago) 29m
kube-system metrics-server-db68b78d-b8kkn 1/1 Running 0 26m
mindx-dl ascend-operator-manager-5d4d89f675-8dzn4 0/1 Pending 0 7m59s
mindx-dl clusterd-5588c5dc88-qmblm 0/1 Pending 0 7m57s
mindx-dl noded-rc5lj 1/1 Running 0 7m59s
mindx-dl resilience-controller-8686bbd76f-qzc2z 0/1 Pending 0 7m57s
monitoring alertmanager-main-0 2/2 Running 0 28m
monitoring alertmanager-main-1 2/2 Running 0 28m
monitoring alertmanager-main-2 2/2 Running 0 28m
monitoring blackbox-exporter-6877f5c5f7-66qn4 3/3 Running 0 28m
monitoring kube-state-metrics-6f6c47f5f4-4pt9s 3/3 Running 0 28m
monitoring node-exporter-ng468 2/2 Running 0 28m
monitoring prometheus-k8s-0 2/2 Running 0 28m
monitoring prometheus-k8s-1 2/2 Running 0 28m
monitoring prometheus-operator-6698d7bc85-72xwx 2/2 Running 0 28m
npu-exporter npu-exporter-vnzk6 0/1 Pending 0 7m57s
npu ascend-runtime-containerd-7s6jv 1/1 Running 0 8m
npu npu-driver-l7rjp 1/1 Running 0 8m
npu npu-feature-discovery-xr7nt 1/1 Running 0 19m
npu npu-node-feature-discovery-gc-5d97746dbc-wvnx6 1/1 Running 0 9m52s
npu npu-node-feature-discovery-master-664666b7bb-7qjfh 1/1 Running 0 9m43s
npu npu-node-feature-discovery-worker-8mj5w 1/1 Running 0 9m38s
npu npu-operator-5678cd59d4-ljvc8 1/1 Running 0 19m
openfuyao-system-controller modify-manifests-master1-rhfqf 0/1 Completed 0 27m
openfuyao-system-controller openfuyao-system-controller-798c4f6598-npmjs 1/1 Running 0 29m
openfuyao-system application-management-service-77457c5c85-g7b6z 2/2 Running 0 28m
openfuyao-system console-service-7fdb88c9c6-g2kp5 1/1 Running 0 28m
openfuyao-system console-website-c78945fcc-dm6s4 1/1 Running 0 28m
openfuyao-system local-harbor-chartmuseum-57fdd9949d-5xwh6 1/1 Running 0 28m
openfuyao-system local-harbor-core-6f7d4cc767-9rp6l 1/1 Running 0 28m
openfuyao-system local-harbor-database-0 1/1 Running 0 28m
openfuyao-system local-harbor-jobservice-7b8c9bf798-ms8qh 1/1 Running 4 (27m ago) 28m
openfuyao-system local-harbor-nginx-78b94f7b74-nflct 1/1 Running 0 28m
openfuyao-system local-harbor-portal-6b8cbf6747-gq6wd 1/1 Running 0 28m
openfuyao-system local-harbor-redis-0 1/1 Running 0 28m
openfuyao-system local-harbor-registry-7879c9d46d-d8wq6 2/2 Running 0 28m
openfuyao-system marketplace-service-5cf7cd6f5b-bmknq 2/2 Running 0 28m
openfuyao-system monitoring-service-6fd8dbd59f-bwq96 2/2 Running 0 28m
openfuyao-system oauth-server-64f6545c48-2hm7z 1/1 Running 0 26m
openfuyao-system oauth-webhook-78cb864fc5-gsmw9 1/1 Running 0 26m
openfuyao-system plugin-management-service-6fd8f64cc4-zvql8 2/2 Running 0 26m
openfuyao-system user-management-operator-9bb7bf64-6q9xg 1/1 Running 0 26m
openfuyao-system web-terminal-service-6f64b888f9-gdxpb 1/1 Running 0 26m
volcano-system volcano-controllers-6ffb787f8d-chk69 0/1 Pending 0 7m58s
volcano-system volcano-scheduler-867f9784bb-tvhzh 0/1 Pending 0 7m57s
[root@master1 fuyao-26-03]# kubectl -n kube-system logs ascend-device-plugin-5hmhf
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
[INFO] 2026/03/27 02:26:56.030212 1 hwlog/api.go:108 devicePlugin.log's logger init success
[INFO] 2026/03/27 02:26:56.030715 1 main.go:187 ascend device plugin starting and the version is v6.0.0_linux-aarch64
[INFO] 2026/03/27 02:26:56.030759 1 main.go:188 ascend device plugin starting scene is center
2026/03/27 02:26:56 command exec failed, exit status 1
[ERROR] 2026/03/27 02:26:56.032950 1 devmanager/devmanager.go:95 deviceManager init failed, prepare dcmi failed, err: cannot found valid driver lib, fromEnv: lib path is invalid, [], fromLdCmd: can't find valid lib: EOF
[ERROR] 2026/03/27 02:26:56.033013 1 main.go:203 init devmanager failed, err: auto init failed, err: get chip info failed, err: device Manager is nil, may encounter an exception during initialization. You can check the system log to confirm
环境检查
参考 https://gitcode.com/openFuyao/sig-installation/blob/master/docs/zh/user_guide/cluster_installation_deployment/environment_pre_check_tool_guide.md
wget https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/env-check/releases/download/latest/bin/linux/arm64/envCheck
wget https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/env-check/releases/download/latest/bin/linux/arm64/config.json文件检查：
[root@localhost env]# ./envCheck query
[INFO][2026-03-27 02:53:25] envCheck tool start
[INFO][2026-03-27 02:53:25] mode: fileQuery
[INFO][2026-03-27 02:53:25] file query start ......
[INFO][2026-03-27 02:53:25] query path: [/root/.kube /etc/kubernetes /usr/local/bin/crictl /etc/sysctl.d/k8s.conf /etc/systemd/system/kubelet.service /etc/systemd/system/kubelet.service.d /var/lib/etcd /var/lib/kubelet /run/containerd/containerd.sock /usr/lib/systemd/system/kubelet.service.d /var/run/containerd/containerd.sock /var/run/docker.sock]
[INFO][2026-03-27 02:53:25] path not exist: /root/.kube
[INFO][2026-03-27 02:53:25] path not exist: /etc/kubernetes
[INFO][2026-03-27 02:53:25] path not exist: /usr/local/bin/crictl
[INFO][2026-03-27 02:53:25] path not exist: /etc/sysctl.d/k8s.conf
[INFO][2026-03-27 02:53:25] path not exist: /etc/systemd/system/kubelet.service
[INFO][2026-03-27 02:53:25] path not exist: /etc/systemd/system/kubelet.service.d
[INFO][2026-03-27 02:53:25] path not exist: /var/lib/etcd
[INFO][2026-03-27 02:53:25] path not exist: /var/lib/kubelet
[INFO][2026-03-27 02:53:25] path not exist: /run/containerd/containerd.sock
[INFO][2026-03-27 02:53:25] path not exist: /usr/lib/systemd/system/kubelet.service.d
[INFO][2026-03-27 02:53:25] path not exist: /var/run/containerd/containerd.sock
[WARNING][2026-03-27 02:53:25] file exist: /var/run/docker.sock
[INFO][2026-03-27 02:53:25] file query completed
Query Time: 2026-03-27 02:53:25
+-------------------------------------------+---------+---------+-------+--------+-------------+
| Path | Exists | Type | Owner | Group | Permissions |
+-------------------------------------------+---------+---------+-------+--------+-------------+
| /root/.kube | Missing | Missing |-------+--------+-------------+
| /etc/kubernetes | Missing | Missing |-------+--------+-------------+
| /usr/local/bin/crictl | Missing | Missing |-------+--------+-------------+
| /etc/sysctl.d/k8s.conf | Missing | Missing |-------+--------+-------------+
| /etc/systemd/system/kubelet.service | Missing | Missing |-------+--------+-------------+
| /etc/systemd/system/kubelet.service.d | Missing | Missing |-------+--------+-------------+
| /var/lib/etcd | Missing | Missing |-------+--------+-------------+
| /var/lib/kubelet | Missing | Missing |-------+--------+-------------+
| /run/containerd/containerd.sock | Missing | Missing |-------+--------+-------------+
| /usr/lib/systemd/system/kubelet.service.d | Missing | Missing |-------+--------+-------------+
| /var/run/containerd/containerd.sock | Missing | Missing |-------+--------+-------------+
| /var/run/docker.sock | Exists | File | root | docker | Srw-rw---- |
+-------------------------------------------+---------+---------+-------+--------+-------------+
+-------------------+-------+
| Summary | Count |
+-------------------+-------+
| Total Checked | 12 |
| Total Exists | 1 |
| Total Missing | 11 |
| Total Directories | 0 |
| Total Files | 1 |
+-------------------+-------+
[INFO][2026-03-27 02:53:25] completed
程序存在性检测：
[root@localhost env]# ./envCheck check
[INFO][2026-03-27 02:54:04] envCheck tool start
[INFO][2026-03-27 02:54:04] mode: programCheck
[INFO][2026-03-27 02:54:04] program check start......
[INFO][2026-03-27 02:54:04] os: linux, arch: arm64
[INFO][2026-03-27 02:54:04] program list to check: [docker kubectl containerd]
[INFO][2026-03-27 02:54:04] check program: docker
[WARNING][2026-03-27 02:54:04] docker installed - version: Docker version 18.09.0, build d51e3ad
[INFO][2026-03-27 02:54:04] check program: kubectl
[INFO][2026-03-27 02:54:04] not install: kubectl
[INFO][2026-03-27 02:54:04] check program: containerd
[WARNING][2026-03-27 02:54:04] containerd installed - version: time="2026-03-27T02:54:04Z" level=warning msg="init error, wrong runtimeTimeout format: time: invalid duration """
containerd version:1.2.0.320.oe2203sp4 871075eb7cc979944ba2d987719cb534bbb87e5c
[INFO][2026-03-27 02:54:04] program check completed
[WARNING][2026-03-27 02:54:04] detected installed application(s): docker, containerd. Please uninstall it(them) yourself
Check Time: 2026-03-27 02:54:04
+------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
| Program | Status | Version | Path |
+------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
| docker | Installed | Docker version 18.09.0, build d51e3ad | /usr/bin/docker |
| kubectl | Not Installed | Unknown | Not found |
| containerd | Installed | time="2026-03-27T02:54:04Z" level=warning msg="init error, wrong runtimeTimeout format: time: invalid duration """
containerd version:1.2.0.320.oe2203sp4 871075eb7cc979944ba2d987719cb534bbb87e5c | /usr/bin/containerd |
+------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
+-----------------+-------+
| Summary | Count |
+-----------------+-------+
| Total Checked | 3 |
| Total Installed | 2 |
| Total Missing | 1 |
+-----------------+-------+
[INFO][2026-03-27 02:54:04] completed
在线部署
部署完毕浏览 NPU Operator 文档才发现需满足 openEuler 22.03 lts 条件，故重装
https://docs.openfuyao.cn/zh/docs/v25.12/user_guide/npu_operator.html#%E5%AE%89%E8%A3%85
- CPU:
Kunpeng-920 - OS:
openEuler 22.03 (LTS-SP4) aarch64 - Fuyao Version:
v26.03 rc3 - docker:
2:18.09.0-346.oe2403sp3
使用 openEuler 22.03 (LTS-SP4) aarch64 cloud 镜像全新安装、扩容硬盘后部署。
[root@localhost fuyao-26.3-rc3]# ./bkeadm_linux_arm64 init --otherRepo cr.openfuyao.cn/openfuyao/bke-online-installed:latest
--hostIP: 10.17.30.131
--domain: deploy.bocloud.k8s
--kubernetesPort: 36443
--imageRepoPort: 40443
--yumRepoPort: 40080
--chartRepoPort: 38080
--ntpServer: cn.pool.ntp.org:123
--runtime: containerd
--runtimeStorage: /var/lib/containerd
--clusterAPI: 1.2.1
--oFVersion: v26.03-rc.3
--versionUrl: https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/version-config/
--enableNTP: true
--agentHealthPort: 58080
--otherRepo: cr.openfuyao.cn/openfuyao/bke-online-installed:latest
Confirm the parameters, press Y to continue N will exit. [Y/N]? y
[bke][2026-03-27 02:57:16][INFO] BKE initialize ...
[bke][2026-03-27 02:57:16][INFO] HOSTNAME: localhost
[bke][2026-03-27 02:57:16][INFO] PLATFORM: openeuler
[bke][2026-03-27 02:57:16][INFO] Version: 22.03
[bke][2026-03-27 02:57:16][INFO] KERNEL: 5.10.0-216.0.0.115.oe2203sp4.aarch64
[bke][2026-03-27 02:57:16][INFO] GOOS: linux
[bke][2026-03-27 02:57:16][INFO] ARCH: arm64
[bke][2026-03-27 02:57:16][INFO] CPU: 8
[bke][2026-03-27 02:57:16][INFO] MEMORY: 16G
[bke][2026-03-27 02:57:16][INFO] BKE Console: ENABLED
[bke][2026-03-27 02:57:16][INFO] BKE initialize environment check...
[bke][2026-03-27 02:57:16][WARN] Client authentication enabled but no credentials provided
2026-03-27T02:57:16.485Z info infrastructure/infrastructure.go:53 The docker client is ready.
[bke][2026-03-27 02:57:16][INFO] set up the host machine zone
[bke][2026-03-27 02:57:16][INFO] set ntp server
Fri Mar 27 02:57:15 AM CST 2026
[bke][2026-03-26 18:57:15][INFO] config local source
[bke][2026-03-26 18:57:15][INFO] 在线模式：domain：deploy.bocloud.k8s 绑定到默认IP：10.17.30.131
[bke][2026-03-26 18:57:49][INFO] iptables -V output: iptables v1.8.7 (legacy)
[bke][2026-03-26 18:57:49][INFO] workDir /bke mod permission success
[bke][2026-03-26 18:57:49][ERROR] Failed to start the container service, open /bke/mount/source_registry/files: no such file or directory最新版似乎修改了命令参数，使用最新参数正常。
[root@stl-fuyao-2603 fuyao-26.3-rc3]# ./bkeadm_linux_arm64 init --onlineImage cr.openfuyao.cn/openfuyao/bke-online-installed:latest
--hostIP: 10.17.30.131
--domain: deploy.bocloud.k8s
--kubernetesPort: 36443
--imageRepoPort: 40443
--yumRepoPort: 40080
--chartRepoPort: 38080
--ntpServer: cn.pool.ntp.org:123
--runtime: containerd
--runtimeStorage: /var/lib/containerd
--clusterAPI: 1.2.1
--oFVersion: v26.03-rc.3
--versionUrl: https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/version-config/
--enableNTP: true
--agentHealthPort: 58080
--onlineImage: cr.openfuyao.cn/openfuyao/bke-online-installed:latest
Confirm the parameters, press Y to continue N will exit. [Y/N]? y
[bke][2026-03-27 11:36:51][INFO] BKE initialize ...
[bke][2026-03-27 11:36:51][INFO] HOSTNAME: stl-fuyao-2603
[bke][2026-03-27 11:36:51][INFO] PLATFORM: openeuler
[bke][2026-03-27 11:36:51][INFO] Version: 22.03
[bke][2026-03-27 11:36:51][INFO] KERNEL: 5.10.0-216.0.0.115.oe2203sp4.aarch64
[bke][2026-03-27 11:36:51][INFO] GOOS: linux
[bke][2026-03-27 11:36:51][INFO] ARCH: arm64
[bke][2026-03-27 11:36:51][INFO] CPU: 8
[bke][2026-03-27 11:36:51][INFO] MEMORY: 16G
[bke][2026-03-27 11:36:51][INFO] BKE Console: ENABLED
[bke][2026-03-27 11:36:51][INFO] BKE initialize environment check...
[bke][2026-03-27 11:36:51][WARN] Client authentication enabled but no credentials provided
[bke][2026-03-27 11:36:51][INFO] set up the host machine zone
[bke][2026-03-27 11:36:51][INFO] set ntp server
Fri Mar 27 11:36:51 AM CST 2026
[bke][2026-03-27 11:36:51][INFO] config local source
[bke][2026-03-27 11:36:51][INFO] 在线模式：domain：deploy.bocloud.k8s 绑定到默认IP：10.17.30.131
[bke][2026-03-27 11:36:51][WARN] 无法解析私有仓库地址，跳过CA证书配置
[bke][2026-03-27 11:36:51][INFO] Using client certificate authentication(CA only)
[bke][2026-03-27 11:36:51][INFO] Download source file...
[bke][2026-03-27 11:36:53][INFO] Writing data...
...
[bke][2026-03-27 13:48:19][INFO] containerd sandbox image: hub.oepkgs.net/openfuyao/pause:3.9
[bke][2026-03-27 13:48:19][INFO] Created offline special hosts.toml: /etc/containerd/certs.d/127.0.0.1:40443/hosts.toml
[bke][2026-03-27 13:48:19][INFO] wait for containerd to start
[bke][2026-03-27 13:48:19][INFO] Waiting for containerd to be ready
...
成功解压： /opt/cni/bin/vrf , 共处理了 4007138 个字符
[bke][2026-03-27 13:48:20][INFO] Start the base dependency service
[bke][2026-03-27 13:48:21][INFO] Image hub.oepkgs.net/openfuyao/registry:2.8.1 is downloading
[bke][2026-03-27 13:48:29][INFO] Wait for the container mirroring service to start...
[bke][2026-03-27 13:48:34][INFO] The container mirroring service is started by containerd.
[bke][2026-03-27 13:48:34][INFO] Image hub.oepkgs.net/openfuyao/nginx:1.23.0-alpine is downloading
[bke][2026-03-27 13:48:43][INFO] Wait for the container yum service to start...
[bke][2026-03-27 13:48:48][INFO] The container yum service is started.
[bke][2026-03-27 13:48:48][INFO] Image hub.oepkgs.net/openfuyao/helm/chartmuseum:v0.16.2 is downloading
[bke][2026-03-27 13:49:05][INFO] Wait for the chart mirroring service to start...
[bke][2026-03-27 13:49:10][INFO] The chart mirroring service is started.
[bke][2026-03-27 13:49:10][INFO] Image hub.oepkgs.net/openfuyao/openebs/nfs-server-alpine:0.9.0 is downloading
[bke][2026-03-27 13:49:17][INFO] Wait for the nfs mirroring service to start...
[bke][2026-03-27 13:49:22][INFO] The nfs mirroring service is started.
[bke][2026-03-27 13:49:22][INFO] Image hub.oepkgs.net/openfuyao/rancher/k3s:v1.25.16-k3s4 is downloading
2026-03-27T13:50:07.914+0800 info k3s/k3s.go:314 params: onlineImage=cr.openfuyao.cn/openfuyao/bke-online-installed:latest otherRepo=, otherRepoIp=, hostIP=10.17.30.131, imageRepo=deploy.bocloud.k8s, imageRepoPort=40443, kubernetesPort=36443
[bke][2026-03-27 13:50:07][INFO] Start the local Kubernetes cluster...
[bke][2026-03-27 13:50:10][ERROR] Failed to copy kubectl from the container
[bke][2026-03-27 13:50:10][ERROR] Failed to start kubernetes exit status 1
[bke][2026-03-27 13:50:10][ERROR] Failed to start cluster API, exit status 1似乎启动失败，居然是因为 tar 没有安装？
[root@stl-fuyao-2603 fuyao-26.3-rc3]# nerdctl cp kubernetes:/bin/k3s /tmp/test-k3s
FATA[0000] unable to copy: failed to find `tar` binary 安装 tar 后解决，顺利部署。
Npu Operator
NAMESPACE NAME READY STATUS RESTARTS AGE
cluster-system bkeagent-deployer-h2flc 1/1 Running 0 65m
ingress-nginx ingress-nginx-admission-create-xqwmf 0/1 Completed 0 62m
ingress-nginx ingress-nginx-admission-patch-4dcjr 0/1 Completed 1 62m
ingress-nginx ingress-nginx-controller-p5nrm 1/1 Running 0 62m
kube-system ascend-device-plugin-xtfkb 0/1 CrashLoopBackOff 7 (4m34s ago) 16m
kube-system calico-kube-controllers-6d75d78f5d-tfjvf 1/1 Running 0 65m
kube-system calico-node-xtd99 1/1 Running 0 65m
kube-system coredns-6c6fdbdb66-24cn4 1/1 Running 0 65m
kube-system coredns-6c6fdbdb66-w29mt 1/1 Running 1 (63m ago) 65m
kube-system etcd-master1 1/1 Running 0 64m
kube-system kube-apiserver-master1 1/1 Running 0 55m
kube-system kube-controller-manager-master1 1/1 Running 1 (55m ago) 65m
kube-system kube-proxy-xxgz4 1/1 Running 0 65m
kube-system kube-scheduler-master1 1/1 Running 1 (55m ago) 65m
kube-system metrics-server-586f979f47-4b6fz 1/1 Running 0 54m
mindx-dl ascend-operator-manager-5d4d89f675-tbjpv 0/1 Pending 0 15m
mindx-dl clusterd-5588c5dc88-2fd5g 1/1 Running 0 16m
mindx-dl resilience-controller-8686bbd76f-jv5lp 1/1 Running 0 16m
monitoring alertmanager-main-0 2/2 Running 0 51m
monitoring alertmanager-main-1 2/2 Running 0 51m
monitoring alertmanager-main-2 2/2 Running 0 51m
monitoring blackbox-exporter-6d6fbbfc96-cl6sg 3/3 Running 0 56m
monitoring kube-state-metrics-677558db89-9rtbl 3/3 Running 0 56m
monitoring node-exporter-4mzzl 2/2 Running 0 56m
monitoring prometheus-k8s-0 2/2 Running 0 51m
monitoring prometheus-k8s-1 2/2 Running 0 51m
monitoring prometheus-operator-5cb64c846d-8m55t 2/2 Running 0 56m
npu ascend-runtime-containerd-pg5q4 1/1 Running 0 16m
npu npu-driver-66dqf 0/1 Init:0/1 0 103s
npu npu-feature-discovery-rr68r 1/1 Running 0 17m
npu npu-operator-5858d99c89-lgd24 1/1 Running 0 17m
npu npu-operator-node-feature-discovery-gc-5cf8bc768d-6w4b4 1/1 Running 0 17m
npu npu-operator-node-feature-discovery-master-5985b5cfcd-swdqs 1/1 Running 0 17m
npu npu-operator-node-feature-discovery-worker-q5ct6 1/1 Running 0 17m
openfuyao-system-controller modify-manifests-master1-sjxrv 0/1 Completed 0 55m
openfuyao-system-controller openfuyao-system-controller-8444679b95-jv8jg 1/1 Running 0 65m
openfuyao-system application-management-service-75799d4dd6-8vm9w 2/2 Running 0 55m
openfuyao-system console-service-84bbd85575-grtwx 1/1 Running 0 55m
openfuyao-system console-website-855c9d8f65-8btkl 1/1 Running 0 61m
openfuyao-system local-harbor-chartmuseum-7f96745849-d7vzk 1/1 Running 0 56m
openfuyao-system local-harbor-core-5f847798b8-khkf4 1/1 Running 1 (51m ago) 56m
openfuyao-system local-harbor-database-0 1/1 Running 0 56m
openfuyao-system local-harbor-jobservice-7d67f4f887-svklq 1/1 Running 3 (47m ago) 56m
openfuyao-system local-harbor-nginx-6449749746-pbzkc 1/1 Running 0 56m
openfuyao-system local-harbor-portal-78bf65c9-rqmk6 1/1 Running 0 56m
openfuyao-system local-harbor-redis-0 1/1 Running 0 56m
openfuyao-system local-harbor-registry-65884895bf-ff2wd 2/2 Running 0 56m
openfuyao-system marketplace-service-5c79cbcbfc-swg9r 2/2 Running 0 55m
openfuyao-system monitoring-service-79fc57c6b4-47vbm 2/2 Running 0 56m
openfuyao-system oauth-server-68b6655d95-c8pzx 1/1 Running 0 54m
openfuyao-system oauth-webhook-6995d46758-g47xn 1/1 Running 0 54m
openfuyao-system plugin-management-service-84bfcd6565-2bmz2 2/2 Running 0 54m
openfuyao-system user-management-operator-8d79bd8b8-jr4sk 1/1 Running 0 54m
openfuyao-system web-terminal-service-6d858d974-hpjw6 1/1 Running 0 54m
volcano-system volcano-controllers-6ffb787f8d-mxxph 1/1 Running 0 16m
volcano-system volcano-scheduler-867f9784bb-vsswv 0/1 Pending 0 16m
kube-system ascend-device-plugin-sbxr7 0/1 CrashLoopBackOff 6 (100s ago) 9m57s[root@master1 ~]# kubectl -n kube-system logs ascend-device-plugin-sbxr7
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
[INFO] 2026/03/27 07:27:47.189809 1 hwlog/api.go:108 devicePlugin.log's logger init success
[INFO] 2026/03/27 07:27:47.190348 1 main.go:187 ascend device plugin starting and the version is v6.0.0_linux-aarch64
[INFO] 2026/03/27 07:27:47.190416 1 main.go:188 ascend device plugin starting scene is center
2026/03/27 07:27:47 command exec failed, exit status 1
[ERROR] 2026/03/27 07:27:47.192892 1 devmanager/devmanager.go:95 deviceManager init failed, prepare dcmi failed, err: cannot found valid driver lib, fromEnv: lib path is invalid, [], fromLdCmd: can't find valid lib: EOF
[ERROR] 2026/03/27 07:27:47.192970 1 main.go:203 init devmanager failed, err: auto init failed, err: get chip info failed, err: device Manager is nil, may encounter an exception during initialization. You can check the system log to confirm驱动安装失败：
[root@master1 ~]# kubectl -n npu logs -f npu-driver-lxv5c -c npu-driver-installer
Checking if /mnt/usr/local/sbin/npu-smi exists...
master1 is not an option, please use -h to view help
[2026-03-27 15:55:00, [INFO] No operation specified, default install operation on node: --
[2026-03-27 15:55:00, [INFO] install npu-driver
[2026-03-27 15:55:00, [INFO] copy npu-install to host
[2026-03-27 15:55:00, [INFO] copy npu-install to host success
[2026-03-27 15:55:00,303677070] [INFO] Install dependency packages
[2026-03-27 15:55:00,305026980] [INFO] Using yum for package installation
5 files removed
repo 4.3 MB/s | 33 kB 00:00
Metadata cache created.
[2026-03-27 15:55:01,143913450] [INFO] jq is already installed, skipping...
[2026-03-27 15:55:01,150749780] [INFO] wget is already installed, skipping...
[2026-03-27 15:55:01,156894190] [INFO] Installing unzip...
No match for argument: unzip
Error: Unable to find a match: unzip
[2026-03-27 15:55:01,599984660] [FATAL] Failed to install unzip using yum
[2026-03-27 15:55:01, [FATAL] install failed: --
因为主机缺少 unizp 导致失败，但是该容器没有任何异常，无法感知错误。
从清单可以看到 ascend-device-plugin-sbxr7 容器一直由于找不到驱动异常退出，而实际原因是 npu-driver-66dqf pod 没有正常安装驱动，但没有退出，反而正常运行。模糊了实际错误点。
安装 unzip 后正常：
[root@master1 ~]# kubectl -n npu logs -f npu-driver-66dqf -c npu-driver-installer
Checking if /mnt/usr/local/sbin/npu-smi exists...
master1 is not an option, please use -h to view help
[2026-03-27 15:57:58, [INFO] No operation specified, default install operation on node: --
[2026-03-27 15:57:58, [INFO] install npu-driver
[2026-03-27 15:57:58, [INFO] copy npu-install to host
[2026-03-27 15:57:58, [INFO] copy npu-install to host success
[2026-03-27 15:57:58,069791170] [INFO] Install dependency packages
[2026-03-27 15:57:58,070929430] [INFO] Using yum for package installation
48 files removed
repo 5.1 MB/s | 33 kB 00:00
OS 67 MB/s | 3.3 MB 00:00
everything 76 MB/s | 17 MB 00:00
EPOL 65 MB/s | 4.7 MB 00:00
debuginfo 69 MB/s | 3.9 MB 00:00
source 59 MB/s | 1.8 MB 00:00
update 79 MB/s | 71 MB 00:00 可以看到花费了大量时间在 init 阶段，这样才是正常的：
npu npu-driver-66dqf 0/1 Init:0/1 0 3m51s虚拟机运行似乎会有这个错误
kubectl -n kube-system delete pod ascend-device-plugin-xtfkb
[Driver] [2026-03-27 16:03:40] [INFO]upgradePercentage:100%
[Driver] [2026-03-27 16:03:42] [INFO]Driver package installed successfully! The new version takes effect immediately.
[Driver] [2026-03-27 16:03:42] [INFO]End time: 2026-03-27 16:03:42
[Firmware] [2026-03-27 16:03:43] [INFO]Start time: 2026-03-27 16:03:43
[Firmware] [2026-03-27 16:03:43] [INFO]LogFile: /var/log/ascend_seclog/ascend_install.log
[Firmware] [2026-03-27 16:03:43] [INFO]OperationLogFile: /var/log/ascend_seclog/operation.log
[Firmware] [2026-03-27 16:03:43] [WARNING]Do not power off or restart the system during the installation/upgrade
[Firmware] [2026-03-27 16:03:43] [ERROR]Not a physical-machine, firmware upgrade does not support.
[Firmware] [2026-03-27 16:03:43] [INFO]End time: 2026-03-27 16:03:43 但驱动等已经部署完毕：
[root@master1 ~]# npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc3 Version: 24.1.rc3 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 176 310P3 | OK | NA 56 0 / 0 |
| 0 0 | 0000:00:16.0 | 0 1838 / 21527 |
+===============================+=================+======================================================+
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===============================+=================+======================================================+
| No running processes found in NPU 176 |
+===============================+=================+======================================================+但是这个容器依然无法正常运行：
[root@master1 ~]# kubectl -n kube-system logs -f -l name=ascend-device-plugin-ds -c device-plugin-01
[INFO] 2026/03/27 08:19:41.677826 1 hwlog/api.go:108 devicePlugin.log's logger init success
[INFO] 2026/03/27 08:19:41.678051 1 main.go:187 ascend device plugin starting and the version is v6.0.0_linux-aarch64
[INFO] 2026/03/27 08:19:41.678116 1 main.go:188 ascend device plugin starting scene is center
[INFO] 2026/03/27 08:19:41.900653 1 devmanager/devmanager.go:104 the dcmi version is 24.1.rc3
[ERROR] 2026/03/27 08:19:41.900744 1 devmanager/devmanager.go:211 get error card quantity: 0
[ERROR] 2026/03/27 08:19:41.900780 1 devmanager/devmanager.go:195 get card list failed for init
[ERROR] 2026/03/27 08:19:41.900828 1 main.go:203 init devmanager failed, err: auto init failed, err: get card list failed for init部署后防火墙问题
另外发现 openEuler iptables 默认，还是 openFuyao 默认配置，iptables 默认 FORWARD 为 Drop，会导致部署后无法访问。
[root@stl-fuyao-2603 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
CNI-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0 /* CNI firewall plugin rules (ingressPolicy: same-bridge) */
CNI-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* CNI firewall plugin rules */
Chain OUTPUT (policy ACCEPT)
target prot opt source destination 需 配置解决
iptables -P FORWARD ACCEPT部署时软件源问题
[root@master1 ~]# cp /etc/yum.repos.d/b
bak/ bke.repo默认移除系统自带软件源，部署后无法再安装软件，需自行配置。
能否通过更加灵活的方式，如配置优先级的方式来规避直接移除软件源。
压缩工具检查
建议参考这种：
root@hosthatch-us1:~# sudo -v ; curl https://rclone.org/install.sh | sudo bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4734 100 4734 0 0 8458 0 --:--:-- --:--:-- --:--:-- 8468
None of the supported tools for extracting zip archives (unzip 7z busybox) were found. Please install one of them and try again.带有明确错误说明。