kubelet 远程调试
tags: golang,container
kubelet 远程调试
1. kubelet启动命令分析
kubelet是一个systemd服务，以使用Kubeadm工具安装的v1.23.4 k8s集群为例，该服务的配置文件路径为/etc/systemd/system/kubelet.service.d/10-kubeadm.conf, 内容如下:
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
以我的测试环境为例，执行ps -ef |grep /usr/bin/kubelet, 可见kubelet启动的完整命令如下：
/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.6
如果需要修改kubelet命令，可以关闭服务后使用相同参数启动。或修改systemd配置文件后重启kubelet服务。
2. 编译kubelet
根据k8s makefile 源码分析，kubelet编译命令如下：
https://github.com/kubernetes/kubernetes/blob/v1.22.4/hack/lib/golang.sh#L679
kube::golang::build_some_binaries() {
...
go install "${build_args[@]}" "$@"
...
}
其中GOLDFLAGS, GOGCFLAGS配置如下：
https://github.com/kubernetes/kubernetes/blob/v1.22.4/hack/lib/golang.sh#L797-L799
kube::golang::build_binaries() {
...
goldflags="${GOLDFLAGS=-s -w} $(kube::version::ldflags)"
goasmflags="-trimpath=${KUBE_ROOT}"
gogcflags="${GOGCFLAGS:-} -trimpath=${KUBE_ROOT}"
...
}
为了保留尽可能多的调试信息，我们需要重新设置这两个编译参数，所以编译kubelet的命令应为
git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
git checkout v1.22.4
./build/shell.sh
make generated_files
make -o generated_files kubelet KUBE_BUILD_PLATFORMS=linux/amd64 GOLDFLAGS="" GOGCFLAGS="all=-N -l"
编译完成后，kubelet二进制文件位于_output/bin/kubelet
3. delve介绍
delve是一个用于Go编程语言的调试器。尽管我们也可以使用gdb调试go语言程序, 但在调试用标准工具链构建的Go程序时，Delve是GDB更好的替代品。它比GDB更能理解Go的运行时、数据结构和表达式。
可以使用如下命令安装dlv:
go install github.com/go-delve/delve/cmd/dlv@latest
使用如下命令使用dlv进行调试:
dlv exec ./hello -- server --config conf/config.toml
以kubelet为例，使用dlv命令行调试的过程如下：
root@st0n3-host:~# dlv exec /usr/bin/kubelet -- --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.6
Type 'help' for list of commands.
(dlv) b main.main
Breakpoint 1 set at 0x502e086 for main.main() _output/dockerized./cmd/kubelet/kubelet.go:39
(dlv) c
> main.main() _output/dockerized./cmd/kubelet/kubelet.go:39 (hits goroutine(1):1 total:1) (PC: 0x502e086)
34: _ "k8s.io/component-base/metrics/prometheus/restclient"
35: _ "k8s.io/component-base/metrics/prometheus/version" // for version metric registration
36: "k8s.io/kubernetes/cmd/kubelet/app"
37: )
38:
=> 39: func main() {
40: command := app.NewKubeletCommand()
41:
42: // kubelet uses a config file and does its own special
43: // parsing of flags and that config file. It initializes
44: // logging after it is done with that. Therefore it does
(dlv)
4. GoLand远程调试kubelet
我们当然可以使用上文描述的命令行形式进行调试，但kubernetes代码量巨大，使用IDE会更方便。
点击调试按钮左侧的Edit Configurations按钮，配置dlv的地址和端口：
使用IDE提示的命令启动kubelet，或将其配置到systemd服务中后重启服务：
root@st0n3-host:~# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
...
ExecStart=/usr/bin/dlv --listen=:10086 --headless=true --api-version=2 --accept-multiclient exec /usr/bin/kubelet -- $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
root@st0n3-host:~# systemctl daemon-reload
root@st0n3-host:~# systemctl restart kubelet.service
此时kubelet命令实际还未真正启动，在GoLand中运行刚刚添加的配置，连接上dlv后，kubelet才会运行。
下好断点，点击debug按钮，我们就可以在IDE中对kubelet进行调试了。
5. 其他容器软件调试命令
5.1 runc
编译
make shell
make EXTRA_FLAGS='-gcflags="all=-N -l"'
调试
mv /usr/bin/runc /usr/bin/runc.bak
cat <<EOF > /usr/bin/runc
#!/bin/bash
dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec /usr/bin/runc.debug -- $*
chmod +x /usr/bin/runc
5.2 docker-cli
编译
修改scripts/build/binary中的编译命令如下：删除LDFLAGS，添加gcflags
root@st0n3:~/cli# git diff
diff --git a/scripts/build/binary b/scripts/build/binary
index e4c5e12a6b..155528e501 100755
--- a/scripts/build/binary
+++ b/scripts/build/binary
@@ -74,7 +74,7 @@ fi
echo "Building $GO_LINKMODE $(basename "${TARGET}")"
export GO111MODULE=auto
-
-go build -o "${TARGET}" -tags "${GO_BUILDTAGS}" --ldflags "${LDFLAGS}" ${GO_BUILDMODE} "${SOURCE}"
+go build -o "${TARGET}" -tags "${GO_BUILDTAGS}" -gcflags="all=-N -l" ${GO_BUILDMODE} "${SOURCE}"
ln -sf "$(basename "${TARGET}")" "$(dirname "${TARGET}")/docker"
make -f docker.Makefile shell
make binary
调试
cat <<EOF > docker.debug
#!/bin/bash
dlv --listen=:2344 --headless=true --api-version=2 --accept-multiclient exec ./docker-cli.debug -- $*
EOF
chmod +x docker.debug
5.3 dockerd
编译
修改hack/make/.binary文件中的编译命令
root@st0n3:~/moby# git diff
diff --git a/hack/make/.binary b/hack/make/.binary
index d56e3f3126..3e23865c81 100644
--- a/hack/make/.binary
+++ b/hack/make/.binary
@@ -81,11 +81,11 @@ hash_files() {
echo "Building: $DEST/$BINARY_FULLNAME"
echo "GOOS=\"${GOOS}\" GOARCH=\"${GOARCH}\" GOARM=\"${GOARM}\""
- go build \
+ set -x
+ go build -gcflags "all=-N -l" \
-o "$DEST/$BINARY_FULLNAME" \
"${BUILDFLAGS[@]}" \
-ldflags "
- $LDFLAGS
$LDFLAGS_STATIC_DOCKER
$DOCKER_LDFLAGS
" \
make BIND_DIR=. shell
hack/make.sh binary
调试
/root/go/bin/dlv --listen=:2343 --headless=true --api-version=2 --accept-multiclient exec /usr/bin/dockerd.debug -- -D -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock
5.4 containerd
编译
make GODEBUG=1 bin/containerd
或修改Makefile后make bin/containerd
root@st0n3:~/containerd# git diff
diff --git a/Makefile b/Makefile
index 266aef335..be5381a33 100644
--- a/Makefile
+++ b/Makefile
@@ -63,7 +63,7 @@ else
endif
ifndef GODEBUG
- EXTRA_LDFLAGS += -s -w
+ # EXTRA_LDFLAGS += -s -w
DEBUG_GO_GCFLAGS :=
DEBUG_TAGS :=
else
@@ -231,7 +231,7 @@ FORCE:
define BUILD_BINARY
@echo "$(WHALE) $@"
-@$(GO) build ${DEBUG_GO_GCFLAGS} ${GO_GCFLAGS} ${GO_BUILD_FLAGS} -o $@ ${GO_LDFLAGS} ${GO_TAGS} ./$<
+@$(GO) build -gcflags="all=-N -l" ${DEBUG_GO_GCFLAGS} ${GO_GCFLAGS} ${GO_BUILD_FLAGS} -o $@ ${GO_LDFLAGS} ${GO_TAGS} ./$<
endef
# Build a binary from a cmd.
调试
dlv --listen=:2346 --headless=true --api-version=2 --accept-multiclient exec /usr/bin/containerd.debug
5.5 buildx
编译
$ sed -i 's/go build/go build -gcflags="all=-N -l"/g' hack/build
$ sed -i 's/${GO_EXTRA_LDFLAGS}//g' hack/build
$ sed -i 's/-trimpath//g' hack/build
$ make binaries
docker buildx bake binaries
[+] Building 7.3s (21/21) FINISHED
...
=> => copying files 76.04MB
$ ls -lah bin/build/buildx
-rwxr-xr-x 1 root root 73M Feb 14 12:52 bin/build/buildx
调试
# cat docker-buildx.debug
#!/bin/bash
CLI="./docker-buildx/buildx-linux-amd64-v0.10.2-debug"
dlv --log-dest=/dev/null --listen=:2347 --headless=true --api-version=2 --accept-multiclient exec $CLI -- $*