---

tags: 源码分析,container

k8s makefile 源码分析

本文编写时，最新release为v1.22.4, 因此下文分析的代码均为v1.22.4分支的代码。

阅读此文前，读者应对makefile语法有基础的理解。

0. 背景

根据build文档，我们只需要执行build/run.sh make即可完成k8s的编译。build/run.sh的功能是启动一个基于docker的编译环境，make即是执行具体编译的动作。我们希望知道，make命令在编译时，究竟执行了什么，因此需要对Makefile的源码进行分析。

项目根目录的Makefile文件是指向build/root/Makefile的软链接。

# ls -lah Makefile
lrwxrwxrwx 1 root root 19 Nov 23 11:41 Makefile -> build/root/Makefile

---

故Makefile的位置为: https://github.com/kubernetes/kubernetes/blob/v1.22.4/build/root/Makefile

1. k8s makefile 目标分析

在Makefile文件中，我们看到有一个help子命令：

https://github.com/kubernetes/kubernetes/blob/v1.22.4/build/root/Makefile#L567

.PHONY: help
ifeq ($(PRINT_HELP),y)
help:
	@echo "$$HELP_INFO"
else
help:
	hack/make-rules/make-help.sh
endif

---

执行make help将会打印出所有子命令及帮助信息, 因为篇幅问题我们不列出全部。这帮助我们了解到all是第一个目标，执行make或make all可以实现对k8s源码的编译：

# make help
--------------------------------------------------------------------------------
all
# Build code.
#
# Args:
#   WHAT: Directory names to build.  If any of these directories has a 'main'
#     package, the build will produce executable files under _output/bin.
#     If not specified, "everything" will be built.
#   GOFLAGS: Extra flags to pass to 'go' when building.
#   GOLDFLAGS: Extra linking flags passed to 'go' when building.
#   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
#
# Example:
#   make
#   make all
#   make all WHAT=cmd/kubelet GOFLAGS=-v
#   make all GOLDFLAGS=""
#     Note: Specify GOLDFLAGS as an empty string for building unstripped binaries, which allows
#           you to use code debugging tools like delve. When GOLDFLAGS is unspecified, it defaults
#           to "-s -w" which strips debug information. Other flags that can be used for GOLDFLAGS 
#           are documented at https://golang.org/cmd/link/
---------------------------------------------------------------------------------
check
# Build and run tests.
#
# Args:
#   WHAT: Directory names to test.  All *_test.go files under these
#     directories will be run.  If not specified, "everything" will be tested.
#   TESTS: Same as WHAT.
#   KUBE_COVER: Whether to run tests with code coverage. Set to 'y' to enable coverage collection.
#   GOFLAGS: Extra flags to pass to 'go' when building.
#   GOLDFLAGS: Extra linking flags to pass to 'go' when building.
#   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
#
# Example:
#   make check
#   make test
#   make check WHAT=./pkg/kubelet GOFLAGS=-v
---------------------------------------------------------------------------------
clean
# Remove all build artifacts.
#
# Example:
#   make clean
#
# TODO(thockin): call clean_generated when we stop committing generated code.
---------------------------------------------------------------------------------
...

---

目标all中定义了要执行的命令是hack/make-rules/build.sh：

https://github.com/kubernetes/kubernetes/blob/v1.22.4/build/root/Makefile#L92

.PHONY: all
ifeq ($(PRINT_HELP),y)
all:
	@echo "$$ALL_HELP_INFO"
else
all: generated_files
	hack/make-rules/build.sh $(WHAT)
endif

---

另外也留意到，有一些目标是用于编译特定文件的：

https://github.com/kubernetes/kubernetes/blob/v1.22.4/build/root/Makefile#L529-L538

# make help
...
kubemark
kube-scheduler
cloud-controller-manager
kubelet
kubectl
kubeadm
kube-proxy
importverifier
gendocs
kube-apiserver
genman
genkubedocs
genyaml
genswaggertypedocs
genutils
kubectl-convert
preferredimports
kube-controller-manager
clicheck
linkcheck
dependencycheck
# Add rules for all directories in cmd/
#
# Example:
#   make kubectl kube-proxy
...

---

这些目标，针对的makefile规则如下，也传递到了hack/make-rules/build.sh, 参数为要编译的package:

EXCLUDE_TARGET=BUILD OWNERS
CMD_TARGET = $(filter-out %$(EXCLUDE_TARGET),$(notdir $(abspath $(wildcard cmd/*/))))
.PHONY: $(CMD_TARGET)
ifeq ($(PRINT_HELP),y)
$(CMD_TARGET):
	@echo "$$CMD_HELP_INFO"
else
$(CMD_TARGET): generated_files
	hack/make-rules/build.sh cmd/$@
endif

---

2. 编译命令分析

hack/make-rules/build.sh文件内容很简短，主要是调用了kube::golang::build_binaries函数。kube::golang::build_binaries函数定义在hack/lib/golang.sh文件中，由source "${KUBE_ROOT}/hack/lib/init.sh"引入进来。

https://github.com/kubernetes/kubernetes/blob/v1.22.4/hack/make-rules/build.sh

KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/../..
KUBE_VERBOSE="${KUBE_VERBOSE:-1}"
source "${KUBE_ROOT}/hack/lib/init.sh"

kube::golang::build_binaries "$@"
kube::golang::place_bins

---

kube::golang::build_binaries函数并不简短，处理了很多参数，最终调用了kube::golang::build_binaries_for_platform函数。

https://github.com/kubernetes/kubernetes/blob/v1.22.4/hack/lib/golang.sh#L782

kube::golang::build_binaries() {
  (
    ...
    kube::golang::build_binaries_for_platform "${platform}"
    ...

---

build_binaries_for_platform根据给定的参数执行静态或非静态编译

https://github.com/kubernetes/kubernetes/blob/v1.22.4/hack/lib/golang.sh#L683

kube::golang::build_binaries_for_platform() {
    ...
    CGO_ENABLED=0 kube::golang::build_some_binaries "${statics[@]}"
    ...
    kube::golang::build_some_binaries "${nonstatics[@]}"
    ...

---

最后,终于来到了最终的编译命令, 因此如果需要自定义编译参数或命令，在此处修改是最方便的。

https://github.com/kubernetes/kubernetes/blob/v1.22.4/hack/lib/golang.sh#L679

kube::golang::build_some_binaries() {
  ...
    go install "${build_args[@]}" "$@"
  ...
}