tags: codeql,container

如何为docker(moby)项目创建codeql数据库

1. 背景

lgtm上的数据库只提供了最新版本供下载，而且对moby项目创建的数据库可能是错误的。

https://lgtm.com/projects/g/moby/moby/logs/languages/lang:go

在build日志我们可以看到，lgtm的build方式和docker的官方build方式并不一致。 另外，对不同的运行环境，docker有不同的编译方式，包括deb,static等，可能一份数据库并不够。那么，如何正确地为docker(moby)项目编译codeql数据库呢？

[2021-06-29 07:19:26] [build-stderr] 2021/06/29 07:19:26 GOPATH set to /opt/src/root.
[2021-06-29 07:19:26] [build-stderr] 2021/06/29 07:19:26 Makefile found.
[2021-06-29 07:19:26] [build-stderr] 2021/06/29 07:19:26 Trying build command make []
[2021-06-29 07:19:26] [build-stdout] mkdir -p autogen
[2021-06-29 07:19:26] [build-stderr] /bin/sh: 1: docker: not found
[2021-06-29 07:19:26] [build-stderr] make: *** [Makefile:184: binary] Error 127
[2021-06-29 07:19:26] [build-stdout] docker build  --build-arg=GO_VERSION  -f "Dockerfile" --output=bundles/ --target=binary --build-arg VERSION --build-arg DOCKER_GITCOMMIT --build-arg PRODUCT --build-arg PLATFORM --build-arg DEFAULT_PRODUCT_LICENSE .
[2021-06-29 07:19:26] [build-stderr] 2021/06/29 07:19:26 Running /usr/bin/make failed, continuing anyway: exit status 2
[2021-06-29 07:19:26] [build-stderr] 2021/06/29 07:19:26 Build failed, continuing to install dependencies.
[2021-06-29 07:19:26] [build-stderr] 2021/06/29 07:19:26 Installing dependencies using `go get -v ./...`.
[2021-06-29 07:19:27] [build-stderr] github.com/docker/docker (download)
[2021-06-29 07:19:46] [build-stderr] package io/fs: unrecognized import path "io/fs": import path does not begin with hostname
[2021-06-29 07:19:48] [build-stderr] 2021/06/29 07:19:48 Running /usr/bin/go failed, continuing anyway: exit status 1
[2021-06-29 07:19:48] [build-stderr] 2021/06/29 07:19:48 Running extractor command '/opt/dist/go/tools/linux64/go-extractor [./...]' from directory '/opt/src/root/src/github.com/moby/moby'.
[2021-06-29 07:19:48] [build-stderr] 2021/06/29 07:19:48 Max threads set to 2
[2021-06-29 07:19:48] [build-stderr] 2021/06/29 07:19:48 Build flags: ''; patterns: './...'
[2021-06-29 07:19:48] [build-stderr] 2021/06/29 07:19:48 Go module mode disabled.
[2021-06-29 07:19:48] [build-stderr] 2021/06/29 07:19:48 Running packages.Load.
[2021-06-29 07:20:09] [build-stderr] 2021/06/29 07:20:09 Done running packages.Load.
[2021-06-29 07:20:09] [build-stderr] 2021/06/29 07:20:09 Extracting universe scope.
[2021-06-29 07:20:09] [build-stderr] 2021/06/29 07:20:09 Done extracting universe scope.
[2021-06-29 07:20:09] [build-stderr] 2021/06/29 07:20:09 Processing package github.com/moby/moby/api.
[2021-06-29 07:20:09] [build-stderr] 2021/06/29 07:20:09 Extracting types for package github.com/moby/moby/api.
[2021-06-29 07:20:09] [build-stderr] 2021/06/29 07:20:09 Warning: encountered errors extracting package `github.com/moby/moby/api`:
[2021-06-29 07:20:09] [build-stderr] 2021/06/29 07:20:09   -: code in directory /opt/src/root/src/github.com/moby/moby/api expects import "github.com/docker/docker/api"

2. How

要解答这个问题，首先要理解codeql创建数据库的过程。

// TODO

3. codeql-go-vendor

https://github.com/ssst0n3/codeql-go-vendor

3.1 WHY?

官方的go-extractor 只支持go mod模式。

对于使用纯vendor模式的go项目，是不优雅的。

例如，下面的codeql查询结果是空的。

https://lgtm.com/query/8418405387172037343/

import go

from CallExpr e
where e.getTarget().getName()="panic"
select e

但是，它本应该可以找出下列会触发panic的函数:

https://github.com/ssst0n3/go-vendor-test/blob/main/vendor/st0n3/st0n3.go

package st0n3

func Crash() {
    panic("crash")
}

3.2 如何使用

  1. 从 releases 页面 下载 go-extractor (注意版本需要与你的codeql-cli一致)。
  2. 复制你刚刚下载的go-extractor，覆盖$HOME/codeql-home/codeql/go/tools/linux64/go-extractor。
  3. 然后就像你往常创建codeql数据库一样。

3.3 Build

cd extractor/cli/go-extractor/
go build

3.4 原理

codeql官方的go-extractor跳过了vendor目录, 我只是又加了回来。

https://github.com/github/codeql-go/blob/codeql-cli/v2.6.2/extractor/extractor.go#L171-L182

noExtractRe := regexp.MustCompile(`.*(^|` + sep + `)(\.\.|vendor)($|` + sep + `).*`)

// extract AST information for all packages
packages.Visit(pkgs, func(pkg *packages.Package) bool {
    return true
}, func(pkg *packages.Package) {
    for root, _ := range wantedRoots {
    relDir, err := filepath.Rel(root, pkgDirs[pkg.PkgPath])
    if err != nil || noExtractRe.MatchString(relDir) {
        // if the path can't be made relative or matches the noExtract regexp skip it
        continue
    }
    ...

4. 编译

4.1 准备编译环境

see how to build docker

git clone https://github.com/moby/moby
cd moby
make BIND_DIR=. shell

4.2 安装 codeql

see https://codeql.github.com/docs/codeql-cli/getting-started-with-the-codeql-cli/

mkdir -p /home/codeql-home
cd /home/codeql-home
wget "https://github.com/github/codeql-cli-binaries/releases/download/v2.6.2/codeql-linux64.zip"
git clone https://github.com/github/codeql codeql-repo
git clone https://github.com/github/codeql-go/
apt-get install unar
unar codeql-linux64.zip
alias codeql=/home/codeql-home/codeql/codeql
codeql resolve languages

4.3 安装 codeql-go-vendor

wget -O /home/codeql-home/codeql/go/tools/linux64/go-extractor "https://github.com/ssst0n3/codeql-go-vendor/releases/download/codeql-cli%2Fv2.6.2-vendor/go-extractor"

4.4 创建codeql数据库

root@cf47b8c234b1:/go/src/github.com/docker/docker# mv Makefile Makefile.bak
root@cf47b8c234b1:/go/src/github.com/docker/docker# cat >build.sh << 'EOF'
#!/bin/bash
hack/make.sh binary
EOF
root@cf47b8c234b1:/go/src/github.com/docker/docker# chmod +x build.sh
root@cf47b8c234b1:/go/src/github.com/docker/docker# GO111MODULE=off codeql database create -l go docker-with-vendor
...