Kubernetes 简介

Kubernetes 是谷歌开源的容器集群管理系统,是 Google 多年大规模容器管理技术 Borg 的开源版本,Kubernetes 发展非常迅速,已经成为容器编排领域的领导者,主要功能包括:

  • 基于容器的应用部署、维护和滚动升级
  • 负载均衡和服务发现
  • 跨机器和跨地区的集群调度
  • 自动伸缩
  • 无状态服务和有状态服务
  • 广泛的 Volume 支持
  • 插件机制保证扩展性

核心组件

Kubernetes 主要由以下几个核心组件组成:

  • etcd 保存了整个集群的状态;
  • apiserver 提供了资源操作的唯一入口,并提供认证、授权、访问控制、API 注册和发现等机制;
  • controller manager 负责维护集群的状态,比如故障检测、自动扩展、滚动更新等;
  • scheduler 负责资源的调度,按照预定的调度策略将 Pod 调度到相应的机器上;
  • kubelet 负责维护容器的生命周期,同时也负责 Volume(CVI)和网络(CNI)的管理;
  • Container runtime 负责镜像管理以及 Pod 和容器的真正运行(CRI);
  • kube-proxy 负责为 Service 提供 cluster 内部的服务发现和负载均衡

准备部署环境

我自己用的三台虚拟机:Centos7.6 系统做好之后需要做一些小准备:

cat /etc/hosts

1
2
3
192.168.65.101 k8s-m1
192.168.65.102 k8s-n1
192.168.65.103 k8s-n2

cat /etc/sysctl.d/k8s.conf

1
2
3
net.ipv4.ip_forward =  1 
net.bridge.bridge-nf-call-ip6tables = 1 
net.bridge.bridge-nf-call-iptables = 1
1
2
3
modprobe bridge && modprobe br_netfilter
lsmod | grep bridge
sysctl -p /etc/sysctl.d/k8s.conf
1.关闭Swap

swapoff -a && sysctl -w vm.swappiness=0

2.修改/etc/selinux

sed -i “s#SELINUX=enforcing#SELINUX=disabled#” /etc/selinux/config

3.生成密钥,上传公钥到其他节点,方便后面免密

ssh-kyeget -t rsa -b 1024

安装必要工具

1.安装CFSSL
1
2
3
curl -o cfssl https://pkg.cfssl.org/R1.2/cfssl_darwin-amd64
curl -o cfssljson https://pkg.cfssl.org/R1.2/cfssljson_darwin-amd64
curl -o https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl cfssljson cfssl-certinfo

mv cfssl cfssljson cfssl-certinfo /usr/local/bin/

2.验证(CFSSL版本为1.2.0或者更高)

cfssl version

创建证书配置

1.证书说明

由于 Etcd 和 Kubernetes 全部采用 TLS 通讯,所以先要生成 TLS 证书,证书生成工具用CFSSL. 证书情况如下 > etcd:使用 ca.pem、kubernetes-key.pem、kubernetes.pem; > kube-apiserver:使用 ca.pem、kubernetes-key.pem、kubernetes.pem; > kubelet:使用 ca.pem; > kube-proxy:使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem; > kubectl:使用 ca.pem、admin-key.pem、admin.pem; > kube-controller-manager:使用 ca-key.pem、ca.pem

mkdir -p /etc/kubernetes/ssl/

cd /etc/kubernetes/ssl

2.创建 CA 配置文件

cat ca-csr.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ],
    "ca": {
       "expiry": "87600h"
    }
}
  • ca-config.json:可以定义多个 profiles,分别指定不同的过期时间、使用场景等参数;后续在签名证书时使用某个 profile;
  • signing:表示该证书可用于签名其它证书;生成的 ca.pem 证书中 CA=TRUE;
  • server auth:表示client可以用该 CA 对server提供的证书进行验证;
  • client auth:表示server可以用该CA对client提供的证书进行验证;
3.创建 CA 证书签名请求

cat ca-config.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
  • “CN”:Common Name,kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name);浏览器使用该字段验证网站是否合法;
  • “O”:Organization,kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group);
生成 CA 证书和私钥
1
2
3
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ ls ca*
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
4.创建 kubernetes 证书

cat kubernetes-csr.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "192.168.65.101",
      "192.168.65.102",
      "192.168.65.103",
      "10.254.0.1",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

需要指定授权使用该证书的 IP 或域名列表,由于该证书后续被 etcd 集群和 kubernetes master 集群使用,所以上面分别指定了 etcd 集群、kubernetes master 集群的主机 IP 和 kubernetes 服务的服务 IP(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP,如 10.254.0.1)。

生成 kubernetes 证书和私钥
1
2
3
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
$ ls kubernetes*
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem
5.创建 admin 证书

cat admin-csr.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
  • 后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权.

    生成 admin 证书和私钥:
    1
    2
    3
    
    $ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
    $ ls admin*
    admin.csr  admin-csr.json  admin-key.pem  admin.pem
6.创建 kube-proxy 证书

cat kube-proxy-csr.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
生成 kube-proxy 客户端证书和私钥
1
2
3
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem
7.使用opsnssl命令校验证书
1
openssl x509  -noout -text -in  kubernetes.pem
  • 确认 Issuer 字段的内容和 ca-csr.json 一致;
  • 确认 Subject 字段的内容和 kubernetes-csr.json 一致;
  • 确认 X509v3 Subject Alternative Name 字段的内容和 kubernetes-csr.json 一致;
  • 确认 X509v3 Key Usage、Extended Key Usage 字段的内容和 ca-config.json 中 kubernetes profile 一致.
8.分发证书到node节点
1
for i in k8s-n1 k8s-n2;do scp /etc/kubernetes/ssl/*.pem $i:/etc/kubernetes/ssl;done

创建 kubeconfig 文件

详细说明戳这里,kubeconfig.它们是 Kubernetes 客户端与 API Server 认证与鉴权的保证.

以下操作只需要在master节点上执行,生成的*.kubeconfig文件可以直接拷贝到node节点的/etc/kubernetes目录下。

1.安装组件
1
2
3
4
5
wget https://dl.k8s.io/v1.13.1/kubernetes-server-linux-amd64.tar.gz
tar -xzvf kubernetes-server-linux-amd64.tar.gz
cp -r kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kubelet,kubeadm} /usr/local/bin/
scp kubernetes/server/bin/{kube-proxy,kubelet} 192.168.65.102:/usr/local/bin/
scp kubernetes/server/bin/{kube-proxy,kubelet} 192.168.65.103:/usr/local/bin/
2.创建 TLS Bootstrapping Token
1
2
3
4
5
cd /etc/kubernetes
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
  • 注意:在进行后续操作前请检查 token.csv 文件,确认其中的 ${BOOTSTRAP_TOKEN} 环境变量已经被真实的值替换。
  • BOOTSTRAP_TOKEN 将被写入到 kube-apiserver 使用的 token.csv 文件和 kubelet 使用的 bootstrap.kubeconfig 文件,如果后续重新生成了 BOOTSTRAP_TOKEN,则需要:
  • 1.更新 token.csv 文件,分发到所有机器 (master 和 node)的 /etc/kubernetes/ 目录下,分发到node节点上非必需;
  • 2.重新生成 bootstrap.kubeconfig 文件,分发到所有 node 机器的 /etc/kubernetes/ 目录下;
  • 3.重启 kube-apiserver 和 kubelet 进程;
  • 4.重新 approve kubelet 的 csr 请求;
3.创建 kubelet bootstrapping kubeconfig 文件
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cd /etc/kubernetes
export KUBE_APISERVER="https://192.168.65.101:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig
# 设置客户端认证参数
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
4.创建 kube-proxy kubeconfig 文件
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
export KUBE_APISERVER="https://192.168.65.101:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
5.创建 kubectl kubeconfig 文件
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
export KUBE_APISERVER="https://192.168.65.101:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER}
# 设置客户端认证参数
kubectl config set-credentials admin \
  --client-certificate=/etc/kubernetes/ssl/admin.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/admin-key.pem
# 设置上下文参数
kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=admin
# 设置默认上下文
kubectl config use-context kubernetes

生成的 kubeconfig 被保存到 ~/.kube/config 文件,注意:~/.kube/config文件拥有对该集群的最高权限,请妥善保管。

6.分发 kubeconfig 文件
1
for i in k8s-n1,k8s-n2;do scp /etc/kubernetes/bootstrap.kubeconfig kube-proxy.kubeconfig token.csv $i:/etc/kubernetes;done

部署ETCD集群

1.下载ETCD二进制文件
1
2
3
4
5
wget https://github.com/coreos/etcd/releases/download/v3.2.24/etcd-v3.2.24-linux-amd64.tar.gz
tar zxvf etcd-v3.2.24-linux-amd64.tar.gz
cd etcd-v3.2.24-linux-amd64
cp etcd  etcdctl /usr/local/bin/
# 分发 etcd  etcdctl 到node节点上
2.创建 etcd 的 systemd unit 文件

cat /usr/lib/systemd/system/etcd.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \
  --name ${ETCD_NAME} \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
  --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
  --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
  --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
  --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
  --initial-cluster etcd1=https://192.168.65.101:2380,etcd2=https://192.168.65.102:2380,etcd3=https://192.168.65.103:2380 \
  --initial-cluster-state new \
  --data-dir=${ETCD_DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  • 注意替换IP地址为你自己的etcd集群的主机IP。
  • 创建 kubernetes.pem 证书时使用的 kubernetes-csr.json 文件的 hosts 字段包含所有 etcd 节点的IP,否则证书校验会出错;
  • 指定 etcd 的工作目录为 /var/lib/etcd,数据目录为 /var/lib/etcd,需在启动服务前创建这个目录,否则启动服务的时候会报
  • 保证通信安全,需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书(trusted-ca-file);
  • –initial-cluster-state 值为 new 时,–name 的参数值必须位于 –initial-cluster 列表中;
3.Etcd环境变量配置文件

cat /etc/etcd/etcd.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# [member]
ETCD_NAME=etcd1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.65.101:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.65.101:2379"

# [cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.65.101:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.65.101:2379"

分发到其他两个etcd节点,只要将上面的IP地址改成相应节点的IP地址即可。ETCD_NAME换成对应节点的etcd1/2/3

4.启动Etcd服务
1
systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

注意: 集群 etcd 要 3 个一起启动,集群模式下单个启动会卡半天最后失败,不要傻等;启动成功后测试如下

1
2
3
4
5
6
7
8
9
etcdctl \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
cluster-health
member 93ff3cf8f9f3ce76 is healthy: got healthy result from https://192.168.65.102:2379
member b9892e454ed62f46 is healthy: got healthy result from https://192.168.65.103:2379
member cb574713e8a13574 is healthy: got healthy result from https://192.168.65.101:2379
cluster is healthy

结果最后一行为 cluster is healthy 时表示集群服务正常,如果报错 请使用journalctl -u etcd 来定位问题

部署Master节点

  • kubernetes master 节点包含的组件:

kube-apiserver kube-scheduler kube-controller-manager kube-scheduler、kube-controller-manager 和 kube-apiserver 三者的功能紧密相关 同时只能有一个 kube-scheduler、kube-controller-manager 进程处于工作状态,如果运行多个,则需要通过选举产生一个 leader

1.配置 kube-apiserver
1.1创建 kube-apiserver的配置文件

cat /usr/lib/systemd/system/kube-apiserver

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[Unit]
Description=Kubernetes API Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/local/bin/kube-apiserver \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_ETCD_SERVERS \
        $KUBE_API_ADDRESS \
        $KUBE_API_PORT \
        $KUBELET_PORT \
        $KUBE_ALLOW_PRIV \
        $KUBE_SERVICE_ADDRESSES \
        $KUBE_ADMISSION_CONTROL \
        $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

cat /etc/kubernetes/config

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=2"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://192.168.65.101:8080"

该配置文件同时被kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy使用.

cat /etc/kubernetes/apiserver

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--advertise-address=192.168.65.101 --bind-address=192.168.65.101 --insecure-bind-address=192.168.65.101"

# The port on the local server to listen on.
KUBE_API_PORT="--secure-port=6443"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.65.101:2379,https://192.168.65.102:2379,https://192.168.65.103:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction"

# Add your own!
KUBE_API_ARGS=" --anonymous-auth=false \
                --apiserver-count=3 \
                --audit-log-maxage=30 \
                --audit-log-maxbackup=3 \
                --audit-log-maxsize=100 \
                --audit-log-path=/var/lib/audit.log \
                --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
                --authorization-mode=Node,RBAC \
                --client-ca-file=/etc/kubernetes/ssl/ca.pem \
                --enable-bootstrap-token-auth \
                --enable-garbage-collector \
                --enable-logs-handler \
                --etcd-cafile=/etc/kubernetes/ssl/ca.pem \
                --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \
                --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \
                --etcd-compaction-interval=5m0s \
                --etcd-count-metric-poll-period=1m0s \
                --event-ttl=48h0m0s \
                --kubelet-https=true \
                --kubelet-timeout=3s \
                --log-flush-frequency=5s \
                --token-auth-file=/etc/kubernetes/token.csv \
                --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
                --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
                --service-node-port-range=30000-50000 \
                --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
                --enable-swagger-ui=true"

kubelet 首次启动时向 kube-apiserver 发送 TLS Bootstrapping 请求,kube-apiserver 验证 kubelet 请求中的 token 是否与它配置的 token 一致,如果一致则自动为 kubelet生成证书和秘钥

2.配置 kube-controller-manager
2.1创建 kube-controller-manager的配置文件

cat /sur/lib/systemd/system/kube-controller-manager

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/controller-manager
ExecStart=/usr/local/bin/kube-controller-manager \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

cat /etc/kubernetes/controller-manager

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="  --bind-address=0.0.0.0 \
                                --cluster-name=kubernetes \
                                --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
                                --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
                                --controllers=*,bootstrapsigner,tokencleaner \
                                --deployment-controller-sync-period=10s \
                                --experimental-cluster-signing-duration=86700h0m0s \
                                --leader-elect=true \
                                --node-monitor-grace-period=40s \
                                --node-monitor-period=5s \
                                --pod-eviction-timeout=5m0s \
                                --terminated-pod-gc-threshold=50 \
                                --root-ca-file=/etc/kubernetes/ssl/ca.pem \
                                --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
                                --feature-gates=RotateKubeletServerCertificate=true"
3.配置 kube-scheduler
3.1创建 kube-scheduler的配置文件

cat /usr/lib/systemd/system/kube-scheduler

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/scheduler
ExecStart=/usr/local/bin/kube-scheduler \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_MASTER \
            $KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

cat /etc/kubernetes/scheduler

1
2
3
4
5
6
7
8
9
###
# kubernetes scheduler config

# default config should be adequate

# Add your own!
KUBE_SCHEDULER_ARGS="   --address=0.0.0.0 \
                        --leader-elect=true \
                        --algorithm-provider=DefaultProvider"
4.启动控制器服务
1
2
3
systemctl daemon-reload
systemctl enable kube-apiserver kube-controller-manager kube-scheduler
systemctl start kube-apiserver kube-controller-manager kube-scheduler
5.验证 master 节点功能
1
2
3
4
5
6
7
[root@k8s-m1 ~]# kubectl get componentstatuses
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   

安装Docker

Docker安装可参考官网 Master和Node都要安装Docker

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine

# 安装
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2

# 导入源
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo

# 更新repo
yum makecache
# 查看版本
yum list docker-ce.x86_64  --showduplicates |sort -r
# 安装Docker
yum install docker-ce-<VERSION STRING>
# 启动
systemctl enable docker && systemctl start docker
# 查看信息
[root@k8s-m1 ~]# docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:03 2018
 OS/Arch:           linux/amd64

部署Node节点

kubelet,kube-proxy,calico kube-proxy,kubelet,docker

1.创建kubelet的配置文件

kubelet 授权 kube-apiserver 的一些操作 exec run logs 等

** RBAC 只需创建一次就可以**

1
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

cat /usr/lib/systemd/system/kubelet.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ mdkir /var/lib/kubelet
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBELET_API_SERVER \
            $KUBELET_ADDRESS \
            $KUBELET_PORT \
            $KUBELET_HOSTNAME \
            $KUBE_ALLOW_PRIV \
            $KUBELET_ARGS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

cat /etc/kubernetes/kubelet

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
###
# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--node-ip=192.168.65.101"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=k8s-m1"

# location of the api-server
# KUBELET_API_SERVER=""

## pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0"

# Add your own!
KUBELET_ARGS="  --cgroup-driver=cgroupfs \
                --cluster-dns=10.254.0.2 \
                --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
                --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
                --cert-dir=/etc/kubernetes/ssl \
                --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \
                --cluster-domain=cluster.local \
                --hairpin-mode promiscuous-bridge \
                --serialize-image-pulls=false"

分发到node节点,IP和hostname修改为对应机器的地址

2.启动kubelet
1
2
systemctl daemon-reload
systemctl enable kubelet && systemctl start kubelet
3.通过kublet的TLS证书请求

kubelet 首次启动时向 kube-apiserver 发送证书签名请求,必须通过后 kubernetes 系统才会将该 Node 加入到集群。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[root@k8s-m1 ~]# kubectl get csr
NAME                                                   AGE   REQUESTOR           CONDITION
node-csr-Kw8n3OujgVJ5JkLaDwP_6DpjgAUZSxjHqHvA-r5tXRs   90s   kubelet-bootstrap   Pending

[root@k8s-m1 ~]# kubectl certificate approve node-csr-Kw8n3OujgVJ5JkLaDwP_6DpjgAUZSxjHqHvA-r5tXRs
certificatesigningrequest.certificates.k8s.io/node-csr-Kw8n3OujgVJ5JkLaDwP_6DpjgAUZSxjHqHvA-r5tXRs approved

[root@k8s-m1 ~]# kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-Kw8n3OujgVJ5JkLaDwP_6DpjgAUZSxjHqHvA-r5tXRs   2m34s   kubelet-bootstrap   Approved,Issued

补充说明

4.启动kubelet报错
  • failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User “kubelet-bootstrap” cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope

kubelet 启动时向 kube-apiserver 发送 TLS bootstrapping 请求,需要先将 bootstrap token 文件中的 kubelet-bootstrap 用户赋予 system:node-bootstrapper cluster 角色(role), 然后 kubelet 才能有权限创建认证请求(certificate signing requests)

1
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
  • –user=kubelet-bootstrap 是在 /etc/kubernetes/token.csv 文件中指定的用户名,同时也写入了 /etc/kubernetes/bootstrap.kubeconfig 文件

$ systemctl status kubelet 修改配置文件,不删除kubelet.kubeconfig文件会报错误:

1
2
3
4
5
6
7
13358 kubelet.go:2266] node "k8s-n1" not found
13358 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
13358 kubelet.go:2266] node "k8s-n1" not found
13358 kubelet_node_status.go:72] Attempting to register node k8s-n1
13358 kubelet_node_status.go:94] Unable to register node "k8s-n1" with API server: nodes "k8s-n1" is forbidden: node "k8s-... node "k8s-n1"
13358 kubelet.go:2266] node "k8s-n1" not found
13358 kubelet.go:2266] node "k8s-n1" not found

在master上通过kubectl get node 获得的列表中,Name显示的名称是通过 客户端kubelet和proxy配置文件中hostname-override配置参数定义的,修改这2个参数为你想要的名称,并且删除kubelet.kubeconfig(这个文件是master认证后客户端自动生成的,如果不删除会报node节点forbidden)文件,重新启动这2个服务,master端重新 kubectl certificate approve name名称 就可以看到新名称。

5.配置 kube-proxy

cat /usr/lib/systemd/system/kube-proxy.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/local/bin/kube-proxy \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_MASTER \
            $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

1.10 官方 ipvs 已经是默认的配置 –masquerade-all 必须添加这项配置,否则 创建 svc 在 ipvs 不会添加规则 打开 ipvs 需要安装 ipvsadm ipset conntrack 软件,

1
2
3
4
5
6
7
8
9
yum install ipset ipvsadm conntrack-tools.x86_64 -y

[root@k8s-m1 ~]# lsmod | grep ip_vs
ip_vs_sh               12688  0 
ip_vs_wrr              12697  0 
ip_vs_rr               12600  5 
ip_vs                 145497  11 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          133095  7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
libcrc32c              12644  4 xfs,ip_vs,nf_nat,nf_conntrack

cat /etc/kubernetes/proxy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="--bind-address=0.0.0.0 \
                 --hostname-override=k8s-m1 \
                 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
                 --proxy-mode=ipvs \
                 --feature-gates=SupportIPVSProxyMode=true \
                 --masquerade-all \
                 --ipvs-scheduler=rr \
                 --cluster-cidr=10.254.0.0/16"

–hostname-override 参数值必须与 kubelet 的值一致,否则 kube-proxy 启动后会找不到该 Node,从而不会创建任何 iptables 规则

启动 kube-proxy
1
2
systemctl daemon-reload
systemctl enable kube-proxy && systemctl start kube-proxy

6.配置 Flannel 网络

1.下载 Flannel

wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

1
2
tar -zvxf flannel-v0.10.0-linux-amd64.tar.gz
mv flanneld mk-docker-opts.sh /usr/local/bin
  • cat /etc/kubernetes/falnneld
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="https://192.168.65.101:2379,https://192.168.65.102:2379,https://192.168.65.103:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/flannel/network"

# Any additional options that you want to pass
FLANNEL_OPTIONS="-etcd-cafile=/etc/kubernetes/ssl/ca.pem -etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem -etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem"
  • cat /usr/lib/systemd/system/flanneld.service
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
 
[Service]
Type=notify
EnvironmentFile=/etc/kubernetes/flanneld
ExecStart=/usr/local/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

配置Docker启动指定子网

修改配置文件EnvironmentFile=/run/flannel/subnet.env,ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS即可

ETCD注册网段

flannel服务启动时主要做了以下几步的工作:从etcd中获取network的配置信息 划分subnet 并在etcd中进行注册 将子网信息记录到/run/flannel/subnet.env中

1
2
3
4
etcdctl --ca-file=/etc/kuberentes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
--key-file=/etc/kubernetes/ssl/ca-key.pem --endpoints="https://192.168.65.101:2379, \
https://192.168.65.102:2379,https://192.168.65.103:2379"  set /flannel/network/config \
'{ "Network": "10.254.0.0/16", "Backend": {"Type": "vxlan"}}'
启动flannel服务
1
2
systemctl daemon-reload && systemctl enable flanneld
systemctl start flanneld && systemctl restart docker
验证服务
  • cat /run/flannel/subnet.env
1
2
3
4
5
6
7
DOCKER_OPT_BIP="--bip=10.254.35.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.254.35.1/24 --ip-masq=false --mtu=1450"

# 查看flannle网络服务
ip a 
1
2
3
4
5
6
kubectl get nodes

NAME     STATUS   ROLES    AGE    VERSION
k8s-m1   Ready    <none>   145m   v1.13.1
k8s-n1   Ready    <none>   55m    v1.13.1
k8s-n2   Ready    <none>   15s    v1.13.1