kubeadm是Kubernetes官方提供的用于快速安装Kubernetes集群的工具,伴随Kubernetes每个版本的发布都会同步更新,kubeadm会对集群配置方面的一些实践做调整,通过实验kubeadm可以学习到Kubernetes官方在集群配置上一些新的最佳实践。

环境配置

| Type | IP | OS | | master | 192.168.65.101 | centos | | node1 | 192.168.65.102 | centos | | node2 | 192.168.65.103 | centos |

免密登陆SSH
1
2
3
ssh-keygen -t rsa
分发公钥
for i in k8s-n1 k8s-n2;do ssh-copy-id -i /root/.ssh/id-rsa.pub $i;done
关闭防火墙和SELINUX
1
2
3
4
systemctl stop firewalld && systemctl disable firewalld
setenforce 0
vi /etc/selinux/config
SELINUX=disabled
关闭SWAP
1
2
3
swapoff -a && sysctl -w vm.swappiness=0
vi /etc/fstab
# UUID=7bff6243-324c-4587-b550-55dc34018ebf swap                    swap    defaults        0 0
设置Docker所需参数
1
2
3
4
5
6
7
8
cat /etc/sysctl.d/k8s.conf

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

编辑 /etc/hosts 文件,配置hostname 通信

1
2
3
192.168.65.101 k8s-m1
192.168.65.102 k8s-n1
192.168.65.103 k8s-n2
安装Docker

Docker安装教程官方文档

1
2
3
4
5
yum -y install yum-utils 
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum list docker-ce --showduplicates | sort -r
yum install docker-ce-(版本) -y
systemctl start docker && systemctl enable docker
加载ipvs相关模块

由于ipvs已经加入到了内核的主干,所以为kube-proxy开启ipvs的前提需要加载以下的内核模块:
在所有的Kubernetes节点执行以下脚本:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cat >/etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件,保证在节点重启后能自动加载所需模块。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[root@k8s-m1 ~]# lsmod | grep -e ip_vs -e nf_conntrack_ipv4
ip_vs_sh               16384  0 
ip_vs_wrr              16384  0 
ip_vs_rr               16384  0 
ip_vs                 151552  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          143360  6 xt_conntrack,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink,ip_vs
libcrc32c              16384  4 nf_conntrack,nf_nat,xfs,ip_vs

# 查看IPVS规则
ipvsadm -ln

接下来还需要确保各个节点上已经安装了ipset软件包yum install ipset。 为了便于查看ipvs的代理规则,最好安装一下管理工具ipvsadm yum install ipvsadm。 如果以上前提条件如果不满足,则即使kube-proxy的配置开启了ipvs模式,也会退回到iptables模式。

使用kubeadm部署Kubernetes

安装kubeadm、kubelet、kubectl

kubelet 在群集中所有节点上运行的核心组件, 用来执行如启动pods和containers等操作。 kubeadm 引导启动k8s集群的命令行工具,用于初始化 Cluster。 kubectl 是 Kubernetes 命令行工具。通过 kubectl 可以部署和管理应用,查看各种资源,创建、删除和更新各种组件。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
#配置kubernetes.repo的源,由于官方源国内无法访问,这里使用阿里云yum源.

cat /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

#在master上安装指定版本 kubelet、kubeadm 和 kubectl,node上安装指定版本 kubelet、kubeadm.

yum clean all
yum makecache
yum install -y kubelet-1.13.3 kubeadm-1.13.3 kubectl-1.13.3

从安装结果可以看出还安装了cri-tools, kubernetes-cni, socat三个依赖: kubernetes-cni(容器网络接口),CNI仅关心容器创建时的网络分配,和当容器被删除时释放网络资源 socat是kubelet的依赖 cri-tools是CRI(Container Runtime Interface)容器运行时接口的命令行工具

启动kubelet

systemctl enable kubelet

部署master

Master节点执行初始化
1
2
3
4
5
kubeadm init \
    --apiserver-advertise-address=192.168.65.101 \
    --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
    --kubernetes-version v1.13.3 \
    --pod-network-cidr=10.244.0.0/16

–apiserver-advertise-address - 指明用 Master 的哪个 interface 与 Cluster 的其他节点通信。如果 Master 有多个 interface,建议明确指定,如果不指定,kubeadm 会自动选择有默认网关的 interface。

–pod-network-cidr - 指定 Pod 网络的范围。Kubernetes 支持多种网络方案,而且不同网络方案对 –pod-network-cidr 有自己的要求,这里设置为 10.244.0.0/16 是因为我们将使用 flannel 网络方案,须设置成这个 CIDR。

–image-repository - 指定初始化需要的镜像源从阿里云镜像仓库拉取.

–kubernetes-version=v1.13.3 - 关闭版本探测,因为它的默认值是stable-1,会导致从https://dl.k8s.io/release/stable-1.txt下载最新的版本号,我们可以将其指定为固定版本来跳过网络请求。

下面记录了完成的初始化输出的内容,根据输出的内容基本上可以看出手动初始化安装一个Kubernetes集群所需要的关键步骤

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
[root@k8s-m1 ~]# kubeadm init \
>     --apiserver-advertise-address=192.168.65.101 \
>     --image-repository registry.aliyuncs.com/google_containers \
>     --kubernetes-version v1.13.3 \
>     --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.13.3
[preflight] Running pre-flight checks
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-m1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.65.101]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-m1 localhost] and IPs [192.168.65.101 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-m1 localhost] and IPs [192.168.65.101 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 25.002295 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-m1" as an annotation
[mark-control-plane] Marking the node k8s-m1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-m1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: irpqr1.r2kuxfpwwwm7wzi3
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.65.101:6443 --token irpqr1.r2kuxfpwwwm7wzi3 --discovery-token-ca-cert-hash sha256:0327e687909354d4b00a69e712d1d8d77f46e8f79c60e2263872be7ecc47d11c

注意记录下初始化结果中的kubeadm join命令,部署worker节点时会用到

  • [kubelet-start] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
  • [certificates]生成相关的各种证书
  • [kubeconfig]生成相关的kubeconfig文件
  • [bootstraptoken]生成token记录下来,后边使用kubeadm join往集群中添加节点时会用到
  • 下面的命令是配置常规用户如何使用kubectl访问集群:

    1
    2
    3
    
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
查看一下集群状态
1
2
3
4
5
kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   

部署网络插件 Flannel

要让 Kubernetes Cluster 能够工作,必须安装 Pod 网络,否则 Pod 之间无法通信。 Kubernetes 支持多种网络方案,这里我们使用 flannel.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
kubectl get pod -n kube-system -o wide

NAME                             READY   STATUS              RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
coredns-78d4cf999f-ch66h         0/1     ContainerCreating   0          5m54s   <none>           k8s-m1   <none>           <none>
coredns-78d4cf999f-dzxf7         0/1     ContainerCreating   0          5m53s   <none>           k8s-m1   <none>           <none>
etcd-k8s-m1                      1/1     Running             0          5m2s    192.168.65.101   k8s-m1   <none>           <none>
kube-apiserver-k8s-m1            1/1     Running             0          5m20s   192.168.65.101   k8s-m1   <none>           <none>
kube-controller-manager-k8s-m1   1/1     Running             0          5m      192.168.65.101   k8s-m1   <none>           <none>
kube-flannel-ds-amd64-kzqdt      0/1     Init:0/1            0          3s      192.168.65.101   k8s-m1   <none>           <none>
kube-proxy-gd4xg                 1/1     Running             0          5m54s   192.168.65.101   k8s-m1   <none>           <none>
kube-scheduler-k8s-m1            1/1     Running             0          5m7s    192.168.65.101   k8s-m1   <none>           <none>
  • 镜像在拉取中,等一会儿。

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    
    kubectl get pod -n kube-system -o wide
    
    NAME                             READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
    coredns-78d4cf999f-ch66h         1/1     Running   0          8m24s   10.244.0.2       k8s-m1   <none>           <none>
    coredns-78d4cf999f-dzxf7         1/1     Running   0          8m23s   10.244.0.3       k8s-m1   <none>           <none>
    etcd-k8s-m1                      1/1     Running   0          7m32s   192.168.65.101   k8s-m1   <none>           <none>
    kube-apiserver-k8s-m1            1/1     Running   0          7m50s   192.168.65.101   k8s-m1   <none>           <none>
    kube-controller-manager-k8s-m1   1/1     Running   0          7m30s   192.168.65.101   k8s-m1   <none>           <none>
    kube-flannel-ds-amd64-kzqdt      1/1     Running   0          2m33s   192.168.65.101   k8s-m1   <none>           <none>
    kube-proxy-gd4xg                 1/1     Running   0          8m24s   192.168.65.101   k8s-m1   <none>           <none>
    kube-scheduler-k8s-m1            1/1     Running   0          7m37s   192.168.65.101   k8s-m1   <none>           <none>
  • 所有的系统 Pod 都成功启动了,刚刚部署的flannel网络插件则在 kube-system 下面新建了一个名叫kube-flannel-ds-amd64-kzqdt的 Pod,这些 Pod 就是容器网络插件在每个节点上的控制组件。

  • Kubernetes 支持容器网络插件,使用的是一个名叫 CNI 的通用接口,它也是当前容器网络的事实标准,容器网络开源项目都可以通过 CNI 接入 Kubernetes

查看master节点状态
1
2
3
kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
k8s-m1   Ready    master   9m50s   v1.13.3

Calico 部署

Calico创建和管理一个扁平的三层网络(不需要overlay),每个容器会分配一个可路由的IP。由于通信时不需要解包和封包,网络性能损耗小,易于排查,且易于水平扩展。

小规模部署时可以通过BGP client直接互联,大规模下可通过指定的BGP Route Reflector来完成,这样保证所有的数据流量都是通过IP路由的方式完成互联的。

Calico基于iptables还提供了丰富而灵活的网络Policy,保证通过各个节点上的ACL来提供Workload的多租户隔离、安全组以及其他可达性限制等功能。

官网下载

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
wget https://docs.projectcalico.org/v3.5/getting-started/kubernetes/installation/hosted/etcd.yaml
kubectl apply -f etcd.yaml

wget https://docs.projectcalico.org/v3.5/getting-started/kubernetes/installation/hosted/calico.yaml
# 修改pod-CIRD
cat calico.yaml
...
- name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16" (创建集群时指定的pod_cidr)

kubectl create -f calico.yaml

部署Node节点

在 k8s-node1 和 k8s-node2 上分别执行如下命令,将其注册到 Cluster 中

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
kubeadm join 192.168.65.101:6443 --token irpqr1.r2kuxfpwwwm7wzi3 --discovery-token-ca-cert-hash sha256:0327e687909354d4b00a69e712d1d8d77f46e8f79c60e2263872be7ecc47d11c

[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "192.168.65.101:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.65.101:6443"
[discovery] Requesting info from "https://192.168.65.101:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.65.101:6443"
[discovery] Successfully established connection with API Server "192.168.65.101:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-n1" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

如果我们忘记了Master节点的加入token,可以使用如下命令来查看

1
2
3
4
kubeadm token list

TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
irpqr1.r2kuxfpwwwm7wzi3   19h       2019-02-20T11:27:53+08:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

默认情况下,token的有效期是24小时,如果我们的token已经过期的话,可以使用以下命令重新生成

1
2
3
kubeadm token create

dm89r6.ohpx3hlu7d94djov

没有–discovery-token-ca-cert-hash的值,可以使用以下命令生成

1
2
3
4
5
6
7
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | \
openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //'

0327e687909354d4b00a69e712d1d8d77f46e8f79c60e2263872be7ecc47d11c

kubeadm join masterIP:6443 --token dm89r6.ohpx3hlu7d94djov --discovery-token-ca-cert-hash sha256:0327e687909354d4b00a69e712d1d8d77f46e8f79c60e2263872be7ecc47d11c --skip-preflight-check
查看节点状态
1
2
3
4
5
kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
k8s-m1   Ready    master   9m50s   v1.13.3
k8s-n1   Ready    <none>   3m1s    v1.13.3
k8s-n2   Ready    <none>   2m56s   v1.13.3

如果node节点状态为NotReady,等一会儿,查看所有节点pod状态,确保所有pod成功拉取到镜像并处于running状态

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
kubectl get pod --all-namespaces -o wide

NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-78d4cf999f-ch66h         1/1     Running   0          16m     10.244.0.2       k8s-m1   <none>           <none>
kube-system   coredns-78d4cf999f-dzxf7         1/1     Running   0          16m     10.244.0.3       k8s-m1   <none>           <none>
kube-system   etcd-k8s-m1                      1/1     Running   0          16m     192.168.65.101   k8s-m1   <none>           <none>
kube-system   kube-apiserver-k8s-m1            1/1     Running   0          16m     192.168.65.101   k8s-m1   <none>           <none>
kube-system   kube-controller-manager-k8s-m1   1/1     Running   0          16m     192.168.65.101   k8s-m1   <none>           <none>
kube-system   kube-flannel-ds-amd64-kzqdt      1/1     Running   0          11m     192.168.65.101   k8s-m1   <none>           <none>
kube-system   kube-flannel-ds-amd64-l9fwk      1/1     Running   0          3m19s   192.168.65.103   k8s-n2   <none>           <none>
kube-system   kube-flannel-ds-amd64-n8bwz      1/1     Running   0          3m24s   192.168.65.102   k8s-n1   <none>           <none>
kube-system   kube-proxy-9zq9z                 1/1     Running   0          3m19s   192.168.65.103   k8s-n2   <none>           <none>
kube-system   kube-proxy-bgmwh                 1/1     Running   0          3m24s   192.168.65.102   k8s-n1   <none>           <none>
kube-system   kube-proxy-gd4xg                 1/1     Running   0          16m     192.168.65.101   k8s-m1   <none>           <none>
kube-system   kube-scheduler-k8s-m1            1/1     Running   0          16m     192.168.65.101   k8s-m1   <none>           <none>
  • 如果有pod提示Init:ImagePullBackOff,说明这个pod的镜像在对应节点上拉取失败,我们可以通过 kubectl describe pod 查看 Pod 具体情况,以确认拉取失败的镜像
  • 可以耐心等待,因为 Kubernetes 会重试,我们也可以自己手工执行 docker pull 去下载这个镜像
  • 如果无法下载镜像,可以从阿里云或者dockerhub镜像仓库下载,然后改回原来的tag即可

测试集群各个组件

部署一个 Nginx Deployment,包含2个Pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# kubectl create deployment nginx --image=nginx:alpine

# kubectl scale deployment nginx --replicas=2

kubectl get pods -l app=nginx -o wide
NAME                     READY   STATUS              RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
nginx-54458cd494-sfd4h   0/1     ContainerCreating   0          10s   <none>       k8s-n2   <none>           <none>
nginx-54458cd494-x2dmq   1/1     Running             0          19s   10.244.1.2   k8s-n1   <none>           <none>

kubectl get pods -l app=nginx -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
nginx-54458cd494-sfd4h   1/1     Running   0          54s   10.244.2.2   k8s-n2   <none>           <none>
nginx-54458cd494-x2dmq   1/1     Running   0          63s   10.244.1.2   k8s-n1   <none>           <none>
验证kube-proxy
1
2
3
4
5
6
kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed

kubectl get services nginx
NAME    TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
nginx   NodePort   10.110.133.100   <none>        80:30402/TCP   16s
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
curl 192.168.65.101:30402

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
.....
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
curl 192.168.65.102:30402

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
......
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
curl 192.168.65.103:30402

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
.....
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

可以通过任意 NodeIP:Port 在集群外部访问这个服务

kube-proxy开启ipvs

修改ConfigMap的kube-system/kube-proxy中的config.conf,mode: “ipvs”

1
kubectl edit cm kube-proxy -n kube-system

重启各个节点上的kube-proxy pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'

pod "kube-proxy-9zq9z" deleted
pod "kube-proxy-bgmwh" deleted
pod "kube-proxy-gd4xg" deleted

kubectl get pod -n kube-system | grep kube-proxy
kube-proxy-dx2kh                 1/1     Running   0          14s
kube-proxy-tm4pm                 1/1     Running   0          16s
kube-proxy-txknb                 1/1     Running   0          18s

查看日志,日志中打印出了Using ipvs Proxier,说明ipvs模式已经开启

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
kubectl logs kube-proxy-dx2kh -n kube-system

I0219 03:57:10.827450       1 server_others.go:189] Using ipvs Proxier.
W0219 03:57:10.827735       1 proxier.go:365] IPVS scheduler not specified, use rr by default
I0219 03:57:10.827902       1 server_others.go:216] Tearing down inactive rules.
I0219 03:57:10.867557       1 server.go:464] Version: v1.13.1
I0219 03:57:10.883878       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0219 03:57:10.884170       1 config.go:202] Starting service config controller
I0219 03:57:10.884178       1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0219 03:57:10.884221       1 config.go:102] Starting endpoints config controller
I0219 03:57:10.884225       1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0219 03:57:10.984487       1 controller_utils.go:1034] Caches are synced for service config controller
I0219 03:57:10.984522       1 controller_utils.go:1034] Caches are synced for endpoints config controller
验证DNS, pod Network

运行Busybox并进入交互模式

1
2
3
4
5
kubectl run -it curl --image=radial/busyboxplus:curl

kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@curl-66959f6557-d4gp6:/ ]$ 

查看是否可以正确解析出集群内的IP,以验证DNS.

1
2
3
4
5
6
7
nslookup nginx

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx
Address 1: 10.110.133.100 nginx.default.svc.cluster.local

通过服务名进行访问,验证kube-proxy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[ root@curl-66959f6557-d4gp6:/ ]$ curl http://nginx/

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
......
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

分别访问一下2个Pod的内网IP,验证跨Node的网络通信

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[ root@curl-66959f6557-d4gp6:/ ]$ curl 10.244.1.2

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
......
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[ root@curl-66959f6557-d4gp6:/ ]$ curl 10.244.2.2

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
......
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

默认配置下Kubernetes不会将Pod调度到Master节点。查看Taints字段默认配置

1
2
3
kubectl describe node k8s-master
.....
Taints:             node-role.kubernetes.io/master:NoSchedule

如果希望将master也当作Node节点使用,可以执行如下命令

1
2
3
kubectl taint node k8s-m1 node-role.kubernetes.io/master-
.....
Taints:             <none>

如果要恢复Master Only状态,执行如下命令

1
2
3
kubectl taint node k8s-m1 node-role.kubernetes.io/master=:NoSchedule
.....
Taints:             node-role.kubernetes.io/master:NoSchedule

卸载集群

  • 撤销kubeadm执行的操作,首先要排除节点,并确保该节点为空, 然后再将其关闭
  • 在Master节点上运行
1
2
kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
  • 然后在需要移除的节点上,重置kubeadm的安装状态
1
kubeadm reset
  • 在master上删除node并不会清理被移除的节点上运行的容器,需要在删除节点上面手动运行清理命令。
  • 如果你想重新配置集群,使用新的参数重新运行kubeadm init或者kubeadm join即可。

Kubernetes常用组件部署

Dashboard (UI) 安装
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
curl -O https://github.com/kubernetes/dashboard/blob/master/aio/deploy/recommended/kubernetes-dashboard.yaml

# 替换所有images
sed -i 's#k8s.gcr.io#gcrxio#g' kubernetes-dashboard.yaml

# 导入文件
kubectl create -f kubernetes-dashboard.yaml

# 查看pod和svc
kubectl get pods,svc -n kube-system
暴露服务

cat kubernetes-dashboard.yaml - 这里添加type: NodePort,添加你想暴露的端口nodePort: 30000(使用默认也可以)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
    - name: dashboard-tls
      port: 443
      targetPort: 8443
      nodePort: 30000
      protocol: TCP
  selector:
    k8s-app: kubernetes-dashboard
1
2
3
4
5
6
7
# 重新导入文件
kubectl delete -f kubernetes-dashboard.yaml

kubectl create -f kubernetes-dashboard.yaml

# 查看pod和svc
kubectl get pods,svc -n kube-system
创建Admin账户

部署成功后可以直接访问 https://NODE_IP:30000 访问,但是想要登录进去查看的话需要使用 kubeconfig 或者 access token 的方式;实际上这个就是 RBAC 授权控制,以下提供一个创建 admin access token 的脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# 首先创建一个 dashboard rbac 超级用户
vim dashboard-admin-rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard-admin
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard-admin
  namespace: kube-system
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# 导入文件
kubectl create -f dashboard-admin-rbac.yaml

serviceaccount/kubernetes-dashboard-admin created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-admin created

# 查看创建的token
kubectl -n kube-system get secret | grep kubernetes-dashboard-admin

kubernetes-dashboard-admin-token-rw2l8  kubernetes.io/service-account-token  3  21s

查看登陆令牌

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
kubectl describe kubernetes-dashboard-admin-token-rw2l8 -n kube-system

Name:         kubernetes-dashboard-admin-token-rw2l8
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard-admin
              kubernetes.io/service-account.uid: 2ab00bf0-35b6-11e9-bb5a-000c29f3419f

Type:  kubernetes.io/service-account-token

  ta
▽===
token:      eyJhbGciOiJSU.....3EKwGr1hSfHe3A
ca.crt:     1025 bytes
namespace:  11 bytes
登陆Web Ui

访问https://NODE_IP:30000,令牌登陆

![]()

部署监控

从Kubernetes 1.11开始,Heapster被标记为已弃用。建议用户使用 metrics-server ,还有第三方监控解决方案,如Prometheus。 - 导入镜像,下载地址点击这里

下载所有yaml到目录metrics-server,并修改metrics-server-deployment.yaml内容

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
containers:
- name: metrics-server
image: registry.cn-hangzhou.aliyuncs.com/mirror_googlecontainers/metrics-server-amd64:v0.3.1
imagePullPolicy: IfNotPresent
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
volumeMounts:
- name: tmp-dir
  mountPath: /tmp

修改镜像默认拉去策略为 IfNotPresent:先从本地镜像拉取,再去官网获取镜像 image更改为阿里云仓库获取 修改使用 IP 连接并且不验证证书

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 执行部署
kubectl create -f metric-server/

# 查看监控数据
kubectl top nodes

NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-m1   277m         13%    1163Mi          62%       
k8s-n1   36m          1%     558Mi           29%       
k8s-n2   35m          1%     543Mi           29%   

kubectl top pods --all-namespaces

NAMESPACE     NAME                                    CPU(cores)   MEMORY(bytes)   
default       curl-66959f6557-d4gp6                   0m           1Mi             
default       nginx-54458cd494-sfd4h                  0m           2Mi             
default       nginx-54458cd494-x2dmq                  0m           2Mi             
kube-system   coredns-78d4cf999f-ch66h                2m           11Mi            
kube-system   coredns-78d4cf999f-dzxf7                2m           12Mi            
kube-system   etcd-k8s-m1                             14m          66Mi            
kube-system   kube-apiserver-k8s-m1                   30m          465Mi           
kube-system   kube-controller-manager-k8s-m1          23m          66Mi            
kube-system   kube-flannel-ds-amd64-kzqdt             2m           12Mi            
kube-system   kube-flannel-ds-amd64-l9fwk             2m           13Mi            
kube-system   kube-flannel-ds-amd64-n8bwz             2m           15Mi            
kube-system   kube-proxy-dx2kh                        2m           20Mi            
kube-system   kube-proxy-tm4pm                        2m           21Mi            
kube-system   kube-proxy-txknb                        2m           21Mi            
kube-system   kube-scheduler-k8s-m1                   7m           15Mi            
kube-system   kubernetes-dashboard-78cbd76497-dxszs   1m           16Mi            
kube-system   metrics-server-5c9b89bc84-tnpxq         1m           14Mi      
Helm的安装

Helm Charts:使在Kubernetes上打包和部署常见应用程序变得简单 Helm是包管理器(类似于yum和apt),而Charts是包(类似于debs和rpms) - 查找并使用已经打包为 Kubernetes charts 的流行软件 - 分享您自己的应用作为 Kubernetes charts - 为 Kubernetes 应用创建可重复执行的构建 - Helm 客户端。它用来创建,拉取,搜索和验证 Charts,初始化 Tiller 服务 - Tiller server。跑在 Kubernetes 集群之内,管理这些应用的发布。

下载包或者其他版本

1
2
3
4
5
6
# 这一步就是安装helm的client端
wget https://storage.googleapis.com/kubernetes-helm/helm-v2.12.3-linux-amd64.tar.gz

tar xvf helm-v2.12.3-linux-amd64.tar.gz

cp linux-amd64/helm /usr/local/bin/

初始化helm server,使用阿里云镜像

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
helm init --upgrade -i \
registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.12.3 \
--stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

Creating /root/.helm 
Creating /root/.helm/repository 
Creating /root/.helm/repository/cache 
Creating /root/.helm/repository/local 
Creating /root/.helm/plugins 
Creating /root/.helm/starters 
Creating /root/.helm/cache/archive 
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /root/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!

# 查看pod
kubectl get pod -n kube-system |grep till
tiller-deploy-7dc7bd8bd9-jhlh8          1/1     Running   0          111s
查看版本
1
2
3
helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"08c1144f5eb3e3b636d9775617287cc26e53dba4", GitTreeState:"clean"}
使用Helm部署应用
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 创建服务账号
kubectl create serviceaccount --namespace kube-system tiller
serviceaccount/tiller created

# 创建集群的角色绑定
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created

# 为应用程序设置serviceAccount
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
deployment.extensions/tiller-deploy patched

# 搜索Helm应用
helm search jenkins
NAME          	CHART VERSION	APP VERSION	DESCRIPTION                                                 
stable/jenkins	0.13.5       	2.73       	Open source continuous integration server. It supports mu...

helm repo list
NAME  	URL                                                   
stable	https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
local 	http://127.0.0.1:8879/charts                       

# 安装应用
helm install stable/jenkins