簡介
目前生產部署kubernetes集群主要兩種方式
kubeadm
Kubeadm是一個K8s部署工具，提供kubeadm init和kubeadm join，用於快速部署Kubernetes集群。
二進制包
從github下載發行版的二進制包，手動部署每個組件，組成Kubernetes集群。
Kubeadm降低部署門檻，但屏蔽了很多細節，遇到問題很難排查。如果想更容易可控，推薦使用二進制包部署Kubernetes集群，雖然手動部署麻煩點，期間可以學習很多工作原理，也利於後期維護。
二進制部署K8s
List
CentOS7.3
cni-plugins-linux-amd64-v0.8.6.tgz
etcd-v3.4.9-linux-amd64.tar.gz
kube-flannel.yml
kubernetes-server-linux-amd64.tar.gz
| 角色 |
IP |
組件 |
| master |
192.168.31.71 |
kube-apiserver，kube-controller-manager，kube-scheduler，etcd |
| Node1 |
192.168.31.74 |
kube-apiserver，kube-controller-manager，kube-scheduler |
| Node2 |
192.168.31.72 |
kubelet，kube-proxy，docker etcd |
初始化環境
# 初始化
init_security() {
systemctl stop firewalld
systemctl disable firewalld &>/dev/null
setenforce 0
sed -i '/^SELINUX=/ s/enforcing/disabled/' /etc/selinux/config
sed -i '/^GSSAPIAu/ s/yes/no/' /etc/ssh/sshd_config
sed -i '/^#UseDNS/ {s/^#//;s/yes/no/}' /etc/ssh/sshd_config
systemctl enable sshd crond &> /dev/null
rpm -e postfix --nodeps
echo -e "\033[32m [安全配置] ==> OK \033[0m"
}
init_security
init_yumsource() {
if [ ! -d /etc/yum.repos.d/backup ];then
mkdir /etc/yum.repos.d/backup
fi
mv /etc/yum.repos.d/* /etc/yum.repos.d/backup 2>/dev/null
if ! ping -c2 www.baidu.com &>/dev/null
then
echo "您無法上外網，不能配置yum源"
exit
fi
curl -o /etc/yum.repos.d/163.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo &>/dev/null
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo &>/dev/null
yum clean all
timedatectl set-timezone Asia/Shanghai
echo "nameserver 114.114.114.114" > /etc/resolv.conf
echo "nameserver 8.8.8.8" >> /etc/resolv.conf
chattr +i /etc/resolv.conf
yum -y install ntpdate
ntpdate -b ntp1.aliyun.com # 對時很重要
echo -e "\033[32m [YUM Source] ==> OK \033[0m"
}
init_yumsource
# 關掉swap分區
swapoff -a
# 如果想永久關掉swap分區，打開如下文件註釋掉swap哪一行即可.
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab #永久
# 配置主機名解析
tail -3 /etc/hosts
192.168.0.121 master
192.168.0.123 node1
192.168.0.124 node2
# 將橋接的IPv4流量傳遞到iptables的鏈
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system # 生效
# 升級內核(非必須,只是性能更好)
wget https://cbs.centos.org/kojifiles/packages/kernel/4.9.220/37.el7/x86_64/kernel-4.9.220-37.el7.x86_64.rpm
rpm -ivh kernel-4.9.220-37.el7.x86_64.rpm
reboot
部署etcd集群
Etcd 是一個分佈式鍵值存儲系統，Kubernetes使用Etcd進行數據存儲，所以先準備一個Etcd數據庫，為解決Etcd單點故障，應採用集群方式部署，這裏使用3台組建集群，可容忍1台機器故障，當然，你也可以使用5台組建集群，可容忍2台機器故障。
| 節點名稱 |
IP |
| etcd-1 |
192.168.31.71 |
| etcd-2 |
192.168.31.72 |
| etcd-3 |
192.168.31.73 |
注：為了節省機器，這裏與K8s節點機器復用。也可以獨立於k8s集群之外部署，只要apiserver能連接到就行。
準備cfssl證書生成工具
cfssl是一個開源的證書管理工具，使用json文件生成證書，相比openssl更方便使用。
找任意一台服務器操作，這裏用Master節點。
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
生成etcd證書
創建工作目錄
mkdir -p ~/TLS/{etcd,k8s}
cd TLS/etcd
自簽CA
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat > ca-csr.json << EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
生成證書
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
ls *pem
ca-key.pem ca.pem
使用自簽CA簽發etcd https證書
創建證書申請文件
cat > server-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"192.168.0.121",
"192.168.0.123",
"192.168.0.124"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
注：上述文件hosts字段中IP為所有etcd節點的集群內部通信IP，一個都不能少！為了方便後期擴容可以多寫幾個預留的IP。
生成證書
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
ls server*pem
server-key.pem server.pem
下載etcd二進制文件
wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
創建工作目錄並解壓二進制包
mkdir /opt/etcd/{bin,cfg,ssl} -p
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
# 配置etcd
cat /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.121:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.121:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.121:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.121:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.0.121:2380,etcd-2=https://192.168.0.123:2380,etcd-3=https://192.168.0.124:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
# ETCD_NAME：節點名稱，集群中唯一
# ETCD_DATA_DIR：數據目錄
# ETCD_LISTEN_PEER_URLS：集群通信監聽地址
# ETCD_LISTEN_CLIENT_URLS：客戶端訪問監聽地址
# ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
# ETCD_ADVERTISE_CLIENT_URLS：客戶端通告地址
# ETCD_INITIAL_CLUSTER：集群節點地址
# ETCD_INITIAL_CLUSTER_TOKEN：集群Token
# ETCD_INITIAL_CLUSTER_STATE：加入集群的當前狀態，new是新集群，existing表示加入已有集群
systemd管理etcd
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
拷貝剛生成證書及生成的文件拷貝到節點2,節點3
cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/
scp -r /opt/etcd/ node1:/opt/
scp -r /opt/etcd/ node2:/opt/
scp /usr/lib/systemd/system/etcd.service node1:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service node2:/usr/lib/systemd/system/
修改節點2和節點3etcd.conf配置文件
node-1
cat /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.123:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.123:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.123:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.123:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.0.121:2380,etcd-2=https://192.168.0.123:2380,etcd-3=https://192.168.0.124:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
# node-2
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.124:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.124:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.124:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.124:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.0.121:2380,etcd-2=https://192.168.0.123:2380,etcd-3=https://192.168.0.124:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
# 啟動服務並設置開機自啟
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
驗證etcd集群狀態
/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.0.121:2379,https://192.168.0.123:2379,https://192.168.0.124:2379" endpoint healthhttps://192.168.0.124:2379 is healthy: successfully committed proposal: took = 13.213712ms
https://192.168.0.121:2379 is healthy: successfully committed proposal: took = 12.907787ms
https://192.168.0.123:2379 is healthy: successfully committed proposal: took = 12.168703ms
# 如果輸出上面信息，就說明集群部署成功。如果有問題第一步先看日誌：/var/log/message 或 journalctl -u etcd
安裝docker
下載安裝docker
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce-19.03.9-3.el7
配置docker鏡像源
mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
啟動並設置開機自啟
systemctl daemon-reload
systemctl start docker
systemctl enable docker
部署Master Node
生成kube-apiserver證書
1. 自簽證書頒發機構(CA)
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat > ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
# 生成證書
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
ls *pem
ca-key.pem ca.pem
使用自簽CA簽發kube-apiserver https 證書
cat /root/TLS/k8s/server-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.0.121",
"192.168.0.123",
"192.168.0.124",
"192.168.0.125",
"192.168.0.100",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
# 上述文件hosts字段中IP為所有Master/LB/VIP IP，一個都不能少！為了方便後期擴容可以多寫幾個預留的IP。
# 生成證書
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
ls server*pem
server-key.pem server.pem
下載解壓二進制包
wget https://dl.k8s.io/v1.18.4/kubernetes-server-linux-amd64.tar.gz
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/
部署kube-apiserver
cat /opt/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://192.168.0.121:2379,https://192.168.0.123:2379,https://192.168.0.124:2379 \
--bind-address=192.168.0.121 \
--secure-port=6443 \
--advertise-address=192.168.0.121 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
# –logtostderr：啟用日誌
# —v：日誌等級
# –log-dir：日誌目錄
# –etcd-servers：etcd集群地址
# –bind-address：監聽地址
# –secure-port：https安全端口
# –advertise-address：集群通告地址
# –allow-privileged：啟用授權
# –service-cluster-ip-range：Service虛擬IP地址段
# –enable-admission-plugins：准入控制模塊
# –authorization-mode：認證授權，啟用RBAC授權和節點自管理
# –enable-bootstrap-token-auth：啟用TLS bootstrap機制
# –token-auth-file：bootstrap token文件
# –service-node-port-range：Service nodeport類型默認分配端口範圍
# –kubelet-client-xxx：apiserver訪問kubelet客戶端證書
# –tls-xxx-file：apiserver https證書
# –etcd-xxxfile：連接Etcd集群證書
# –audit-log-xxx：審計日誌
拷貝剛生成證書
cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/
啟用TLS Bootstrapping機制
TLS Bootstraping：Master apiserver啟用TLS認證后，Node節點kubelet和kube-proxy要與kube-apiserver進行通信，必須使用CA簽發的有效證書才可以，當Node節點很多時，這種客戶端證書頒發需要大量工作，同樣也會增加集群擴展複雜度。為了簡化流程，Kubernetes引入了TLS bootstraping機制來自動頒發客戶端證書，kubelet會以一個低權限用戶自動向apiserver申請證書，kubelet的證書由apiserver動態簽署。所以強烈建議在Node上使用這種方式，目前主要用於kubelet，kube-proxy還是由我們統一頒發一個證書
TLS bootstraping 工作流程
創建上述配置文件中token文件
cat > /opt/kubernetes/cfg/token.csv << EOF
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF
# 格式：token，用戶名，UID，用戶組
token也可自行生成替換
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
systemd管理apiserver
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
啟動設置開機啟動
systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver
授權kubelet-bootstrap用戶允許請求證書
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
部署kube-controller-manager
創建配置文件
cat /opt/kubernetes/cfg/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
# –master：通過本地非安全本地端口8080連接apiserver。
# –leader-elect：當該組件啟動多個時，自動選舉（HA）
# –cluster-signing-cert-file/–cluster-signing-key-file：自動為kubelet頒發證書的CA，與apiserver保持一致
systemd管理controller-manager
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
啟動設置開機啟動
systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager
部署kube-scheduler
創建配置文件
cat /opt/kubernetes/cfg/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect --master=127.0.0.1:8080 --bind-address=127.0.0.1"
# –master：通過本地非安全本地端口8080連接apiserver。
# –leader-elect：當該組件啟動多個時，自動選舉（HA）
systemd管理scheduler
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
啟動並設置開機自啟動
systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler
查看集群狀態
# 所有組件都已經啟動成功，通過kubectl工具查看當前集群組件狀態：
kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
部署worker node
下面還是在Master Node上操作，即同時作為Worker Node
創建工作目錄並拷貝二進制文件
在所有worker node創建工作目錄
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
從master節點拷貝
cd kubernetes/server/bin
cp kubelet kube-proxy /opt/kubernetes/bin # 本地拷貝,註釋這裏操作還是master節點,
部署kubelet
創建配置文件
cat /opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=master \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
# –hostname-override：显示名稱，集群中唯一
# –network-plugin：啟用CNI
# –kubeconfig：空路徑，會自動生成，後面用於連接apiserver
# –bootstrap-kubeconfig：首次啟動向apiserver申請證書
# –config：配置參數文件
# –cert-dir：kubelet證書生成目錄
# –pod-infra-container-image：管理Pod網絡容器的鏡像
配置參數文件
cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
生成bootstrap.kubeconfig文件
KUBE_APISERVER="https://192.168.0.121:6443" # apiserver IP:PORT
TOKEN="c47ffb939f5ca36231d9e3121a252940" # 與token.csv里保持一致
# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
kubectl config set-credentials "kubelet-bootstrap" \
--token=${TOKEN} \
--kubeconfig=bootstrap.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user="kubelet-bootstrap" \
--kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
# 拷貝到配置文件路徑
cp bootstrap.kubeconfig /opt/kubernetes/cfg
systemd管理kubelet
cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
啟動並設置開機自啟
systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
批准kubelet證書並加入集群
# 查看kubelet證書請求
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A 6m3s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
# 批准申請
kubectl certificate approve node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A
# 查看節點
kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready <none> 123m v1.18.4
# 由於網絡插件還沒有部署，節點會沒有準備就緒 NotReady
部署kube-proxy
創建配置文件
cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF
創建參數文件
cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master
clusterCIDR: 10.0.0.0/24
EOF
生成kube-proxy.kubeconfig文件
生成kube-proxy證書
# 切換工作目錄
cd TLS/k8s
# 創建證書請求文件
cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
# 生成證書
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
ls kube-proxy*pem
kube-proxy-key.pem kube-proxy.pem
生成kubeconfig文件
KUBE_APISERVER="https://192.168.0.121:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=./kube-proxy.pem \
--client-key=./kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
# 拷貝配置文件到指定路徑
cp kube-proxy.kubeconfig /opt/kubernetes/cfg/
systemd管理kube-proxy
cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
啟動並設置開機啟動
systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy
部署CNI網絡
準備二進制文件
wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
# 解壓二進制文件並移動到默認工作目錄
mkdir /opt/cni/bin
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin
# 部署cni網絡
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml
# 默認鏡像地址無法訪問，修改為docker hub鏡像倉庫。
kubectl apply -f kube-flannel.yml
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-amd64-2pc95 1/1 Running 0 72s
kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready <none> 41m v1.18.4
# 部署好網絡插件,Node準備就緒
授權apiserver訪問kubelet
cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
- pods/log
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
EOF
kubectl apply -f apiserver-to-kubelet-rbac.yaml
新增加Worker Node
拷貝已部署好的Node相關文件到新節點
scp -r /opt/kubernetes/ node1:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service node1:/usr/lib/systemd/system
scp -r /opt/cni/ node1:/opt/
scp /opt/kubernetes/ssl/ca.pem node1:/opt/kubernetes/ssl
刪除kubelet證書和kubeconfig文件
rm -f /opt/kubernetes/cfg/kubelet.kubeconfig
rm -f /opt/kubernetes/ssl/kubelet*
# 這幾個文件是證書申請審批后自動生成的，每個Node不同，必須刪除重新生成。
修改主機名並設置開機自啟動
vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=node1
vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: node1
# 啟動並設置開機啟動
systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy
再master上批准Node kubelet證書申請
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-4zTjsaVSrhuyhIGqsefxzVoZDCNKei-aE2jyTP81Uro 89s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
kubectl certificate approve node-csr-4zTjsaVSrhuyhIGqsefxzVoZDCNKei-aE2jyTP81Uro
kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready <none> 138m v1.18.4
node1 Ready <none> 120m v1.18.4
node2 Ready <none> 112m v1.18.4
部署Dashboard
下載dashboard.yaml文件
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
修改yaml配置文件使其端口暴露外部訪問
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
# 默認Dashboard只能集群內部訪問，修改Service為NodePort類型，暴露到外部：
vi recommended.yaml
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
nodePort: 30001 # 修改這裏
type: NodePort # 修改這裏
selector:
k8s-app: kubernetes-dashboard
kubectl apply -f recommended.yaml
kubectl get pods,svc -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-694557449d-69x7g 1/1 Running 0 111m
pod/kubernetes-dashboard-9774cc786-kwgkt 1/1 Running 0 111m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.0.0.3 <none> 8000/TCP 111m
service/kubernetes-dashboard NodePort 10.0.0.122 <none> 443:30001/TCP 111m
創建service account並綁定默認cluster-admin管理員集群角色
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
# 接下來訪問https://node ip:30001
# 然後將上面過濾出來的token複製上面即可訪問dashboard
# 我們可以部署個Nginx測試下集群可用性
kubectl run --generator=run-pod/v1 nginx-test2 --image=daocloud.io/library/nginx --port=80 --replicas=1
kubectl get pods -o wide
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-test2 1/1 Running 0 89m 10.244.2.2 node2 <none> <none>
# 我們去相應的節點訪問指定IP即可訪問
[root@node1 ~]# curl -I -s 10.244.2.2 |grep 200
HTTP/1.1 200 OK
部署CoreDNS
CoreDNS用於集群內部Service名稱解析
kubectl apply -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns
configmap/coredns created
deployment.apps/coredns created
DNS解析測試
kubectl run -it --rm dns-test --image=busybox:1.28.4 sh
If you don't see a command prompt, try pressing enter.
/ # nslookup kubernetes
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
Name: kubernetes
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
此篇文章借鑒於公眾號DevOps技術棧 ，作者阿良
本站聲明:網站內容來源於博客園,如有侵權,請聯繫我們,我們將及時處理
【其他文章推薦】
※新北清潔公司,居家、辦公、裝潢細清專業服務
※別再煩惱如何寫文案,掌握八大原則!
※網頁設計一頭霧水該從何著手呢? 台北網頁設計公司幫您輕鬆架站!
※超省錢租車方案
※教你寫出一流的銷售文案?
美國豪華電動車製造商特斯拉 (Tesla) 宣佈，位於加州費利蒙 (Fremont) 的廠房最近進行了一次大改造、大舉採納機器人，預期到了 2015 年底年產能可望從今年的 35,000 台進一步攀升 50%。 特斯拉 Fremont 組裝廠最近一度暫停生產 2 週、以便升級組裝設備，而最大的變化就在組裝線改採先進的機器人，可把整輛車以最精確的方式舉高，還能節省空間。這些最新的機器人應該很快就能替汽車安裝電池，讓人類不必再負擔如此吃重的工作，同時還能將安裝的時間從原本的 4 分鐘縮短至 2 分鐘。 特斯拉並以電影「X 戰警」的角色為這些機器人命名：查爾斯 (Xavier) 是組裝線入口處的舉重機，任務是把汽車從電軌挪至地板；金鋼狼 (Wolverine)、猛獸 (Beast) 是舉重力量更為強大的機器人；暴風女 (Storm)、鋼人 (Colossus) 位在底盤組裝線的最末端；火神 (Vulcan)、法官 (Havok) 則組隊把車送回電軌。 除了機器人之外，特斯拉也把組裝線改造的更為流暢與自動化，現在一週已能生產約 1,000 輛車，未來有機會透過小幅度的修正把產能進一步拉高。特斯拉本次在 2 週的改造期間內總計安裝了 10 台全球最大的機器人。
美國豪華電動車製造商特斯拉 (Tesla ) 為何要將電動休旅車 Model X 的發表時間點從原先的 2015 年第二季延後到第三季？大摩（Morgan Stanley） 認為，這可能是與 Model X 的鷹翼門 (falcon doors) 有關。 MarketWatch、ValueWalk 18 日報導，大摩分析師 Adam Jonas 17 日發表研究報告指出，鷹翼門的設計與機械原理頗為複雜，它擁有 2 個鏈接點，而傳統的鷗翼門 (gull-wing doors) 則只有一個鏈接點。 使用鷹翼門的目的是要讓 Model X 在狹窄的空間 (例如停車場與車庫) 也能順利開門。不過，大摩認為，鷹翼門有 2 個鏈接點，開啟後所需的空間恐怕會比鷗翼門還要多。 大摩猜測，特斯拉可能會再度推延 Model X 的發表時程，也許會遞延至 2017 年初。不過，該證券認為特斯拉不會輕易放棄鷗翼門，因為這會是 Model X 的一大賣點。 （Source：
