云原生环境搭建第一篇:k8s-1.27.3集群部署
总共分几个章节
- k8s集群搭建
- harbor仓库搭建
- ceph存储搭建
- kubesphere搭建
- devops平台搭建
- 部署一个java微服务架构应用上线
- 改造为服务网格
目前为第一章节
初步搭建一个四节点k8s1.27.2集群。先准备好机器
host | hostname | os | role | hardware |
---|---|---|---|---|
10.20.24.51 | 10-20-24-51 | centos7.9 | control-plane | cpu:4c 内存: 16G 硬盘:500G |
10.20.24.54 | 10-20-24-54 | centos7.9 | worker | cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T |
10.20.24.55 | 10-20-24-55 | centos7.9 | worker | cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T |
10.20.24.56 | 10-20-24-56 | centos7.9 | worker | cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T |
预留了52、53节点,后续扩容集群做contrl-plane节点高可用。
所有work节点各分配一块1T硬盘,后续做ceph存储用。
# 0. 准备工作
下面的操作所有节点全部执行,后面如果要给集群新增节点也要做这个操作
# 0.1 所有节点全部关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
systemctl is-enabled firewalld
1
2
3
2
3
# 0.2 配置ntp server同步时间
yum install ntp;
vi /etc/ntp.conf # 新增ntp server 10.20.24.15
1
2
2
# 0.3 永久关闭selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
1
# 0.4 关闭swap分区
sed -i '/swap/s/^/#/g' /etc/fstab
1
# 0.5 配置hosts
cat >>/etc/hosts <<EOF
10.20.24.51 10-20-24-51
10.20.24.54 10-20-24-54
10.20.24.55 10-20-24-55
10.20.24.56 10-20-24-56
EOF
1
2
3
4
5
6
7
2
3
4
5
6
7
# 0.6 重启
reboot
1
# 1. 配置k8s环境
下面的操作所有节点全部执行,后面如果要给集群新增节点也要做这个操作
# 1.1 配置内核参数
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# 使配置生效
sysctl --system
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
这些配置参数的含义是:
net.bridge.bridge-nf-call-ip6tables = 1
: 当通过桥接网络接收到IPv6数据包时,将调用ip6tables
的规则进行处理。net.bridge.bridge-nf-call-iptables = 1
: 当通过桥接网络接收到IPv4数据包时,将调用iptables
的规则进行处理。net.ipv4.ip_forward = 1
: 允许IPv4的数据包转发,即使数据包的目标不是本机。
k8s通过iptables实现服务发现和网络流量路由,pod通信。这一步很重要。没有设置的话会导致集群网络通信故障,如pod无法通信。
# 1.2 安装并设置开机加载ipvs相关内核模块
yum -y install conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git
cat > /etc/modules-load.d/ipvs.conf <<EOF
# Load IPVS at boot
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
nf_conntrack_ipv4
EOF
systemctl enable --now systemd-modules-load
1
2
3
4
5
6
7
8
9
10
11
12
13
2
3
4
5
6
7
8
9
10
11
12
13
ip_vs
,ip_vs_rr
,ip_vs_wrr
,ip_vs_sh
是IPVS相关的内核模块。它们提供了不同的负载均衡算法(round-robin,加权轮询,最短任务优先)。nf_conntrack
和nf_conntrack_ipv4
是用于网络连接跟踪的内核模块,这在防火墙和NAT中非常重要。
确认一下是否加载成功
[root@10-20-24-51 ~]# lsmod |egrep "ip_vs|nf_conntrack_ipv4"
nf_conntrack_ipv4 15053 26
nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
ip_vs_sh 12688 0
ip_vs_wrr 12697 0
ip_vs_rr 12600 0
ip_vs 145458 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 139264 10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
# 1.3 安装containerd
顺别介绍一下历史背景。早期docker势大,但docker没有实现docker借口,k8s只能用dockershim做适配器来兼容docker,使其可以接入cri,这个dockershim在k8s1.24版本就被放弃维护了。containerd是从docker中分离出来的开源项目,强调简单性、健壮性和可移植性。它负责以下工作
- 管理容器的生命周期(从创建容器到销毁容器)
- 拉取/推送容器镜像
- 存储管理(管理镜像及容器数据的存储)
- 调用 runc 运行容器(与 runc 等容器运行时交互,runc是oci 开放容器标准的一个实现。oci就是创建容器需要做一些 namespaces 和 cgroups 的配置,以及挂载 root 文件系统等操作的规范)
- 管理容器网络接口及网络
yum -y install yum-utils device-mapper-persistent-data lvm2
# 添加阿里源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 配置 containerd
cat >>/etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
# 立刻加载 overlay模块
modprobe overlay
# 立刻加载 br_netfilter模块
modprobe br_netfilter
# 安装containerd
yum install containerd.io -y
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
overlay
是一个文件系统类型,它支持在不改变底层文件的情况下,将改动保存在另一个分离的文件层。它常用于 Docker 和其他容器运行时中,用来创建容器的文件系统。br_netfilter
是一个网络相关的内核模块,它允许 iptables 和其他网络工具对桥接流量进行过滤。这在 Kubernetes 网络设置中很重要,特别是在使用 overlay 网络(如 flannel、Calico 等)时。
# 1.4 配置containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 使用systemd管理cgroups
sed -i '/SystemdCgroup/s/false/true/g' /etc/containerd/config.toml
# 配置sadnbox image从阿里云拉取
sed -i '/sandbox_image/s/registry.k8s.io/registry.aliyuncs.com\/google_containers/g' /etc/containerd/config.toml
# 启动containerd
systemctl enable containerd
systemctl start containerd
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
# 2. 安装kubeamd、kubelet、kubectl
下面的操作所有节点全部执行,后面如果要给集群新增节点也要做这个操作
# 2.1 添加阿里源
cat >/etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
# 2.2 安装并启动
# 查看可用版本
yum list kubelet --showduplicates |grep 1.27
# 开始安装 这篇文档写下时,最新版本为1.27.3 我直接安装的最新版
yum -y install kubectl-1.27.3 kubelet-1.27.3 kubeadm-1.27.3
# 启动
systemctl enable kubelet
systemctl start kubelet
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
# 3. 部署master节点
以下操作只在master节点执行
# 使用kubeadm初始化
# 查看所需镜像
kubeadm config images list --kubernetes-version=v1.27.3
registry.k8s.io/kube-apiserver:v1.27.3
registry.k8s.io/kube-controller-manager:v1.27.3
registry.k8s.io/kube-scheduler:v1.27.3
registry.k8s.io/kube-proxy:v1.27.3
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/coredns/coredns:v1.10.1
# 初始化
kubeadm init --kubernetes-version=1.27.3 \
--apiserver-advertise-address=10.20.24.51 \
--image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=172.16.0.0/16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiserver-advertise-address
写master的ippod-network-cidr
写个不冲突的网段image-repository
指定从阿里云拉取镜像
命令执行完成后会返回以一长段内容,主要看最后部分
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.20.24.51:6443 --token l906wz.0fydt3hcfbogwlo9 \
--discovery-token-ca-cert-hash sha256:2604d3aab372a483b26bcbdafdb54d7746226975c3a317db07d94eccdfca51be
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
然后按提示操作:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
1
2
3
4
2
3
4
$HOME/.kube/config
这个文件可以复制到本地,这样就可以在本地执行kubectl命令了,前提是先安装kubectlytg@ytgdeMacBook-Pro ~ % kubectl get nodes NAME STATUS ROLES AGE VERSION 10-20-24-51 Ready control-plane 13d v1.27.3 10-20-24-54 Ready <none> 13d v1.27.3 10-20-24-55 Ready <none> 13d v1.27.3 10-20-24-56 Ready <none> 13d v1.27.3
1
2
3
4
5
6
查看节点状态和pod就绪状态
kubectl get node
NAME STATUS ROLES AGE VERSION
10-20-24-51 NotReady control-plane 50s v1.27.3
kubectl get pods -A
ytg@ytgdeMacBook-Pro ~ % kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7bdc4cb885-fs2tz 1/1 Pending 0 13d
coredns-7bdc4cb885-wk7c9 1/1 Pending 0 13d
etcd-10-20-24-51 1/1 Running 0 13d
kube-apiserver-10-20-24-51 1/1 Running 0 13d
kube-controller-manager-10-20-24-51 1/1 Running 0 13d
kube-proxy-mfzmq 1/1 Running 3 (25h ago) 13d
kube-scheduler-10-20-24-51 1/1 Running 0 13d
1
2
3
4
5
6
7
8
9
10
11
12
13
2
3
4
5
6
7
8
9
10
11
12
13
10-20-24-51状态为NotReady
是因为coredns没有启动,dns启动是因为没有网络插件
# 3.2 部署calico
kubectl apply -f https://docs.tigera.io/archive/v3.24/manifests/calico.yaml
ytg@ytgdeMacBook-Pro ~ % kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6849cf9bcf-gv6xx 1/1 Running 0 13d
calico-node-2d7xx 1/1 Running 0 13d
coredns-7bdc4cb885-fs2tz 1/1 Running 0 13d
coredns-7bdc4cb885-wk7c9 1/1 Running 0 13d
etcd-10-20-24-51 1/1 Running 0 13d
kube-apiserver-10-20-24-51 1/1 Running 0 13d
kube-controller-manager-10-20-24-51 1/1 Running 0 13d
kube-proxy-mfzmq 1/1 Running 3 (25h ago) 13d
kube-scheduler-10-20-24-51 1/1 Running 0 13d
kubectl get node
NAME STATUS ROLES AGE VERSION
10-20-24-51 Ready control-plane 13d v1.27.3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 4. worker节点加入集群
# 所有worker节点都执行
kubeadm join 10.20.24.51:6443 --token l906wz.0fydt3hcfbogwlo9 \
--discovery-token-ca-cert-hash sha256:2604d3aab372a483b26bcbdafdb54d7746226975c3a317db07d94eccdfca51be
1
2
3
2
3
查看状态
kubectl get nodes
NAME STATUS ROLES AGE VERSION
10-20-24-51 Ready control-plane 13d v1.27.3
10-20-24-54 Ready <none> 13d v1.27.3
10-20-24-55 Ready <none> 13d v1.27.3
10-20-24-56 Ready <none> 13d v1.27.3
1
2
3
4
5
6
2
3
4
5
6
# 5. 其他
# 5.1 安装命令补全
yum -y install bash-completion
echo "source <(kubectl completion bash)" >> /etc/profile
source /etc/profile
1
2
3
2
3
# 5.2 kubernetes-dashboardan安装
# 下载yaml
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
# 添加nodeport
vi recommended.yaml
1
2
3
4
5
2
3
4
5
修改如下内容
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort # 增加内容
ports:
- port: 443
targetPort: 8443
nodePort: 30000 # 增加内容
selector:
k8s-app: kubernetes-dashboard
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 安装
kubectl apply -f recommended.yaml
# 查看进度
kubectl get all -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-5cb4f4bb9c-h549p 1/1 Running 3 (26h ago) 13d
pod/kubernetes-dashboard-6967859bff-cm4tl 1/1 Running 4 (26h ago) 13d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.108.31.72 <none> 8000/TCP 13d
service/kubernetes-dashboard NodePort 10.102.47.173 <none> 443:30000/TCP 13d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dashboard-metrics-scraper 1/1 1 1 13d
deployment.apps/kubernetes-dashboard 1/1 1 1 13d
NAME DESIRED CURRENT READY AGE
replicaset.apps/dashboard-metrics-scraper-5cb4f4bb9c 1 1 1 13d
replicaset.apps/kubernetes-dashboard-6967859bff 1 1 1 13d
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
创建admin用户
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin
namespace: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
name: kubernetes-dashboard-admin
namespace: kubernetes-dashboard
annotations:
kubernetes.io/service-account.name: "admin"
type: kubernetes.io/service-account-token
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 创建admin用户token
kubectl -n kubernetes-dashboard create token admin
# 获取token
Token=$(kubectl -n kubernetes-dashboard get secret |awk '/kubernetes-dashboard-admin/ {print $1}')
kubectl describe secrets -n kubernetes-dashboard ${Token} |grep token |awk 'NR==NF {print $2}'
1
2
3
4
5
6
2
3
4
5
6
然后就可以使用token登陆了,地址是 集群任意节点IP:30000
下一篇更新harbor