云原生环境搭建第一篇:k8s-1.27.3集群部署

总共分几个章节

  1. k8s集群搭建
  2. harbor仓库搭建
  3. ceph存储搭建
  4. kubesphere搭建
  5. devops平台搭建
  6. 部署一个java微服务架构应用上线
  7. 改造为服务网格

目前为第一章节


初步搭建一个四节点k8s1.27.2集群。先准备好机器

host hostname os role hardware
10.20.24.51 10-20-24-51 centos7.9 control-plane cpu:4c 内存: 16G 硬盘:500G
10.20.24.54 10-20-24-54 centos7.9 worker cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T
10.20.24.55 10-20-24-55 centos7.9 worker cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T
10.20.24.56 10-20-24-56 centos7.9 worker cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T

预留了52、53节点,后续扩容集群做contrl-plane节点高可用。

所有work节点各分配一块1T硬盘,后续做ceph存储用。

# 0. 准备工作

下面的操作所有节点全部执行,后面如果要给集群新增节点也要做这个操作

# 0.1 所有节点全部关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
systemctl is-enabled firewalld
1
2
3

# 0.2 配置ntp server同步时间

yum install ntp;
vi /etc/ntp.conf # 新增ntp server 10.20.24.15
1
2

# 0.3 永久关闭selinux

sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
1

# 0.4 关闭swap分区

sed -i '/swap/s/^/#/g' /etc/fstab
1

# 0.5 配置hosts

cat >>/etc/hosts <<EOF

10.20.24.51 10-20-24-51
10.20.24.54 10-20-24-54
10.20.24.55 10-20-24-55
10.20.24.56 10-20-24-56
EOF
1
2
3
4
5
6
7

# 0.6 重启

reboot
1

# 1. 配置k8s环境

下面的操作所有节点全部执行,后面如果要给集群新增节点也要做这个操作

# 1.1 配置内核参数

cat > /etc/sysctl.d/k8s.conf <<EOF

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# 使配置生效
sysctl --system
1
2
3
4
5
6
7
8
9

这些配置参数的含义是:

  1. net.bridge.bridge-nf-call-ip6tables = 1: 当通过桥接网络接收到IPv6数据包时,将调用ip6tables的规则进行处理。
  2. net.bridge.bridge-nf-call-iptables = 1: 当通过桥接网络接收到IPv4数据包时,将调用iptables的规则进行处理。
  3. net.ipv4.ip_forward = 1: 允许IPv4的数据包转发,即使数据包的目标不是本机。

k8s通过iptables实现服务发现和网络流量路由,pod通信。这一步很重要。没有设置的话会导致集群网络通信故障,如pod无法通信。

# 1.2 安装并设置开机加载ipvs相关内核模块

yum -y install conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git
cat > /etc/modules-load.d/ipvs.conf <<EOF

# Load IPVS at boot
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
nf_conntrack_ipv4
EOF

systemctl enable --now systemd-modules-load
1
2
3
4
5
6
7
8
9
10
11
12
13
  • ip_vsip_vs_rrip_vs_wrrip_vs_sh 是IPVS相关的内核模块。它们提供了不同的负载均衡算法(round-robin,加权轮询,最短任务优先)。
  • nf_conntracknf_conntrack_ipv4 是用于网络连接跟踪的内核模块,这在防火墙和NAT中非常重要。

确认一下是否加载成功

[root@10-20-24-51 ~]# lsmod |egrep "ip_vs|nf_conntrack_ipv4"
nf_conntrack_ipv4      15053  26
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
ip_vs_sh               12688  0
ip_vs_wrr              12697  0
ip_vs_rr               12600  0
ip_vs                 145458  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          139264  10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c              12644  4 xfs,ip_vs,nf_nat,nf_conntrack
1
2
3
4
5
6
7
8
9

# 1.3 安装containerd

顺别介绍一下历史背景。早期docker势大,但docker没有实现docker借口,k8s只能用dockershim做适配器来兼容docker,使其可以接入cri,这个dockershim在k8s1.24版本就被放弃维护了。containerd是从docker中分离出来的开源项目,强调简单性、健壮性和可移植性。它负责以下工作

  • 管理容器的生命周期(从创建容器到销毁容器)
  • 拉取/推送容器镜像
  • 存储管理(管理镜像及容器数据的存储)
  • 调用 runc 运行容器(与 runc 等容器运行时交互,runc是oci 开放容器标准的一个实现。oci就是创建容器需要做一些 namespaces 和 cgroups 的配置,以及挂载 root 文件系统等操作的规范)
  • 管理容器网络接口及网络
yum -y install yum-utils device-mapper-persistent-data lvm2
# 添加阿里源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 配置 containerd 
cat >>/etc/modules-load.d/containerd.conf <<EOF

overlay
br_netfilter
EOF

# 立刻加载 overlay模块
modprobe overlay
# 立刻加载 br_netfilter模块
modprobe br_netfilter
# 安装containerd
yum install containerd.io -y
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
  • overlay 是一个文件系统类型,它支持在不改变底层文件的情况下,将改动保存在另一个分离的文件层。它常用于 Docker 和其他容器运行时中,用来创建容器的文件系统。
  • br_netfilter 是一个网络相关的内核模块,它允许 iptables 和其他网络工具对桥接流量进行过滤。这在 Kubernetes 网络设置中很重要,特别是在使用 overlay 网络(如 flannel、Calico 等)时。

# 1.4 配置containerd

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 使用systemd管理cgroups
sed -i '/SystemdCgroup/s/false/true/g' /etc/containerd/config.toml
# 配置sadnbox image从阿里云拉取
sed -i '/sandbox_image/s/registry.k8s.io/registry.aliyuncs.com\/google_containers/g' /etc/containerd/config.toml

# 启动containerd
systemctl enable containerd
systemctl start containerd
1
2
3
4
5
6
7
8
9
10

# 2. 安装kubeamd、kubelet、kubectl

下面的操作所有节点全部执行,后面如果要给集群新增节点也要做这个操作

# 2.1 添加阿里源

cat >/etc/yum.repos.d/kubernetes.repo <<EOF

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
1
2
3
4
5
6
7
8
9
10

# 2.2 安装并启动

# 查看可用版本
yum list kubelet --showduplicates |grep 1.27

# 开始安装   这篇文档写下时,最新版本为1.27.3  我直接安装的最新版
yum -y install kubectl-1.27.3 kubelet-1.27.3 kubeadm-1.27.3

# 启动
systemctl enable kubelet
systemctl start kubelet
1
2
3
4
5
6
7
8
9

# 3. 部署master节点

以下操作只在master节点执行

# 使用kubeadm初始化

# 查看所需镜像
kubeadm config images list --kubernetes-version=v1.27.3
registry.k8s.io/kube-apiserver:v1.27.3
registry.k8s.io/kube-controller-manager:v1.27.3
registry.k8s.io/kube-scheduler:v1.27.3
registry.k8s.io/kube-proxy:v1.27.3
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/coredns/coredns:v1.10.1

# 初始化
kubeadm init --kubernetes-version=1.27.3 \
--apiserver-advertise-address=10.20.24.51 \
--image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=172.16.0.0/16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
  • apiserver-advertise-address写master的ip
  • pod-network-cidr写个不冲突的网段
  • image-repository指定从阿里云拉取镜像

命令执行完成后会返回以一长段内容,主要看最后部分

Your Kubernetes control-plane has initialized successfully!

 

To start using your cluster, you need to run the following as a regular user:

 

    mkdir -p $HOME/.kube

    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

    sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

    export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

 

kubeadm join 10.20.24.51:6443 --token l906wz.0fydt3hcfbogwlo9 \
        --discovery-token-ca-cert-hash sha256:2604d3aab372a483b26bcbdafdb54d7746226975c3a317db07d94eccdfca51be
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

然后按提示操作:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
1
2
3
4

$HOME/.kube/config这个文件可以复制到本地,这样就可以在本地执行kubectl命令了,前提是先安装kubectl

ytg@ytgdeMacBook-Pro ~ % kubectl get nodes
NAME          STATUS   ROLES           AGE   VERSION
10-20-24-51   Ready    control-plane   13d   v1.27.3
10-20-24-54   Ready    <none>          13d   v1.27.3
10-20-24-55   Ready    <none>          13d   v1.27.3
10-20-24-56   Ready    <none>          13d   v1.27.3
1
2
3
4
5
6

查看节点状态和pod就绪状态

kubectl get node
NAME                 STATUS          ROLES             AGE     VERSION
10-20-24-51          NotReady        control-plane    50s       v1.27.3
kubectl get pods -A
ytg@ytgdeMacBook-Pro ~ % kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
coredns-7bdc4cb885-fs2tz                   1/1     Pending   0             13d
coredns-7bdc4cb885-wk7c9                   1/1     Pending   0             13d
etcd-10-20-24-51                           1/1     Running   0             13d
kube-apiserver-10-20-24-51                 1/1     Running   0             13d
kube-controller-manager-10-20-24-51        1/1     Running   0             13d
kube-proxy-mfzmq                           1/1     Running   3 (25h ago)   13d
kube-scheduler-10-20-24-51                 1/1     Running   0             13d
1
2
3
4
5
6
7
8
9
10
11
12
13

10-20-24-51状态为NotReady是因为coredns没有启动,dns启动是因为没有网络插件

# 3.2 部署calico

kubectl apply -f https://docs.tigera.io/archive/v3.24/manifests/calico.yaml

ytg@ytgdeMacBook-Pro ~ % kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
calico-kube-controllers-6849cf9bcf-gv6xx   1/1     Running   0             13d
calico-node-2d7xx                          1/1     Running   0             13d
coredns-7bdc4cb885-fs2tz                   1/1     Running   0             13d
coredns-7bdc4cb885-wk7c9                   1/1     Running   0             13d
etcd-10-20-24-51                           1/1     Running   0             13d
kube-apiserver-10-20-24-51                 1/1     Running   0             13d
kube-controller-manager-10-20-24-51        1/1     Running   0             13d
kube-proxy-mfzmq                           1/1     Running   3 (25h ago)   13d
kube-scheduler-10-20-24-51                 1/1     Running   0             13d

kubectl get node
NAME                 STATUS          ROLES             AGE     VERSION
10-20-24-51          Ready        control-plane    		13d      v1.27.3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# 4. worker节点加入集群

# 所有worker节点都执行
kubeadm join 10.20.24.51:6443 --token l906wz.0fydt3hcfbogwlo9 \
        --discovery-token-ca-cert-hash sha256:2604d3aab372a483b26bcbdafdb54d7746226975c3a317db07d94eccdfca51be
1
2
3

查看状态

kubectl get nodes
NAME          STATUS   ROLES           AGE   VERSION
10-20-24-51   Ready    control-plane   13d   v1.27.3
10-20-24-54   Ready    <none>          13d   v1.27.3
10-20-24-55   Ready    <none>          13d   v1.27.3
10-20-24-56   Ready    <none>          13d   v1.27.3
1
2
3
4
5
6

# 5. 其他

# 5.1 安装命令补全

yum -y install bash-completion
echo "source <(kubectl completion bash)" >> /etc/profile
source /etc/profile
1
2
3

# 5.2 kubernetes-dashboardan安装

# 下载yaml
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

# 添加nodeport
vi recommended.yaml
1
2
3
4
5

修改如下内容

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort       # 增加内容
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30000  # 增加内容
  selector:
    k8s-app: kubernetes-dashboard
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 安装
kubectl apply -f recommended.yaml

# 查看进度
kubectl get all -n kubernetes-dashboard
NAME                                             READY   STATUS    RESTARTS      AGE
pod/dashboard-metrics-scraper-5cb4f4bb9c-h549p   1/1     Running   3 (26h ago)   13d
pod/kubernetes-dashboard-6967859bff-cm4tl        1/1     Running   4 (26h ago)   13d

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.108.31.72    <none>        8000/TCP        13d
service/kubernetes-dashboard        NodePort    10.102.47.173   <none>        443:30000/TCP   13d

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dashboard-metrics-scraper   1/1     1            1           13d
deployment.apps/kubernetes-dashboard        1/1     1            1           13d

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/dashboard-metrics-scraper-5cb4f4bb9c   1         1         1       13d
replicaset.apps/kubernetes-dashboard-6967859bff        1         1         1       13d
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

创建admin用户

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin
  namespace: kubernetes-dashboard  
---
apiVersion: v1
kind: Secret
metadata:
  name: kubernetes-dashboard-admin
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/service-account.name: "admin"
type: kubernetes.io/service-account-token
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 创建admin用户token
kubectl -n kubernetes-dashboard create token admin

# 获取token
 Token=$(kubectl -n kubernetes-dashboard get secret |awk '/kubernetes-dashboard-admin/ {print $1}')
 kubectl describe secrets -n kubernetes-dashboard ${Token} |grep token |awk 'NR==NF {print $2}'
1
2
3
4
5
6

然后就可以使用token登陆了,地址是 集群任意节点IP:30000

下一篇更新harbor

上次更新: 2023/7/12 16:50:43