云原生环境搭建第一篇：k8s-1.27.3集群部署

总共分几个章节

1. k8s集群搭建
2. harbor仓库搭建
3. ceph存储搭建
4. kubesphere搭建
5. devops平台搭建
6. 部署一个java微服务架构应用上线
7. 改造为服务网格

目前为第一章节

---

初步搭建一个四节点k8s1.27.2集群。先准备好机器

host	hostname	os	role	hardware
10.20.24.51	10-20-24-51	centos7.9	control-plane	cpu:4c 内存: 16G 硬盘:500G
10.20.24.54	10-20-24-54	centos7.9	worker	cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T
10.20.24.55	10-20-24-55	centos7.9	worker	cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T
10.20.24.56	10-20-24-56	centos7.9	worker	cpu:4c 内存: 16G 硬盘1:500G 硬盘2:1T

预留了52、53节点，后续扩容集群做contrl-plane节点高可用。

所有work节点各分配一块1T硬盘，后续做ceph存储用。

0. 准备工作

下面的操作所有节点全部执行，后面如果要给集群新增节点也要做这个操作

0.1 所有节点全部关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
systemctl is-enabled firewalld

0.2 配置ntp server同步时间

yum install ntp;
vi /etc/ntp.conf # 新增ntp server 10.20.24.15

0.3 永久关闭selinux

sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

0.4 关闭swap分区

sed -i '/swap/s/^/#/g' /etc/fstab

0.5 配置hosts

cat >>/etc/hosts <<EOF

10.20.24.51 10-20-24-51
10.20.24.54 10-20-24-54
10.20.24.55 10-20-24-55
10.20.24.56 10-20-24-56
EOF

0.6 重启

reboot

1. 配置k8s环境

下面的操作所有节点全部执行，后面如果要给集群新增节点也要做这个操作

1.1 配置内核参数

cat > /etc/sysctl.d/k8s.conf <<EOF

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# 使配置生效
sysctl --system

这些配置参数的含义是：

1. net.bridge.bridge-nf-call-ip6tables = 1： 当通过桥接网络接收到IPv6数据包时，将调用ip6tables的规则进行处理。
2. net.bridge.bridge-nf-call-iptables = 1： 当通过桥接网络接收到IPv4数据包时，将调用iptables的规则进行处理。
3. net.ipv4.ip_forward = 1： 允许IPv4的数据包转发，即使数据包的目标不是本机。

k8s通过iptables实现服务发现和网络流量路由，pod通信。这一步很重要。没有设置的话会导致集群网络通信故障，如pod无法通信。

1.2 安装并设置开机加载ipvs相关内核模块

yum -y install conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git
cat > /etc/modules-load.d/ipvs.conf <<EOF

# Load IPVS at boot
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
nf_conntrack_ipv4
EOF

systemctl enable --now systemd-modules-load

• ip_vs，ip_vs_rr，ip_vs_wrr，ip_vs_sh 是IPVS相关的内核模块。它们提供了不同的负载均衡算法（round-robin，加权轮询，最短任务优先）。
• nf_conntrack 和 nf_conntrack_ipv4 是用于网络连接跟踪的内核模块，这在防火墙和NAT中非常重要。

确认一下是否加载成功

[root@10-20-24-51 ~]# lsmod |egrep "ip_vs|nf_conntrack_ipv4"
nf_conntrack_ipv4      15053  26
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
ip_vs_sh               12688  0
ip_vs_wrr              12697  0
ip_vs_rr               12600  0
ip_vs                 145458  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          139264  10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c              12644  4 xfs,ip_vs,nf_nat,nf_conntrack

1.3 安装containerd

顺别介绍一下历史背景。早期docker势大，但docker没有实现docker借口，k8s只能用dockershim做适配器来兼容docker，使其可以接入cri，这个dockershim在k8s1.24版本就被放弃维护了。containerd是从docker中分离出来的开源项目，强调简单性、健壮性和可移植性。它负责以下工作

• 管理容器的生命周期（从创建容器到销毁容器）
• 拉取/推送容器镜像
• 存储管理（管理镜像及容器数据的存储）
• 调用 runc 运行容器（与 runc 等容器运行时交互，runc是oci 开放容器标准的一个实现。oci就是创建容器需要做一些 namespaces 和 cgroups 的配置，以及挂载 root 文件系统等操作的规范）
• 管理容器网络接口及网络

yum -y install yum-utils device-mapper-persistent-data lvm2
# 添加阿里源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 配置 containerd 
cat >>/etc/modules-load.d/containerd.conf <<EOF

overlay
br_netfilter
EOF

# 立刻加载 overlay模块
modprobe overlay
# 立刻加载 br_netfilter模块
modprobe br_netfilter
# 安装containerd
yum install containerd.io -y

• overlay 是一个文件系统类型，它支持在不改变底层文件的情况下，将改动保存在另一个分离的文件层。它常用于 Docker 和其他容器运行时中，用来创建容器的文件系统。
• br_netfilter 是一个网络相关的内核模块，它允许 iptables 和其他网络工具对桥接流量进行过滤。这在 Kubernetes 网络设置中很重要，特别是在使用 overlay 网络（如 flannel、Calico 等）时。

1.4 配置containerd

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 使用systemd管理cgroups
sed -i '/SystemdCgroup/s/false/true/g' /etc/containerd/config.toml
# 配置sadnbox image从阿里云拉取
sed -i '/sandbox_image/s/registry.k8s.io/registry.aliyuncs.com\/google_containers/g' /etc/containerd/config.toml

# 启动containerd
systemctl enable containerd
systemctl start containerd

2. 安装kubeamd、kubelet、kubectl

下面的操作所有节点全部执行，后面如果要给集群新增节点也要做这个操作

2.1 添加阿里源

cat >/etc/yum.repos.d/kubernetes.repo <<EOF

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2.2 安装并启动

# 查看可用版本
yum list kubelet --showduplicates |grep 1.27

# 开始安装   这篇文档写下时，最新版本为1.27.3  我直接安装的最新版
yum -y install kubectl-1.27.3 kubelet-1.27.3 kubeadm-1.27.3

# 启动
systemctl enable kubelet
systemctl start kubelet

3. 部署master节点

以下操作只在master节点执行

使用kubeadm初始化

# 查看所需镜像
kubeadm config images list --kubernetes-version=v1.27.3
registry.k8s.io/kube-apiserver:v1.27.3
registry.k8s.io/kube-controller-manager:v1.27.3
registry.k8s.io/kube-scheduler:v1.27.3
registry.k8s.io/kube-proxy:v1.27.3
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/coredns/coredns:v1.10.1

# 初始化
kubeadm init --kubernetes-version=1.27.3 \
--apiserver-advertise-address=10.20.24.51 \
--image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=172.16.0.0/16

• apiserver-advertise-address写master的ip
• pod-network-cidr写个不冲突的网段
• image-repository指定从阿里云拉取镜像

命令执行完成后会返回以一长段内容，主要看最后部分

Your Kubernetes control-plane has initialized successfully!

 

To start using your cluster, you need to run the following as a regular user:

 

    mkdir -p $HOME/.kube

    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

    sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

    export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

 

kubeadm join 10.20.24.51:6443 --token l906wz.0fydt3hcfbogwlo9 \
        --discovery-token-ca-cert-hash sha256:2604d3aab372a483b26bcbdafdb54d7746226975c3a317db07d94eccdfca51be

然后按提示操作:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf

$HOME/.kube/config这个文件可以复制到本地，这样就可以在本地执行kubectl命令了，前提是先安装kubectl

ytg@ytgdeMacBook-Pro ~ % kubectl get nodes
NAME          STATUS   ROLES           AGE   VERSION
10-20-24-51   Ready    control-plane   13d   v1.27.3
10-20-24-54   Ready    <none>          13d   v1.27.3
10-20-24-55   Ready    <none>          13d   v1.27.3
10-20-24-56   Ready    <none>          13d   v1.27.3

查看节点状态和pod就绪状态

kubectl get node
NAME                 STATUS          ROLES             AGE     VERSION
10-20-24-51          NotReady        control-plane    50s       v1.27.3
kubectl get pods -A
ytg@ytgdeMacBook-Pro ~ % kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
coredns-7bdc4cb885-fs2tz                   1/1     Pending   0             13d
coredns-7bdc4cb885-wk7c9                   1/1     Pending   0             13d
etcd-10-20-24-51                           1/1     Running   0             13d
kube-apiserver-10-20-24-51                 1/1     Running   0             13d
kube-controller-manager-10-20-24-51        1/1     Running   0             13d
kube-proxy-mfzmq                           1/1     Running   3 (25h ago)   13d
kube-scheduler-10-20-24-51                 1/1     Running   0             13d

10-20-24-51状态为NotReady是因为coredns没有启动，dns启动是因为没有网络插件

3.2 部署calico

kubectl apply -f https://docs.tigera.io/archive/v3.24/manifests/calico.yaml

ytg@ytgdeMacBook-Pro ~ % kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
calico-kube-controllers-6849cf9bcf-gv6xx   1/1     Running   0             13d
calico-node-2d7xx                          1/1     Running   0             13d
coredns-7bdc4cb885-fs2tz                   1/1     Running   0             13d
coredns-7bdc4cb885-wk7c9                   1/1     Running   0             13d
etcd-10-20-24-51                           1/1     Running   0             13d
kube-apiserver-10-20-24-51                 1/1     Running   0             13d
kube-controller-manager-10-20-24-51        1/1     Running   0             13d
kube-proxy-mfzmq                           1/1     Running   3 (25h ago)   13d
kube-scheduler-10-20-24-51                 1/1     Running   0             13d

kubectl get node
NAME                 STATUS          ROLES             AGE     VERSION
10-20-24-51          Ready        control-plane    		13d      v1.27.3

4. worker节点加入集群

# 所有worker节点都执行
kubeadm join 10.20.24.51:6443 --token l906wz.0fydt3hcfbogwlo9 \
        --discovery-token-ca-cert-hash sha256:2604d3aab372a483b26bcbdafdb54d7746226975c3a317db07d94eccdfca51be

查看状态

kubectl get nodes
NAME          STATUS   ROLES           AGE   VERSION
10-20-24-51   Ready    control-plane   13d   v1.27.3
10-20-24-54   Ready    <none>          13d   v1.27.3
10-20-24-55   Ready    <none>          13d   v1.27.3
10-20-24-56   Ready    <none>          13d   v1.27.3

5. 其他

5.1 安装命令补全

yum -y install bash-completion
echo "source <(kubectl completion bash)" >> /etc/profile
source /etc/profile

5.2 kubernetes-dashboardan安装

# 下载yaml
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

# 添加nodeport
vi recommended.yaml

修改如下内容

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort       # 增加内容
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30000  # 增加内容
  selector:
    k8s-app: kubernetes-dashboard

# 安装
kubectl apply -f recommended.yaml

# 查看进度
kubectl get all -n kubernetes-dashboard
NAME                                             READY   STATUS    RESTARTS      AGE
pod/dashboard-metrics-scraper-5cb4f4bb9c-h549p   1/1     Running   3 (26h ago)   13d
pod/kubernetes-dashboard-6967859bff-cm4tl        1/1     Running   4 (26h ago)   13d

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.108.31.72    <none>        8000/TCP        13d
service/kubernetes-dashboard        NodePort    10.102.47.173   <none>        443:30000/TCP   13d

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dashboard-metrics-scraper   1/1     1            1           13d
deployment.apps/kubernetes-dashboard        1/1     1            1           13d

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/dashboard-metrics-scraper-5cb4f4bb9c   1         1         1       13d
replicaset.apps/kubernetes-dashboard-6967859bff        1         1         1       13d

创建admin用户

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin
  namespace: kubernetes-dashboard  
---
apiVersion: v1
kind: Secret
metadata:
  name: kubernetes-dashboard-admin
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/service-account.name: "admin"
type: kubernetes.io/service-account-token

# 创建admin用户token
kubectl -n kubernetes-dashboard create token admin

# 获取token
 Token=$(kubectl -n kubernetes-dashboard get secret |awk '/kubernetes-dashboard-admin/ {print $1}')
 kubectl describe secrets -n kubernetes-dashboard ${Token} |grep token |awk 'NR==NF {print $2}'

然后就可以使用token登陆了，地址是 集群任意节点IP:30000

下一篇更新harbor