您当前的位置： 首页 > 慢生活 > 程序人生 网站首页程序人生

64、Kubernetes - Helm 及其它功能性组件 - EFK 日志_[]

发布时间：2023-02-04 10:44:02编辑：雪饮阅读（）

Step1

集群pod日志路径

[root@k8s-master01 ~]# ls /var/log/containers/

coredns-5c98db65d4-d78lt_kube-system_coredns-1f8e1e32095ca480c01c2e28aaceb67357a                                                                                                    7182dafdf15250c732d93623a5a82.log

coredns-5c98db65d4-d78lt_kube-system_coredns-bfb01835837b5e7f078924f1dc8086f3fea                                                                                                    4aaf974e2b970cdee9bb064fffcb2.log

coredns-5c98db65d4-smd45_kube-system_coredns-c645824e2985280f7d262f2ff63e8343fcb                                                                                                    ed55e8296f0179090dd1cb7d97935.log

coredns-5c98db65d4-smd45_kube-system_coredns-f77c1133fa6b38c7ac0eb354f07125875f3                                                                                                    cdb0ce03bf8b6f8bf53db28500963.log

etcd-k8s-master01_kube-system_etcd-298740f542cfdc295c4d444d7c9399b151379333b6fb0                                                                                                    e554cfb4822702e3e28.log

etcd-k8s-master01_kube-system_etcd-f959b59b93097da59676aa5faf6b19ac303f00e92168f                                                                                                    324e17828e21d247a4a.log

kube-apiserver-k8s-master01_kube-system_kube-apiserver-23dae05d015396725f6e15fe5                                                                                                    2af4e6f5dcefe1db63fae73a91140dd8a43ff74.log

kube-apiserver-k8s-master01_kube-system_kube-apiserver-fc8d0e0383e7c4b7a19e5cca6                                                                                                    96c04254367ccf4a2a27b4221f0a4b9ed5a1274.log

kube-controller-manager-k8s-master01_kube-system_kube-controller-manager-761b291                                                                                                    ad4577e887207f24c58e81acc843ac32cb5d775e22b91efcfadfb307b.log

kube-controller-manager-k8s-master01_kube-system_kube-controller-manager-cf5f01b                                                                                                    f6e675530e38b9c5e38b4cb7e3a95419174c42edcc776c3b9a2d180c1.log

kube-flannel-ds-amd64-9rxst_kube-system_install-cni-61a714e802d3ff13cd80ed1daf78                                                                                                    d0d64e5fc17d960257e61720c47e52ee9de0.log

kube-flannel-ds-amd64-9rxst_kube-system_kube-flannel-2c2e49d1b2fa26ff27711889a0b                                                                                                    0103212bab43eb55947a5a51dd03201d94c81.log

kube-flannel-ds-amd64-9rxst_kube-system_kube-flannel-5bfefffa025cbd2e31d79a52807                                                                                                    f1836a6fd3078f5549e520aa56fd740c63650.log

kube-proxy-stpht_kube-system_kube-proxy-89f6dedaa08c21586cfd4b5b8a3a451f187b312e                                                                                                    46c3082c615d641042881d9f.log

kube-proxy-stpht_kube-system_kube-proxy-d32ff8e3d052956cc51c6612b7acf77b96e4f0b1                                                                                                    8f94fa788438dacaabbd39d0.log

kube-scheduler-k8s-master01_kube-system_kube-scheduler-5a3e755bc423adfd5db5567d3                                                                                                    8d1836709c77ef7c1377b4fdf20507abc9d0607.log

kube-scheduler-k8s-master01_kube-system_kube-scheduler-604374c4cccbba218a3810a79                                                                                                    a823131cd964d0ebeb887ee7daf5be8a002191d.log

kube-state-metrics-6b976749f9-zj9pb_monitoring_addon-resizer-b883a7d8165ce83fd1d                                                                                                    3f1779214dad59f044588c6495c717e98c30279f0fe2d.log

kube-state-metrics-6b976749f9-zj9pb_monitoring_addon-resizer-c89ca3f36e9d05db6e7                                                                                                    b37051732686f1910c0b0bae13db620753b4c2990f0d4.log

kube-state-metrics-6b976749f9-zj9pb_monitoring_kube-rbac-proxy-main-39352a890ca0                                                                                                    9d781303f4338f01e8f834f42bd5e9bc2304022994f67ca739a7.log

kube-state-metrics-6b976749f9-zj9pb_monitoring_kube-rbac-proxy-main-e20067b5eab8                                                                                                    2b4e42e457ace18e2038b824c79f8e583838c5d85b6baa962458.log

kube-state-metrics-6b976749f9-zj9pb_monitoring_kube-rbac-proxy-self-3ce3dc1401cb                                                                                                    070182d3aace974c40a073ade3b7e661ff4cb48f8ef425b7fecb.log

kube-state-metrics-6b976749f9-zj9pb_monitoring_kube-rbac-proxy-self-d24c99c9bc0f                                                                                                    7cc90df7da511821e4a2a77022a1e0430075dd0e575006ac97b7.log

kube-state-metrics-6b976749f9-zj9pb_monitoring_kube-state-metrics-11a9df7d63e834                                                                                                    dea659385caff87fb6f5c4fe221bcd55a0ff69324d206ab92c.log

kube-state-metrics-6b976749f9-zj9pb_monitoring_kube-state-metrics-d2c766341d4c93                                                                                                    d6f548337a80903ab73e899e68e97294bde7daacb19379b892.log

node-exporter-5cnw9_monitoring_kube-rbac-proxy-5a33306e8c4ad8336f8b9a18480b8a653                                                                                                    287fcb88ffba9eef3d5548297852ea4.log

node-exporter-5cnw9_monitoring_kube-rbac-proxy-ff8319dc15586c31cc69e07d05b11a1ac                                                                                                    6706e1d3e0b6e79e06fefea901629e1.log

node-exporter-5cnw9_monitoring_node-exporter-420ee11a653cdfa9a982ad71905485ca057                                                                                                    dbbfa2d71141d4b54196ff9c192dc.log

node-exporter-5cnw9_monitoring_node-exporter-8edfbbbcde3eb46272bf5c94202af48540a                                                                                                    ae75d43eaa371760825e437b7c3dc.log

prometheus-adapter-6998484678-6cjqm_monitoring_prometheus-adapter-494760eb9a5adf                                                                                                    2e91077cd16444c249b37bb660c0f09340f2a8e0a9219081b2.log

prometheus-adapter-6998484678-6cjqm_monitoring_prometheus-adapter-d5dfc0f4e39e13                                                                                                    67f108c02e1bb613e0c9c5b6f042581362109e27cc37336c17.log

prometheus-operator-5dc9567958-96xvn_monitoring_prometheus-operator-1eae6b6f1f22                                                                                                    9f459307bafd95e121d3a0505ffa885b4e0b2b931cef20a1d21e.log

prometheus-operator-5dc9567958-96xvn_monitoring_prometheus-operator-e94a8365e0a4                                                                                                    7360cf0bfe1274d872b93ef9e6f74d30a87567acf9ecede5b761.log

就是说这个路径是集群的日志路径，好像是每个pod都要挂载这个路径吧

都挂载后数据就到efk的e上了



然后k就负责展现，应该就是类似于web界面呈现了。

Step2

部署E集群

添加仓库报错。。。

[root@k8s-master01 ~]# helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator

Error: looks like "http://storage.googleapis.com/kubernetes-charts-incubator" is not a valid chart repository or cannot be reached: Get "http://storage.googleapis.com/kubernetes-charts-incubator/index.yaml": dial tcp: lookup storage.googleapis.com on 114.114.114.114:53: read udp 192.168.66.10:57956->114.114.114.114:53: i/o timeout

暂时先不管吧

然后创建efk名称空间

[root@k8s-master01 ~]# kubectl create namespace efk

namespace/efk created

然后获取elasticsearch

[root@k8s-master01 ~]# helm fetch incubator/elasticsearch

Error: repo incubator not found

这里又报错，暂时不管，以上这两个错误都是因为政策限制，需要翻墙才能获取到。

那么正常情况下我们应该得到的是

elasticsearch-1.10.2.tgz

 

 

对于fetch不成功是因为上面仓库添加报错，所以我们尝试下

[root@k8s-master01 ~]# helm repo add incubator https://charts.helm.sh/incubator

"incubator" has been added to your repositories

 

 

实际上我们应该在独立的efk目录，这样比较规范看起来不乱

创建并进入efk目录

[root@k8s-master01 ~]# mkdir /usr/local/install-k8s/efk

[root@k8s-master01 ~]# cd /usr/local/install-k8s/efk

然后我们再来fetch

[root@k8s-master01 efk]# helm fetch incubator/elasticsearch

不错，竟然版本都和老师的是一样的

[root@k8s-master01 efk]# ls

elasticsearch-1.10.2.tgz

解压后主要是配置这几个地方

[root@k8s-master01 efk]# cat elasticsearch/values.yaml

# Default values for elasticsearch.

# This is a YAML-formatted file.

# Declare variables to be passed into your templates.

appVersion: "6.4.2"

 

image:

  repository: "docker.elastic.co/elasticsearch/elasticsearch-oss"

  tag: "6.4.2"

  pullPolicy: "IfNotPresent"

  # If specified, use these secrets to access the image

  # pullSecrets:

  #   - registry-secret

 

initImage:

  repository: "busybox"

  tag: "latest"

  pullPolicy: "Always"

 

cluster:

  name: "elasticsearch"

  # If you want X-Pack installed, switch to an image that includes it, enable this option and toggle the features you want

  # enabled in the environment variables outlined in the README

  xpackEnable: false

  # Some settings must be placed in a keystore, so they need to be mounted in from a secret.

  # Use this setting to specify the name of the secret

  # keystoreSecret: eskeystore

  config: {}

  # Custom parameters, as string, to be added to ES_JAVA_OPTS environment variable

  additionalJavaOpts: ""

  env:

    # IMPORTANT: https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#minimum_master_nodes

    # To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting so that each master-eligible

    # node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.

    MINIMUM_MASTER_NODES: "1"

 

client:

  name: client

  replicas: 1

  serviceType: ClusterIP

  loadBalancerIP: {}

  loadBalancerSourceRanges: {}

## (dict) If specified, apply these annotations to the client service

#  serviceAnnotations:

#    example: client-svc-foo

  heapSize: "512m"

  antiAffinity: "soft"

  nodeAffinity: {}

  nodeSelector: {}

  tolerations: []

  resources:

    limits:

      cpu: "1"

      # memory: "1024Mi"

    requests:

      cpu: "25m"

      memory: "512Mi"

  priorityClassName: ""

  ## (dict) If specified, apply these annotations to each client Pod

  # podAnnotations:

  #   example: client-foo

  podDisruptionBudget:

    enabled: false

    minAvailable: 1

    # maxUnavailable: 1

 

master:

  name: master

  exposeHttp: false

  replicas: 1

  heapSize: "512m"

  persistence:

    enabled: false

    accessMode: ReadWriteOnce

    name: data

    size: "4Gi"

    # storageClass: "ssd"

  antiAffinity: "soft"

  nodeAffinity: {}

  nodeSelector: {}

  tolerations: []

  resources:

    limits:

      cpu: "1"

      # memory: "1024Mi"

    requests:

      cpu: "25m"

      memory: "512Mi"

  priorityClassName: ""

  ## (dict) If specified, apply these annotations to each master Pod

  # podAnnotations:

  #   example: master-foo

  podDisruptionBudget:

    enabled: false

    minAvailable: 2  # Same as `cluster.env.MINIMUM_MASTER_NODES`

    # maxUnavailable: 1

  updateStrategy:

    type: OnDelete

 

data:

  name: data

  exposeHttp: false

  replicas: 1

  heapSize: "1536m"

  persistence:

    enabled: false

    accessMode: ReadWriteOnce

    name: data

    size: "30Gi"

    # storageClass: "ssd"

  terminationGracePeriodSeconds: 3600

  antiAffinity: "soft"

  nodeAffinity: {}

  nodeSelector: {}

  tolerations: []

  resources:

    limits:

      cpu: "1"

      # memory: "2048Mi"

    requests:

      cpu: "25m"

      memory: "1536Mi"

  priorityClassName: ""

  ## (dict) If specified, apply these annotations to each data Pod

  # podAnnotations:

  #   example: data-foo

  podDisruptionBudget:

    enabled: false

    # minAvailable: 1

    maxUnavailable: 1

  updateStrategy:

    type: OnDelete

 

## Additional init containers

extraInitContainers: |

 

MINIMUM_MASTER_NODES仅配置为1以及后面几个replicas: 1和enabled: false

的配置分别是：

对于第一个配置，由于elasticsearch单个好像需要8g内存，而我集群已经占用了好多内存，我宿主机总共16g内存，所以不敢这样浪。

对于第二个配置，想必是同样的原因。

对于第三个配置，就是说没有多余的pvc给它用了。

 

按理来说修改好后就可以使用helm来安装了

[root@k8s-master01 elasticsearch]# helm install els1 --namespace efk -f values.yaml .

WARNING: This chart is deprecated

Error: template: elasticsearch/templates/configmap.yaml:58:7: executing "elasticsearch/templates/configmap.yaml" at <(.Values.cluster.xpackEnable) and .Values.appVersion gt 6.3>: c                                       an't give argument to non-function .Values.cluster.xpackEnable

看来helm3安装不了的

 

 

尝试下之前有备份的helm2咯

[root@k8s-master01 elasticsearch]# helmv2 install --name els1 --namespace=efk -f values.yaml

Error: could not find tiller

[root@k8s-master01 elasticsearch]# helmv2 install --name els1 --namespace=efk -f values.yaml .

Error: could not find tiller

看来helm2也不行。。。

 

再次以helm2尝试新的解决方案

kubectl create serviceaccount --namespace kube-system tiller

kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

 

 

kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

 

下面这个错误无所谓

[root@k8s-master01 elasticsearch]# helmv2 init --service-account tiller --upgrade

$HELM_HOME has been configured at /root/.helm.

Error: error when upgrading: current Tiller version is newer, use --force-upgrade to downgrade

 

 

[root@k8s-master01 elasticsearch]# helmv2 init --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -

deployment.apps/tiller-deploy configured

service/tiller-deploy configured

 

问题虽然解决了，但是新的问题又出来了。。。

 

[root@k8s-master01 elasticsearch]# helmv2 install --name els1 --namespace=efk -f values.yaml .

Error: Get https://10.96.0.1:443/version?timeout=32s: dial tcp 10.96.0.1:443: i/o timeout

 

 

 

再次尝试的新方案（这个方案好像就算能成，也会影响后面的事情的）

[root@k8s-master01 elasticsearch]# kubectl taint nodes --all node-role.kubernetes.io/master-

node/k8s-master01 untainted

taint "node-role.kubernetes.io/master:" not found

taint "node-role.kubernetes.io/master:" not found

还是不行，那么再尝试debug调试helmv2

[root@k8s-master01 elasticsearch]# helmv2 install --name els1 --namespace=efk --debug -f values.yaml .

[debug] Created tunnel using local port: '40951'

 

[debug] SERVER: "127.0.0.1:40951"

 

[debug] Original chart version: ""

[debug] CHART PATH: /usr/local/install-k8s/efk/elasticsearch

 

Error: Get https://10.96.0.1:443/version?timeout=32s: dial tcp 10.96.0.1:443: i/o timeout




 

最后我这里怀疑是那个values.yaml中的镜像需要先拉取下来

docker.elastic.co/elasticsearch/elasticsearch-oss: 6.4.2

然后还是有问题

 

尝试更换网络方案

各节点重置集群时候在非master节点时候有个比较令人关注的问题

[root@k8s-node01 ~]# kubeadm reset

[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.

[reset] Are you sure you want to proceed? [y/N]: y

[preflight] Running pre-flight checks

W0202 21:27:08.399841   37067 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory

[reset] No etcd config found. Assuming external etcd

[reset] Please, manually reset etcd to prevent further issues

[reset] Stopping the kubelet service

[reset] Unmounting mounted directories in "/var/lib/kubelet"

E0202 21:28:49.177580   37067 cleanupnode.go:82] [reset] Failed to remove containers: failed to remove running container 6501f686ca53: output: Error response from daemon: Could not kill running container 6501f686ca53ed9398e5fc3930b8b3d51d6b23006913026b5b74f5b91fce65c8, cannot remove - container 6501f686ca53 PID 2890 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes

, error: exit status 1

[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]

[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

 

The reset process does not reset or clean up iptables rules or IPVS tables.

If you wish to reset iptables, you must do so manually.

For example:

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

 

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)

to reset your system's IPVS tables.

 

The reset process does not clean your kubeconfig files and you must remove them manually.

Please, check the contents of the $HOME/.kube/config file.

 

如果该方案不行就还原原来的flannel方案并尝试etcd

 

 

实际上集群初始化的另外一种方式中

--pod-network-cidr=172.16.0.0/16

pod-network-cidr相当于

用配置文件初始化时候的podSubnet: "10.224.0.0/16"

 

 

 

# 在集群各节点清理flannel网络的残留文件

ifconfig cni0 down

ip link delete cni0

ifconfig flannel.1 down

ip link delete flannel.1

rm -rf /var/lib/cni

rm -rf /etc/cni/net.d

下载caico（应该是这个yaml吧https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico.yaml）

 

[root@k8s-master01 ~]# cat calico.yaml

---

# Source: calico/templates/calico-kube-controllers.yaml

# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict

 

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

  name: calico-kube-controllers

  namespace: kube-system

  labels:

    k8s-app: calico-kube-controllers

spec:

  maxUnavailable: 1

  selector:

    matchLabels:

      k8s-app: calico-kube-controllers

---

# Source: calico/templates/calico-kube-controllers.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: calico-kube-controllers

  namespace: kube-system

---

# Source: calico/templates/calico-node.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: calico-node

  namespace: kube-system

---

# Source: calico/templates/calico-etcd-secrets.yaml

# The following contains k8s Secrets for use with a TLS enabled etcd cluster.

# For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/

apiVersion: v1

kind: Secret

type: Opaque

metadata:

  name: calico-etcd-secrets

  namespace: kube-system

data:

  # Populate the following with etcd TLS configuration if desired, but leave blank if

  # not using TLS for etcd.

  # The keys below should be uncommented and the values populated with the base64

  # encoded contents of each file that would be associated with the TLS data.

  # Example command for encoding a file contents: cat <file> | base64 -w 0

  # etcd-key: null

  # etcd-cert: null

  # etcd-ca: null

---

# Source: calico/templates/calico-config.yaml

# This ConfigMap is used to configure a self-hosted Calico installation.

kind: ConfigMap

apiVersion: v1

metadata:

  name: calico-config

  namespace: kube-system

data:

  # Configure this with the location of your etcd cluster.

  etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"

  # If you're using TLS enabled etcd uncomment the following.

  # You must also populate the Secret below with these files.

  etcd_ca: ""   # "/calico-secrets/etcd-ca"

  etcd_cert: "" # "/calico-secrets/etcd-cert"

  etcd_key: ""  # "/calico-secrets/etcd-key"

  # Typha is disabled.

  typha_service_name: "none"

  # Configure the backend to use.

  calico_backend: "bird"

 

  # Configure the MTU to use for workload interfaces and tunnels.

  # By default, MTU is auto-detected, and explicitly setting this field should not be required.

  # You can override auto-detection by providing a non-zero value.

  veth_mtu: "0"

 

  # The CNI network configuration to install on each node. The special

  # values in this config will be automatically populated.

  cni_network_config: |-

    {

      "name": "k8s-pod-network",

      "cniVersion": "0.3.1",

      "plugins": [

        {

          "type": "calico",

          "log_level": "info",

          "log_file_path": "/var/log/calico/cni/cni.log",

          "etcd_endpoints": "__ETCD_ENDPOINTS__",

          "etcd_key_file": "__ETCD_KEY_FILE__",

          "etcd_cert_file": "__ETCD_CERT_FILE__",

          "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",

          "mtu": __CNI_MTU__,

          "ipam": {

              "type": "calico-ipam"

          },

          "policy": {

              "type": "k8s"

          },

          "kubernetes": {

              "kubeconfig": "__KUBECONFIG_FILEPATH__"

          }

        },

        {

          "type": "portmap",

          "snat": true,

          "capabilities": {"portMappings": true}

        },

        {

          "type": "bandwidth",

          "capabilities": {"bandwidth": true}

        }

      ]

    }

---

# Source: calico/templates/calico-kube-controllers-rbac.yaml

# Include a clusterrole for the kube-controllers component,

# and bind it to the calico-kube-controllers serviceaccount.

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: calico-kube-controllers

rules:

  # Pods are monitored for changing labels.

  # The node controller monitors Kubernetes nodes.

  # Namespace and serviceaccount labels are used for policy.

  - apiGroups: [""]

    resources:

      - pods

      - nodes

      - namespaces

      - serviceaccounts

    verbs:

      - watch

      - list

      - get

  # Watch for changes to Kubernetes NetworkPolicies.

  - apiGroups: ["networking.k8s.io"]

    resources:

      - networkpolicies

    verbs:

      - watch

      - list

---

# Source: calico/templates/calico-node-rbac.yaml

# Include a clusterrole for the calico-node DaemonSet,

# and bind it to the calico-node serviceaccount.

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: calico-node

rules:

  # Used for creating service account tokens to be used by the CNI plugin

  - apiGroups: [""]

    resources:

      - serviceaccounts/token

    resourceNames:

      - calico-node

    verbs:

      - create

  # The CNI plugin needs to get pods, nodes, and namespaces.

  - apiGroups: [""]

    resources:

      - pods

      - nodes

      - namespaces

    verbs:

      - get

  # EndpointSlices are used for Service-based network policy rule

  # enforcement.

  - apiGroups: ["discovery.k8s.io"]

    resources:

      - endpointslices

    verbs:

      - watch

      - list

  - apiGroups: [""]

    resources:

      - endpoints

      - services

    verbs:

      # Used to discover service IPs for advertisement.

      - watch

      - list

  # Pod CIDR auto-detection on kubeadm needs access to config maps.

  - apiGroups: [""]

    resources:

      - configmaps

    verbs:

      - get

  - apiGroups: [""]

    resources:

      - nodes/status

    verbs:

      # Needed for clearing NodeNetworkUnavailable flag.

      - patch

---

# Source: calico/templates/calico-kube-controllers-rbac.yaml

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: calico-kube-controllers

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: calico-kube-controllers

subjects:

- kind: ServiceAccount

  name: calico-kube-controllers

  namespace: kube-system

---

# Source: calico/templates/calico-node-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: calico-node

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: calico-node

subjects:

- kind: ServiceAccount

  name: calico-node

  namespace: kube-system

---

# Source: calico/templates/calico-node.yaml

# This manifest installs the calico-node container, as well

# as the CNI plugins and network config on

# each master and worker node in a Kubernetes cluster.

kind: DaemonSet

apiVersion: apps/v1

metadata:

  name: calico-node

  namespace: kube-system

  labels:

    k8s-app: calico-node

spec:

  selector:

    matchLabels:

      k8s-app: calico-node

  updateStrategy:

    type: RollingUpdate

    rollingUpdate:

      maxUnavailable: 1

  template:

    metadata:

      labels:

        k8s-app: calico-node

    spec:

      nodeSelector:

        kubernetes.io/os: linux

      hostNetwork: true

      tolerations:

        # Make sure calico-node gets scheduled on all nodes.

        - effect: NoSchedule

          operator: Exists

        # Mark the pod as a critical add-on for rescheduling.

        - key: CriticalAddonsOnly

          operator: Exists

        - effect: NoExecute

          operator: Exists

      serviceAccountName: calico-node

      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force

      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.

      terminationGracePeriodSeconds: 0

      priorityClassName: system-node-critical

      initContainers:

        # This container installs the CNI binaries

        # and CNI network config file on each node.

        - name: install-cni

          image: docker.io/calico/cni:v3.25.0

          imagePullPolicy: IfNotPresent

          command: ["/opt/cni/bin/install"]

          envFrom:

          - configMapRef:

              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.

              name: kubernetes-services-endpoint

              optional: true

          env:

            # Name of the CNI config file to create.

            - name: CNI_CONF_NAME

              value: "10-calico.conflist"

            # The CNI network config to install on each node.

            - name: CNI_NETWORK_CONFIG

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: cni_network_config

            # The location of the etcd cluster.

            - name: ETCD_ENDPOINTS

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_endpoints

            # CNI MTU Config variable

            - name: CNI_MTU

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: veth_mtu

            # Prevents the container from sleeping forever.

            - name: SLEEP

              value: "false"

          volumeMounts:

            - mountPath: /host/opt/cni/bin

              name: cni-bin-dir

            - mountPath: /host/etc/cni/net.d

              name: cni-net-dir

            - mountPath: /calico-secrets

              name: etcd-certs

          securityContext:

            privileged: true

        # This init container mounts the necessary filesystems needed by the BPF data plane

        # i.e. bpf at /sys/fs/bpf and cgroup2 at /run/calico/cgroup. Calico-node initialisation is executed

        # in best effort fashion, i.e. no failure for errors, to not disrupt pod creation in iptable mode.

        - name: "mount-bpffs"

          image: docker.io/calico/node:v3.25.0

          imagePullPolicy: IfNotPresent

          command: ["calico-node", "-init", "-best-effort"]

          volumeMounts:

            - mountPath: /sys/fs

              name: sys-fs

              # Bidirectional is required to ensure that the new mount we make at /sys/fs/bpf propagates to the host

              # so that it outlives the init container.

              mountPropagation: Bidirectional

            - mountPath: /var/run/calico

              name: var-run-calico

              # Bidirectional is required to ensure that the new mount we make at /run/calico/cgroup propagates to the host

              # so that it outlives the init container.

              mountPropagation: Bidirectional

            # Mount /proc/ from host which usually is an init program at /nodeproc. It's needed by mountns binary,

            # executed by calico-node, to mount root cgroup2 fs at /run/calico/cgroup to attach CTLB programs correctly.

            - mountPath: /nodeproc

              name: nodeproc

              readOnly: true

          securityContext:

            privileged: true

      containers:

        # Runs calico-node container on each Kubernetes node. This

        # container programs network policy and routes on each

        # host.

        - name: calico-node

          image: docker.io/calico/node:v3.25.0

          imagePullPolicy: IfNotPresent

          envFrom:

          - configMapRef:

              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.

              name: kubernetes-services-endpoint

              optional: true

          env:

            # The location of the etcd cluster.

            - name: ETCD_ENDPOINTS

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_endpoints

            # Location of the CA certificate for etcd.

            - name: ETCD_CA_CERT_FILE

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_ca

            # Location of the client key for etcd.

            - name: ETCD_KEY_FILE

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_key

            # Location of the client certificate for etcd.

            - name: ETCD_CERT_FILE

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_cert

            # Set noderef for node controller.

            - name: CALICO_K8S_NODE_REF

              valueFrom:

                fieldRef:

                  fieldPath: spec.nodeName

            # Choose the backend to use.

            - name: CALICO_NETWORKING_BACKEND

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: calico_backend

            # Cluster type to identify the deployment type

            - name: CLUSTER_TYPE

              value: "k8s,bgp"

            # Auto-detect the BGP IP address.

            - name: IP

              value: "autodetect"

            # Enable IPIP

            - name: CALICO_IPV4POOL_IPIP

              value: "Always"

            # Enable or Disable VXLAN on the default IP pool.

            - name: CALICO_IPV4POOL_VXLAN

              value: "Never"

            # Enable or Disable VXLAN on the default IPv6 IP pool.

            - name: CALICO_IPV6POOL_VXLAN

              value: "Never"

            # Set MTU for tunnel device used if ipip is enabled

            - name: FELIX_IPINIPMTU

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: veth_mtu

            # Set MTU for the VXLAN tunnel device.

            - name: FELIX_VXLANMTU

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: veth_mtu

            # Set MTU for the Wireguard tunnel device.

            - name: FELIX_WIREGUARDMTU

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: veth_mtu

            # The default IPv4 pool to create on startup if none exists. Pod IPs will be

            # chosen from this range. Changing this value after installation will have

            # no effect. This should fall within `--cluster-cidr`.

            # - name: CALICO_IPV4POOL_CIDR

            #   value: "192.168.0.0/16"

            # Disable file logging so `kubectl logs` works.

            - name: CALICO_DISABLE_FILE_LOGGING

              value: "true"

            # Set Felix endpoint to host default action to ACCEPT.

            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION

              value: "ACCEPT"

            # Disable IPv6 on Kubernetes.

            - name: FELIX_IPV6SUPPORT

              value: "false"

            - name: FELIX_HEALTHENABLED

              value: "true"

          securityContext:

            privileged: true

          resources:

            requests:

              cpu: 250m

          lifecycle:

            preStop:

              exec:

                command:

                - /bin/calico-node

                - -shutdown

          livenessProbe:

            exec:

              command:

              - /bin/calico-node

              - -felix-live

              - -bird-live

            periodSeconds: 10

            initialDelaySeconds: 10

            failureThreshold: 6

            timeoutSeconds: 10

          readinessProbe:

            exec:

              command:

              - /bin/calico-node

              - -felix-ready

              - -bird-ready

            periodSeconds: 10

            timeoutSeconds: 10

          volumeMounts:

            # For maintaining CNI plugin API credentials.

            - mountPath: /host/etc/cni/net.d

              name: cni-net-dir

              readOnly: false

            - mountPath: /lib/modules

              name: lib-modules

              readOnly: true

            - mountPath: /run/xtables.lock

              name: xtables-lock

              readOnly: false

            - mountPath: /var/run/calico

              name: var-run-calico

              readOnly: false

            - mountPath: /var/lib/calico

              name: var-lib-calico

              readOnly: false

            - mountPath: /calico-secrets

              name: etcd-certs

            - name: policysync

              mountPath: /var/run/nodeagent

            # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the

            # parent directory.

            - name: bpffs

              mountPath: /sys/fs/bpf

            - name: cni-log-dir

              mountPath: /var/log/calico/cni

              readOnly: true

      volumes:

        # Used by calico-node.

        - name: lib-modules

          hostPath:

            path: /lib/modules

        - name: var-run-calico

          hostPath:

            path: /var/run/calico

        - name: var-lib-calico

          hostPath:

            path: /var/lib/calico

        - name: xtables-lock

          hostPath:

            path: /run/xtables.lock

            type: FileOrCreate

        - name: sys-fs

          hostPath:

            path: /sys/fs/

            type: DirectoryOrCreate

        - name: bpffs

          hostPath:

            path: /sys/fs/bpf

            type: Directory

        # mount /proc at /nodeproc to be used by mount-bpffs initContainer to mount root cgroup2 fs.

        - name: nodeproc

          hostPath:

            path: /proc

        # Used to install CNI.

        - name: cni-bin-dir

          hostPath:

            path: /opt/cni/bin

        - name: cni-net-dir

          hostPath:

            path: /etc/cni/net.d

        # Used to access CNI logs.

        - name: cni-log-dir

          hostPath:

            path: /var/log/calico/cni

        # Mount in the etcd TLS secrets with mode 400.

        # See https://kubernetes.io/docs/concepts/configuration/secret/

        - name: etcd-certs

          secret:

            secretName: calico-etcd-secrets

            defaultMode: 0400

        # Used to create per-pod Unix Domain Sockets

        - name: policysync

          hostPath:

            type: DirectoryOrCreate

            path: /var/run/nodeagent

---

# Source: calico/templates/calico-kube-controllers.yaml

# See https://github.com/projectcalico/kube-controllers

apiVersion: apps/v1

kind: Deployment

metadata:

  name: calico-kube-controllers

  namespace: kube-system

  labels:

    k8s-app: calico-kube-controllers

spec:

  # The controllers can only have a single active instance.

  replicas: 1

  selector:

    matchLabels:

      k8s-app: calico-kube-controllers

  strategy:

    type: Recreate

  template:

    metadata:

      name: calico-kube-controllers

      namespace: kube-system

      labels:

        k8s-app: calico-kube-controllers

    spec:

      nodeSelector:

        kubernetes.io/os: linux

      tolerations:

        # Mark the pod as a critical add-on for rescheduling.

        - key: CriticalAddonsOnly

          operator: Exists

        - key: node-role.kubernetes.io/master

          effect: NoSchedule

        - key: node-role.kubernetes.io/control-plane

          effect: NoSchedule

      serviceAccountName: calico-kube-controllers

      priorityClassName: system-cluster-critical

      # The controllers must run in the host network namespace so that

      # it isn't governed by policy that would prevent it from working.

      hostNetwork: true

      containers:

        - name: calico-kube-controllers

          image: docker.io/calico/kube-controllers:v3.25.0

          imagePullPolicy: IfNotPresent

          env:

            # The location of the etcd cluster.

            - name: ETCD_ENDPOINTS

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_endpoints

            # Location of the CA certificate for etcd.

            - name: ETCD_CA_CERT_FILE

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_ca

            # Location of the client key for etcd.

            - name: ETCD_KEY_FILE

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_key

            # Location of the client certificate for etcd.

            - name: ETCD_CERT_FILE

              valueFrom:

                configMapKeyRef:

                  name: calico-config

                  key: etcd_cert

            # Choose which controllers to run.

            - name: ENABLED_CONTROLLERS

              value: policy,namespace,serviceaccount,workloadendpoint,node

          volumeMounts:

            # Mount in the etcd TLS secrets.

            - mountPath: /calico-secrets

              name: etcd-certs

          livenessProbe:

            exec:

              command:

              - /usr/bin/check-status

              - -l

            periodSeconds: 10

            initialDelaySeconds: 10

            failureThreshold: 6

            timeoutSeconds: 10

          readinessProbe:

            exec:

              command:

              - /usr/bin/check-status

              - -r

            periodSeconds: 10

      volumes:

        # Mount in the etcd TLS secrets with mode 400.

        # See https://kubernetes.io/docs/concepts/configuration/secret/

        - name: etcd-certs

          secret:

            secretName: calico-etcd-secrets

            defaultMode: 0440

依赖的镜像

docker pull docker.io/calico/cni:v3.20.0 && docker pull docker.io/calico/pod2daemon-flexvol:v3.20.0 && docker pull docker.io/calico/node:v3.20.0 && docker pull docker.io/calico/kube-controllers:v3.20.0

 

 

新的问题

[root@k8s-master01 ~]# kubectl apply -f calico.yaml

serviceaccount/calico-kube-controllers created

serviceaccount/calico-node created

secret/calico-etcd-secrets created

configmap/calico-config created

clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created

clusterrole.rbac.authorization.k8s.io/calico-node created

clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created

clusterrolebinding.rbac.authorization.k8s.io/calico-node created

daemonset.apps/calico-node created

deployment.apps/calico-kube-controllers created

error: unable to recognize "calico.yaml": no matches for kind "PodDisruptionBudget" in version "policy/v1"

下载正确版本的calico

[root@k8s-master01 ~]# rm -rf calico.yaml

wget https://docs.projectcalico.org/v3.8/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

安装

[root@k8s-master01 ~]# kubectl apply -f calico.yaml

configmap/calico-config configured

customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created

customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created

clusterrole.rbac.authorization.k8s.io/calico-kube-controllers configured

clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged

clusterrole.rbac.authorization.k8s.io/calico-node configured

clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged

daemonset.apps/calico-node configured

serviceaccount/calico-node unchanged

deployment.apps/calico-kube-controllers configured

serviceaccount/calico-kube-controllers unchanged

 

 

然后各节点加入集群后重新efk的e的搭建

[root@k8s-master01 elasticsearch]# kubectl create namespace efk

namespace/efk created

[root@k8s-master01 elasticsearch]#

[root@k8s-master01 elasticsearch]# helmv2 install --name els1 --namespace=efk -f values.yaml .

Error: could not find tiller

[root@k8s-master01 elasticsearch]# kubectl create serviceaccount --namespace kube-system tiller

serviceaccount/tiller created

[root@k8s-master01 elasticsearch]# kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created

[root@k8s-master01 elasticsearch]# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

Error from server (NotFound): deployments.extensions "tiller-deploy" not found

又一个新问题。。。

尝试

[root@k8s-master01 elasticsearch]# helmv2 init --service-account tiller --skip-refresh

$HELM_HOME has been configured at /root/.helm.

 

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

 

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.

To prevent this, run `helm init` with the --tiller-tls-verify flag.

For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

Happy Helming!

然后再次执行就ok

[root@k8s-master01 elasticsearch]# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

deployment.extensions/tiller-deploy patched (no change)

继续

[root@k8s-master01 elasticsearch]# helmv2 init --service-account tiller --upgrade

$HELM_HOME has been configured at /root/.helm.

 

Tiller (the Helm server-side component) has been upgraded to the current version.

Happy Helming!

[root@k8s-master01 elasticsearch]# helmv2 init --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply

deployment.apps/tiller-deploy configured

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply

service/tiller-deploy configured

 

 

终于成功。。。

[root@k8s-master01 elasticsearch]# helmv2 install --name els1 --namespace=efk -f values.yaml .

NAME:   els1

LAST DEPLOYED: Fri Feb  3 14:02:08 2023

NAMESPACE: efk

STATUS: DEPLOYED

 

RESOURCES:

==> v1/ConfigMap

NAME                DATA  AGE

els1-elasticsearch  4     1s

 

==> v1/Pod(related)

NAME                                        READY  STATUS    RESTARTS  AGE

els1-elasticsearch-client-59bcdcbfb7-7xvs7  0/1    Init:0/1  0         1s

 

==> v1/Service

NAME                          TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)   AGE

els1-elasticsearch-client     ClusterIP  10.100.179.134  <none>       9200/TCP  1s

els1-elasticsearch-discovery  ClusterIP  None            <none>       9300/TCP  1s

 

==> v1beta1/Deployment

NAME                       READY  UP-TO-DATE  AVAILABLE  AGE

els1-elasticsearch-client  0/1    1           0          1s

 

==> v1beta1/StatefulSet

NAME                       READY  AGE

els1-elasticsearch-data    0/1    1s

els1-elasticsearch-master  0/1    0s

 

 

NOTES:

The elasticsearch cluster has been installed.

 

***

Please note that this chart has been deprecated and moved to stable.

Going forward please use the stable version of this chart.

***

 

Elasticsearch can be accessed:

 

  * Within your cluster, at the following DNS name at port 9200:

 

    els1-elasticsearch-client.efk.svc

 

  * From outside the cluster, run these commands in the same shell:

 

    export POD_NAME=$(kubectl get pods --namespace efk -l "app=elasticsearch,component=client,release=els1" -o jsonpath="{.items[0].metadata.name}")

    echo "Visit http://127.0.0.1:9200 to use Elasticsearch"

kubectl port-forward --namespace efk $POD_NAME 9200:9200

 

等到这几个pod全部起来

[root@k8s-master01 elasticsearch]# kubectl get pod -n efk -w

NAME                                         READY   STATUS    RESTARTS   AGE

els1-elasticsearch-client-59bcdcbfb7-7xvs7   1/1     Running   2          7m33s

els1-elasticsearch-data-0                    1/1     Running   0          7m31s

els1-elasticsearch-master-0                  1/1     Running   0          7m31s

查看elasticsearch-client的svc的ip地址

[root@k8s-master01 elasticsearch]# kubectl get svc -n efk

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

els1-elasticsearch-client      ClusterIP   10.100.179.134   <none>        9200/TCP   11m

els1-elasticsearch-discovery   ClusterIP   None             <none>        9300/TCP   11m

然后新开会话测试下

[root@k8s-master01 ~]# kubectl run cirror-$RANDOM --rm -it --image=cirros -- /bin/sh

kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.

If you don't see a command prompt, try pressing enter.

/ # curl 10.100.179.134:9200/_cat/nodes

192.168.85.193 20 96 0 0.86 0.98 0.90 mi * els1-elasticsearch-master-0

192.168.58.195  6 97 0 0.58 0.94 1.12 di - els1-elasticsearch-data-0

192.168.58.194 19 97 0 0.58 0.94 1.12 i  - els1-elasticsearch-client-59bcdcbfb7-7xvs7

/ #

Step3

接下来部署f集群

[root@k8s-master01 elasticsearch]# cd ..

[root@k8s-master01 efk]# helmv2 fetch stable/fluentd-elasticsearch

解压并进入配置values

[root@k8s-master01 efk]# cd fluentd-elasticsearch

[root@k8s-master01 fluentd-elasticsearch]# cat values.yaml

image:

  repository: gcr.io/google-containers/fluentd-elasticsearch

## Specify an imagePullPolicy (Required)

## It's recommended to change this to 'Always' if the image tag is 'latest'

## ref: http://kubernetes.io/docs/user-guide/images/#updating-images

  tag: v2.3.2

  pullPolicy: IfNotPresent

 

## Configure resource requests and limits

## ref: http://kubernetes.io/docs/user-guide/compute-resources/

##

resources: {}

  # limits:

  #   cpu: 100m

  #   memory: 500Mi

  # requests:

  #   cpu: 100m

  #   memory: 200Mi

 

elasticsearch:

  host: '10.100.179.134'

  port: 9200

  scheme: 'http'

  ssl_version: TLSv1_2

  buffer_chunk_limit: 2M

  buffer_queue_limit: 8

  logstash_prefix: 'logstash'

 

# If you want to add custom environment variables, use the env dict

# You can then reference these in your config file e.g.:

#     user "#{ENV['OUTPUT_USER']}"

env:

  # OUTPUT_USER: my_user

 

# If you want to add custom environment variables from secrets, use the secret list

secret:

# - name: ELASTICSEARCH_PASSWORD

#   secret_name: elasticsearch

#   secret_key: password

 

rbac:

  create: true

 

serviceAccount:

  # Specifies whether a ServiceAccount should be created

  create: true

  # The name of the ServiceAccount to use.

  # If not set and create is true, a name is generated using the fullname template

  name:

 

## Specify if a Pod Security Policy for node-exporter must be created

## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/

##

podSecurityPolicy:

  enabled: false

  annotations: {}

    ## Specify pod annotations

    ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#apparmor

    ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp

    ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#sysctl

    ##

    # seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'

    # seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'

    # apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'

 

livenessProbe:

  enabled: true

 

annotations: {}

 

podAnnotations: {}

  # prometheus.io/scrape: "true"

  # prometheus.io/port: "24231"

 

## DaemonSet update strategy

## Ref: https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/

updateStrategy:

  type: RollingUpdate

 

tolerations: {}

  # - key: node-role.kubernetes.io/master

  #   operator: Exists

  #   effect: NoSchedule

 

nodeSelector: {}

 

service: {}

  # type: ClusterIP

  # ports:

  #   - name: "monitor-agent"

  #     port: 24231

 

configMaps:

  system.conf: |-

    <system>

      root_dir /tmp/fluentd-buffers/

    </system>

  containers.input.conf: |-

    # This configuration file for Fluentd / td-agent is used

    # to watch changes to Docker log files. The kubelet creates symlinks that

    # capture the pod name, namespace, container name & Docker container ID

    # to the docker logs for pods in the /var/log/containers directory on the host.

    # If running this fluentd configuration in a Docker container, the /var/log

    # directory should be mounted in the container.

    #

    # These logs are then submitted to Elasticsearch which assumes the

    # installation of the fluent-plugin-elasticsearch & the

    # fluent-plugin-kubernetes_metadata_filter plugins.

    # See https://github.com/uken/fluent-plugin-elasticsearch &

    # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for

    # more information about the plugins.

    #

    # Example

    # =======

    # A line in the Docker log file might look like this JSON:

    #

    # {"log":"2014/09/25 21:15:03 Got request with path wombat\n",

    #  "stream":"stderr",

    #   "time":"2014-09-25T21:15:03.499185026Z"}

    #

    # The time_format specification below makes sure we properly

    # parse the time format produced by Docker. This will be

    # submitted to Elasticsearch and should appear like:

    # $ curl 'http://elasticsearch-logging:9200/_search?pretty'

    # ...

    # {

    #      "_index" : "logstash-2014.09.25",

    #      "_type" : "fluentd",

    #      "_id" : "VBrbor2QTuGpsQyTCdfzqA",

    #      "_score" : 1.0,

    #      "_source":{"log":"2014/09/25 22:45:50 Got request with path wombat\n",

    #                 "stream":"stderr","tag":"docker.container.all",

    #                 "@timestamp":"2014-09-25T22:45:50+00:00"}

    #    },

    # ...

    #

    # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log

    # record & add labels to the log record if properly configured. This enables users

    # to filter & search logs on any metadata.

    # For example a Docker container's logs might be in the directory:

    #

    #  /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b

    #

    # and in the file:

    #

    #  997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log

    #

    # where 997599971ee6... is the Docker ID of the running container.

    # The Kubernetes kubelet makes a symbolic link to this file on the host machine

    # in the /var/log/containers directory which includes the pod name and the Kubernetes

    # container name:

    #

    #    synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log

    #    ->

    #    /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log

    #

    # The /var/log directory on the host is mapped to the /var/log directory in the container

    # running this instance of Fluentd and we end up collecting the file:

    #

    #   /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log

    #

    # This results in the tag:

    #

    #  var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log

    #

    # The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name

    # which are added to the log message as a kubernetes field object & the Docker container ID

    # is also added under the docker field object.

    # The final tag is:

    #

    #   kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log

    #

    # And the final log record look like:

    #

    # {

    #   "log":"2014/09/25 21:15:03 Got request with path wombat\n",

    #   "stream":"stderr",

    #   "time":"2014-09-25T21:15:03.499185026Z",

    #   "kubernetes": {

    #     "namespace": "default",

    #     "pod_name": "synthetic-logger-0.25lps-pod",

    #     "container_name": "synth-lgr"

    #   },

    #   "docker": {

    #     "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b"

    #   }

    # }

    #

    # This makes it easier for users to search for logs by pod name or by

    # the name of the Kubernetes container regardless of how many times the

    # Kubernetes pod has been restarted (resulting in a several Docker container IDs).

    # Json Log Example:

    # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"}

    # CRI Log Example:

    # 2016-02-17T00:04:05.931087621Z stdout F [info:2016-02-16T16:04:05.930-08:00] Some log text here

    <source>

      @id fluentd-containers.log

      @type tail

      path /var/log/containers/*.log

      pos_file /var/log/fluentd-containers.log.pos

      time_format %Y-%m-%dT%H:%M:%S.%NZ

      tag raw.kubernetes.*

      format json

      read_from_head true

    </source>

    # Detect exceptions in the log output and forward them as one log entry.

    <match raw.kubernetes.**>

      @id raw.kubernetes

      @type detect_exceptions

      remove_tag_prefix raw

      message log

      stream stream

      multiline_flush_interval 5

      max_bytes 500000

      max_lines 1000

    </match>

  system.input.conf: |-

    # Example:

    # 2015-12-21 23:17:22,066 [salt.state       ][INFO    ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081

    <source>

      @id minion

      @type tail

      format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/

      time_format %Y-%m-%d %H:%M:%S

      path /var/log/salt/minion

      pos_file /var/log/salt.pos

      tag salt

    </source>

    # Example:

    # Dec 21 23:17:22 gke-foo-1-1-4b5cbd14-node-4eoj startupscript: Finished running startup script /var/run/google.startup.script

    <source>

      @id startupscript.log

      @type tail

      format syslog

      path /var/log/startupscript.log

      pos_file /var/log/startupscript.log.pos

      tag startupscript

    </source>

    # Examples:

    # time="2016-02-04T06:51:03.053580605Z" level=info msg="GET /containers/json"

    # time="2016-02-04T07:53:57.505612354Z" level=error msg="HTTP Error" err="No such image: -f" statusCode=404

    <source>

      @id docker.log

      @type tail

      format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/

      path /var/log/docker.log

      pos_file /var/log/docker.log.pos

      tag docker

    </source>

    # Example:

    # 2016/02/04 06:52:38 filePurge: successfully removed file /var/etcd/data/member/wal/00000000000006d0-00000000010a23d1.wal

    <source>

      @id etcd.log

      @type tail

      # Not parsing this, because it doesn't have anything particularly useful to

      # parse out of it (like severities).

      format none

      path /var/log/etcd.log

      pos_file /var/log/etcd.log.pos

      tag etcd

    </source>

    # Multi-line parsing is required for all the kube logs because very large log

    # statements, such as those that include entire object bodies, get split into

    # multiple lines by glog.

    # Example:

    # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537]

    <source>

      @id kubelet.log

      @type tail

      format multiline

      multiline_flush_interval 5s

      format_firstline /^\w\d{4}/

      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

      time_format %m%d %H:%M:%S.%N

      path /var/log/kubelet.log

      pos_file /var/log/kubelet.log.pos

      tag kubelet

    </source>

    # Example:

    # I1118 21:26:53.975789       6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed

    <source>

      @id kube-proxy.log

      @type tail

      format multiline

      multiline_flush_interval 5s

      format_firstline /^\w\d{4}/

      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

      time_format %m%d %H:%M:%S.%N

      path /var/log/kube-proxy.log

      pos_file /var/log/kube-proxy.log.pos

      tag kube-proxy

    </source>

    # Example:

    # I0204 07:00:19.604280       5 handlers.go:131] GET /api/v1/nodes: (1.624207ms) 200 [[kube-controller-manager/v1.1.3 (linux/amd64) kubernetes/6a81b50] 127.0.0.1:38266]

    <source>

      @id kube-apiserver.log

      @type tail

      format multiline

      multiline_flush_interval 5s

      format_firstline /^\w\d{4}/

      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

      time_format %m%d %H:%M:%S.%N

      path /var/log/kube-apiserver.log

      pos_file /var/log/kube-apiserver.log.pos

      tag kube-apiserver

    </source>

    # Example:

    # I0204 06:55:31.872680       5 servicecontroller.go:277] LB already exists and doesn't need update for service kube-system/kube-ui

    <source>

      @id kube-controller-manager.log

      @type tail

      format multiline

      multiline_flush_interval 5s

      format_firstline /^\w\d{4}/

      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

      time_format %m%d %H:%M:%S.%N

      path /var/log/kube-controller-manager.log

      pos_file /var/log/kube-controller-manager.log.pos

      tag kube-controller-manager

    </source>

    # Example:

    # W0204 06:49:18.239674       7 reflector.go:245] pkg/scheduler/factory/factory.go:193: watch of *api.Service ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [2578313/2577886]) [2579312]

    <source>

      @id kube-scheduler.log

      @type tail

      format multiline

      multiline_flush_interval 5s

      format_firstline /^\w\d{4}/

      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

      time_format %m%d %H:%M:%S.%N

      path /var/log/kube-scheduler.log

      pos_file /var/log/kube-scheduler.log.pos

      tag kube-scheduler

    </source>

    # Example:

    # I1104 10:36:20.242766       5 rescheduler.go:73] Running Rescheduler

    <source>

      @id rescheduler.log

      @type tail

      format multiline

      multiline_flush_interval 5s

      format_firstline /^\w\d{4}/

      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

      time_format %m%d %H:%M:%S.%N

      path /var/log/rescheduler.log

      pos_file /var/log/rescheduler.log.pos

      tag rescheduler

    </source>

    # Example:

    # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf

    <source>

      @id glbc.log

      @type tail

      format multiline

      multiline_flush_interval 5s

      format_firstline /^\w\d{4}/

      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

      time_format %m%d %H:%M:%S.%N

      path /var/log/glbc.log

      pos_file /var/log/glbc.log.pos

      tag glbc

    </source>

    # Example:

    # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf

    <source>

      @id cluster-autoscaler.log

      @type tail

      format multiline

      multiline_flush_interval 5s

      format_firstline /^\w\d{4}/

      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/

      time_format %m%d %H:%M:%S.%N

      path /var/log/cluster-autoscaler.log

      pos_file /var/log/cluster-autoscaler.log.pos

      tag cluster-autoscaler

    </source>

    # Logs from systemd-journal for interesting services.

    <source>

      @id journald-docker

      @type systemd

      matches [{ "_SYSTEMD_UNIT": "docker.service" }]

      <storage>

        @type local

        persistent true

        path /var/log/journald-docker.pos

      </storage>

      read_from_head true

      tag docker

    </source>

    <source>

      @id journald-kubelet

      @type systemd

      matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]

      <storage>

        @type local

        persistent true

        path /var/log/journald-kubelet.pos

      </storage>

      read_from_head true

      tag kubelet

    </source>

    <source>

      @id journald-node-problem-detector

      @type systemd

      matches [{ "_SYSTEMD_UNIT": "node-problem-detector.service" }]

      <storage>

        @type local

        persistent true

        path /var/log/journald-node-problem-detector.pos

      </storage>

      read_from_head true

      tag node-problem-detector

    </source>

  forward.input.conf: |-

    # Takes the messages sent over TCP

    <source>

      @type forward

    </source>

  monitoring.conf: |-

    # Prometheus Exporter Plugin

    # input plugin that exports metrics

    <source>

      @type prometheus

    </source>

    <source>

      @type monitor_agent

    </source>

    # input plugin that collects metrics from MonitorAgent

    <source>

      @type prometheus_monitor

      <labels>

        host ${hostname}

      </labels>

    </source>

    # input plugin that collects metrics for output plugin

    <source>

      @type prometheus_output_monitor

      <labels>

        host ${hostname}

      </labels>

    </source>

    # input plugin that collects metrics for in_tail plugin

    <source>

      @type prometheus_tail_monitor

      <labels>

        host ${hostname}

      </labels>

    </source>

  output.conf: |

    # Enriches records with Kubernetes metadata

    <filter kubernetes.**>

      @type kubernetes_metadata

    </filter>

 

    <match **>

      @id elasticsearch

      @type elasticsearch

      @log_level info

      include_tag_key true

      type_name _doc

      host "#{ENV['OUTPUT_HOST']}"

      port "#{ENV['OUTPUT_PORT']}"

      scheme "#{ENV['OUTPUT_SCHEME']}"

      ssl_version "#{ENV['OUTPUT_SSL_VERSION']}"

      logstash_format true

      logstash_prefix "#{ENV['LOGSTASH_PREFIX']}"

      reconnect_on_error true

      <buffer>

        @type file

        path /var/log/fluentd-buffers/kubernetes.system.buffer

        flush_mode interval

        retry_type exponential_backoff

        flush_thread_count 2

        flush_interval 5s

        retry_forever

        retry_max_interval 30

        chunk_limit_size "#{ENV['OUTPUT_BUFFER_CHUNK_LIMIT']}"

        queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT']}"

        overflow_action block

      </buffer>

    </match>

 

# extraVolumes:

#   - name: es-certs

#     secret:

#       defaultMode: 420

#       secretName: es-certs

# extraVolumeMounts:

#   - name: es-certs

#     mountPath: /certs

#     readOnly: true

安装

[root@k8s-master01 fluentd-elasticsearch]# helmv2 install --name flu1 --namespace=efk -f values.yaml .

NAME:   flu1

LAST DEPLOYED: Fri Feb  3 14:26:27 2023

NAMESPACE: efk

STATUS: DEPLOYED

 

RESOURCES:

==> v1/ClusterRole

NAME                        AGE

flu1-fluentd-elasticsearch  1s

 

==> v1/ClusterRoleBinding

NAME                        AGE

flu1-fluentd-elasticsearch  0s

 

==> v1/ConfigMap

NAME                        DATA  AGE

flu1-fluentd-elasticsearch  6     1s

 

==> v1/DaemonSet

NAME                        DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE

flu1-fluentd-elasticsearch  2        2        0      2           0          <none>         0s

 

==> v1/Pod(related)

NAME                              READY  STATUS             RESTARTS  AGE

flu1-fluentd-elasticsearch-2kbm6  0/1    ContainerCreating  0         0s

flu1-fluentd-elasticsearch-pklpw  0/1    ContainerCreating  0         0s

 

==> v1/ServiceAccount

NAME                        SECRETS  AGE

flu1-fluentd-elasticsearch  1        1s

 

 

NOTES:

1. To verify that Fluentd has started, run:

 

  kubectl --namespace=efk get pods -l "app.kubernetes.io/name=fluentd-elasticsearch,app.kubernetes.io/instance=flu1"

 

THIS APPLICATION CAPTURES ALL CONSOLE OUTPUT AND FORWARDS IT TO elasticsearch . Anything that might be identifying,

including things like IP addresses, container images, and object names will NOT be anonymized.

需要的docker镜像依赖

docker pull gcr.io/google-containers/fluentd-elasticsearch:v2.3.2

依赖的镜像导入后等待running

[root@k8s-master01 fluentd-elasticsearch]# kubectl get pod -n efk

NAME                                         READY   STATUS    RESTARTS   AGE

els1-elasticsearch-client-59bcdcbfb7-7xvs7   1/1     Running   2          42m

els1-elasticsearch-data-0                    1/1     Running   0          42m

els1-elasticsearch-master-0                  1/1     Running   0          42m

flu1-fluentd-elasticsearch-2kbm6             1/1     Running   0          17m

flu1-fluentd-elasticsearch-pklpw             1/1     Running   0          17m

Step4

开始部署k集群

[root@k8s-master01 fluentd-elasticsearch]# cd ..

[root@k8s-master01 efk]# helm fetch stable/kibana --version 0.14.8

Error: chart "kibana" matching 0.14.8 not found in stable index. (try 'helm repo update'): no chart version found for kibana-0.14.8

又出差。。。

Repo更新后亦不行

[root@k8s-master01 efk]# helm repo list

NAME            URL

stable          https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

incubator       https://charts.helm.sh/incubator

[root@k8s-master01 efk]#

[root@k8s-master01 efk]#

[root@k8s-master01 efk]# helm repo update

Hang tight while we grab the latest from your chart repositories...

...Successfully got an update from the "stable" chart repository

...Successfully got an update from the "incubator" chart repository

Update Complete. ⎈Happy Helming!⎈

[root@k8s-master01 efk]# helm fetch stable/kibana --version 0.14.8

Error: chart "kibana" matching 0.14.8 not found in stable index. (try 'helm repo update'): no chart version found for kibana-0.14.8

更换helm仓库ok

[root@k8s-master01 efk]# helm repo remove stable

"stable" has been removed from your repositories

[root@k8s-master01 efk]# helm repo add stable  https://charts.helm.sh/stable

"stable" has been added to your repositories

[root@k8s-master01 efk]# helm fetch stable/kibana --version 0.14.8

解压进入

[root@k8s-master01 efk]# tar -zxvf kibana-0.14.8.tgz

kibana/Chart.yaml

tar: kibana/Chart.yaml：不可信的旧时间戳 1970-01-01 08:00:00

kibana/values.yaml

tar: kibana/values.yaml：不可信的旧时间戳 1970-01-01 08:00:00

kibana/templates/NOTES.txt

tar: kibana/templates/NOTES.txt：不可信的旧时间戳 1970-01-01 08:00:00

kibana/templates/_helpers.tpl

tar: kibana/templates/_helpers.tpl：不可信的旧时间戳 1970-01-01 08:00:00

kibana/templates/configmap-dashboardimport.yaml

tar: kibana/templates/configmap-dashboardimport.yaml：不可信的旧时间戳 1970-01-01 08:00:00

kibana/templates/configmap.yaml

tar: kibana/templates/configmap.yaml：不可信的旧时间戳 1970-01-01 08:00:00

kibana/templates/deployment.yaml

tar: kibana/templates/deployment.yaml：不可信的旧时间戳 1970-01-01 08:00:00

kibana/templates/ingress.yaml

tar: kibana/templates/ingress.yaml：不可信的旧时间戳 1970-01-01 08:00:00

kibana/templates/service.yaml

tar: kibana/templates/service.yaml：不可信的旧时间戳 1970-01-01 08:00:00

kibana/.helmignore

tar: kibana/.helmignore：不可信的旧时间戳 1970-01-01 08:00:00

kibana/OWNERS

tar: kibana/OWNERS：不可信的旧时间戳 1970-01-01 08:00:00

kibana/README.md

tar: kibana/README.md：不可信的旧时间戳 1970-01-01 08:00:00

[root@k8s-master01 efk]# cd kibana

配置elasticsearch访问url（svc地址）

[root@k8s-master01 kibana]# cat values.yaml

image:

  repository: "docker.elastic.co/kibana/kibana-oss"

  tag: "6.4.2"

  pullPolicy: "IfNotPresent"

 

commandline:

  args:

 

env: {}

  # All Kibana configuration options are adjustable via env vars.

  # To adjust a config option to an env var uppercase + replace `.` with `_`

  # Ref: https://www.elastic.co/guide/en/kibana/current/settings.html

  #

  # ELASTICSEARCH_URL: http://elasticsearch-client:9200

  # SERVER_PORT: 5601

  # LOGGING_VERBOSE: "true"

  # SERVER_DEFAULTROUTE: "/app/kibana"

 

files:

  kibana.yml:

    ## Default Kibana configuration from kibana-docker.

    server.name: kibana

    server.host: "0"

    elasticsearch.url: http://10.100.179.134:9200

 

    ## Custom config properties below

    ## Ref: https://www.elastic.co/guide/en/kibana/current/settings.html

    # server.port: 5601

    # logging.verbose: "true"

    # server.defaultRoute: "/app/kibana"

 

service:

  type: ClusterIP

  externalPort: 443

  internalPort: 5601

  # authProxyPort: 5602 To be used with authProxyEnabled and a proxy extraContainer

  ## External IP addresses of service

  ## Default: nil

  ##

  # externalIPs:

  # - 192.168.0.1

  #

  ## LoadBalancer IP if service.type is LoadBalancer

  ## Default: nil

  ##

  # loadBalancerIP: 10.2.2.2

  annotations:

    # Annotation example: setup ssl with aws cert when service.type is LoadBalancer

    # service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:EXAMPLE_CERT

  labels:

    ## Label example: show service URL in `kubectl cluster-info`

    # kubernetes.io/cluster-service: "true"

  ## Limit load balancer source ips to list of CIDRs (where available)

  # loadBalancerSourceRanges: []

 

ingress:

  enabled: false

  # hosts:

    # - chart-example.local

  # annotations:

  #   kubernetes.io/ingress.class: nginx

  #   kubernetes.io/tls-acme: "true"

  # tls:

    # - secretName: chart-example-tls

    #   hosts:

    #     - chart-example.local

 

# service account that will run the pod. Leave commented to use the default service account.

# serviceAccountName: kibana

 

livenessProbe:

  enabled: false

  initialDelaySeconds: 30

  timeoutSeconds: 10

 

readinessProbe:

  enabled: false

  initialDelaySeconds: 30

  timeoutSeconds: 10

 

# Enable an authproxy. Specify container in extraContainers

authProxyEnabled: false

 

extraContainers: |

# - name: proxy

#   image: quay.io/gambol99/keycloak-proxy:latest

#   args:

#     - --resource=uri=/*

#     - --discovery-url=https://discovery-url

#     - --client-id=client

#     - --client-secret=secret

#     - --listen=0.0.0.0:5602

#     - --upstream-url=http://127.0.0.1:5601

#   ports:

#     - name: web

#       containerPort: 9090

resources: {}

  # limits:

  #   cpu: 100m

  #   memory: 300Mi

  # requests:

  #   cpu: 100m

  #   memory: 300Mi

 

priorityClassName: ""

 

# Affinity for pod assignment

# Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

# affinity: {}

 

# Tolerations for pod assignment

# Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

tolerations: []

 

# Node labels for pod assignment

# Ref: https://kubernetes.io/docs/user-guide/node-selection/

nodeSelector: {}

 

podAnnotations: {}

replicaCount: 1

revisionHistoryLimit: 3

 

# to export a dashboard from a running kibana 6.3.x use:

# curl --user <username>:<password> -XGET https://kibana.yourdomain.com:5601/api/kibana/dashboards/export?dashboard=<some-dashboard-uuid> > my-dashboard.json

# you can find an example dashboard for kubernests with fluentd-elasticsearch chart here: https://github.com/monotek/kibana-dashboards/blob/master/k8s-fluentd-elasticsearch.json

dashboardImport:

  xpackauth:

    enabled: false

    username: myuser

    password: mypass

  dashboards: {}

 

# List of pluginns to install using initContainer

plugins:

  # - https://github.com/sivasamyk/logtrail/releases/download/v0.1.29/logtrail-6.4.0-0.1.29.zip

  # - other_plugin

需要依赖的镜像

docker pull docker.elastic.co/kibana/kibana-oss:6.4.2

然后安装

[root@k8s-master01 kibana]# helmv2 install --name kib1 --namespace=efk -f values.yaml .

NAME:   kib1

LAST DEPLOYED: Fri Feb  3 15:19:13 2023

NAMESPACE: efk

STATUS: DEPLOYED

 

RESOURCES:

==> v1/ConfigMap

NAME         DATA  AGE

kib1-kibana  1     0s

 

==> v1/Pod(related)

NAME                         READY  STATUS   RESTARTS  AGE

kib1-kibana-6c49f68cf-4lblm  0/1    Pending  0         0s

 

==> v1/Service

NAME         TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE

kib1-kibana  ClusterIP  10.110.166.165  <none>       443/TCP  0s

 

==> v1beta1/Deployment

NAME         READY  UP-TO-DATE  AVAILABLE  AGE

kib1-kibana  0/1    1           0          0s

 

 

NOTES:

To verify that kib1-kibana has started, run:

 

  kubectl --namespace=efk get pods -l "app=kibana"

 

Kibana can be accessed:

 

  * From outside the cluster, run these commands in the same shell:

 

    export POD_NAME=$(kubectl get pods --namespace efk -l "app=kibana,release=kib1" -o jsonpath="{.items[0].metadata.name}")

    echo "Visit http://127.0.0.1:5601 to use Kibana"

kubectl port-forward --namespace efk $POD_NAME 5601:5601

据说e和k的版本是要对应的，比较严格。

已运行

[root@k8s-master01 kibana]# kubectl get pod -n efk

NAME                                         READY   STATUS    RESTARTS   AGE

els1-elasticsearch-client-59bcdcbfb7-7xvs7   1/1     Running   2          78m

els1-elasticsearch-data-0                    1/1     Running   0          78m

els1-elasticsearch-master-0                  1/1     Running   0          78m

flu1-fluentd-elasticsearch-2kbm6             1/1     Running   0          53m

flu1-fluentd-elasticsearch-pklpw             1/1     Running   0          53m

kib1-kibana-6c49f68cf-4lblm                  1/1     Running   0          72s

kibana是也由svc，但默认不暴露到节点端口

[root@k8s-master01 kibana]# kubectl get svc -n efk

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE

els1-elasticsearch-client      ClusterIP   10.100.179.134   <none>        9200/TCP   79m

els1-elasticsearch-discovery   ClusterIP   None             <none>        9300/TCP   79m

kib1-kibana                    ClusterIP   10.110.166.165   <none>        443/TCP    2m18s

 

向节点暴漏

[root@k8s-master01 kibana]# kubectl edit svc kib1-kibana -n efk

service/kib1-kibana edited

 

spec：

type: NodePort

获得暴露节点端口

[root@k8s-master01 kibana]# kubectl get svc -n efk

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE

els1-elasticsearch-client      ClusterIP   10.100.179.134   <none>        9200/TCP        82m

els1-elasticsearch-discovery   ClusterIP   None             <none>        9300/TCP        82m

kib1-kibana                    NodePort    10.110.166.165   <none>        443:32594/TCP   5m20s

 

 

虽然部署在节点1上，但是master也可访问

[root@k8s-master01 kibana]# kubectl get pod -n efk -o wide

NAME                                         READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES

els1-elasticsearch-client-59bcdcbfb7-7xvs7   1/1     Running   2          86m     192.168.58.194   k8s-node02   <none>           <none>

els1-elasticsearch-data-0                    1/1     Running   0          86m     192.168.58.195   k8s-node02   <none>           <none>

els1-elasticsearch-master-0                  1/1     Running   0          86m     192.168.85.193   k8s-node01   <none>           <none>

flu1-fluentd-elasticsearch-2kbm6             1/1     Running   0          62m     192.168.85.195   k8s-node01   <none>           <none>

flu1-fluentd-elasticsearch-pklpw             1/1     Running   0          62m     192.168.58.197   k8s-node02   <none>           <none>

kib1-kibana-6c49f68cf-4lblm                  1/1     Running   0          9m36s   192.168.85.196   k8s-node01   <none>           <none>

 

http://192.168.66.10:32594/



可能是calico网络插件的作用，flannel貌似有问题，但是也可能是rbac配置问题。

Step5

Efk分布式日志系统

可能是新的问题



内存不足？



 

 

快速调试

删除elasticsearch

helmv2 delete --purge els1

重建elasticsearch

helmv2 install --name els1 --namespace=efk -f values.yaml .

查看elasticsearch访问svc地址

kubectl get svc -n efk | grep client

删除fluentd-elasticsearch

helmv2 delete --purge flu1

配置elasticsearch访问地址

vi values.yaml

       elasticsearch

              host

重建fluentd-elasticsearch

helmv2 install --name flu1 --namespace=efk -f values.yaml .

 

删除kibana

helmv2 delete --purge kib1

配置kibana

vi values.yaml

files

 kibana.yml

   elasticsearch.url

重建kibana

helmv2 install --name kib1 --namespace=efk -f values.yaml .

配置kibana的svc为端口访问

kubectl edit svc kib1-kibana -n efk

spec

type: NodePort

查看kibana外部访问端口

kubectl get svc -n efk | grep kib1

 

 

 

 

 

 

 难道是重启次数大于0，甚至于达到2导致

[root@k8s-master01 kibana]# kubectl get pod -n efk

NAME                                         READY   STATUS    RESTARTS   AGE

els1-elasticsearch-client-59bcdcbfb7-tt8fl   1/1     Running   2          30m

els1-elasticsearch-data-0                    1/1     Running   0          36m

els1-elasticsearch-master-0                  1/1     Running   0          15m

flu1-fluentd-elasticsearch-2kncj             1/1     Running   0          12m

flu1-fluentd-elasticsearch-5sw8l             1/1     Running   0          12m

kib1-kibana-6c49f68cf-5xpf4                  1/1     Running   0          2m49s

 

 

查看日志

[root@k8s-master01 elasticsearch]# kubectl logs kib1-kibana-6c49f68cf-5xpf4 -n efk

选取一部分

 

{"type":"response","@timestamp":"2023-02-03T09:42:18Z","tags":[],"pid":1,"method":"get","statusCode":304,"req":{"url":"/bundles/4b5a84aaf1c9485e060c503a0ff8cadb.woff2","method":"get","headers":{"host":"192.168.66.10:30821","connection":"keep-alive","origin":"http://192.168.66.10:30821","if-none-match":"\"574ea2698c03ae9477db2ea3baf460ee32f1a7ea\"","if-modified-since":"Wed, 26 Sep 2018 13:58:44 GMT","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.70","accept":"*/*","referer":"http://192.168.66.10:30821/bundles/commons.style.css","accept-encoding":"gzip, deflate","accept-language":"zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6"},"remoteAddress":"192.168.32.128","userAgent":"192.168.32.128","referer":"http://192.168.66.10:30821/bundles/commons.style.css"},"res":{"statusCode":304,"responseTime":1,"contentLength":9},"message":"GET /bundles/4b5a84aaf1c9485e060c503a0ff8cadb.woff2 304 1ms - 9.0B"}

有304错误

 

 

这个先留这里待继续关注。

对于上面提到的说是性能或内存不够之类的，我家里笔记本16g几乎全部用到k8s，比起公司还少一个我平常工作的节点，省了2g吧或者4g吧。（忘了公司那个工作节点的内存了）。

但即便这样也会出现Service Unavailable错误。

但有一点就算efk名称空间的所有pod都重启数为0，就很棒哈。

难道是前面的etcd问题？

 

最后就是说实际上我这里发现我是有安装etcd的。。。

[root@k8s-master01 kibana]# kubectl get pod -n kube-system | grep etcd

etcd-k8s-master01                          1/1     Running   0          111m

所以暂时就先这样吧，后面如果有机会再解决呗。

 

按正常情况应是可以进入这里创建基于时间序列分片





以今天时间创建索引序列



通过时间序列去分片



选择timestamp去创建



 

 



 

查看日志，比较慢，需要稍微等下



 

 

有没有点这里呢？



 

然后就出来了