ELK-k8s部署ELK集群
阿俊
|
|
1,195
|
0
|
Centos,ELK,Kubernetes,linux

879 字
|
36 分钟
本文最后更新于 1071 天前，其中的信息可能已经有所发展或是发生改变。

ELK-k8s部署ELK集群

系统架构

image-20230913154432607

因测试or生产集群模式都是3master，2node，因此采用基本上2工作节点上部署。

创建StorageClass和Namespace

创建StorageClass

sc.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: logging-local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

因为使用是本地磁盘，需要在node节点创建目录：mkdir -p /data/elk/es

创建Namespace

ns.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: logging
  labels:
    app: elk

创建证书

docker run --name es-certutil -i -w /tmp elasticsearch:7.17.9 /bin/sh -c  \
    "elasticsearch-certutil ca --out /tmp/es-ca.p12 --pass '' && \
    elasticsearch-certutil cert --name security-master --dns \
    security-master --ca /tmp/es-ca.p12 --pass '' --ca-pass '' --out /tmp/elastic-certificates.p12"

docker cp es-certutil:/tmp/elastic-certificates.p12 ./

docker rm -f es-certutil

生成 ssl 认证要使用的 secret elastic-certificates：

# 生成 ssl 认证要使用的 secret elastic-certificates
sudo kubectl -n logging create secret generic elastic-certificates --from-file=./elastic-certificates.p12

部署ES集群

elasticsearch.configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: logging
  name: elk-elasticsearch-config
  labels:
    app: elk-elasticsearch
data:
  elasticsearch.yml: |-
    cluster.name: elk-elasticsearch-cluster
    # 发现节点的地址，discovery.seed_hosts的值应包括所有master候选节点
    discovery.seed_hosts: elk-elasticsearch
    # 初始化集群时，ES从中选出master节点
    # 对应metadata.name名称加编号，编号从0开始
    # cluster.initial_master_nodes: elk-elasticsearch-0,elk-elasticsearch-1,elk-elasticsearch-2,elk-elasticsearch-3,elk-elasticsearch-4
    cluster.initial_master_nodes: elk-elasticsearch-0,elk-elasticsearch-1
    discovery.zen.minimum_master_nodes: 2

    http.cors.enabled: true
    http.cors.allow-origin: "*"
    network.host: 0.0.0.0

    node.master: true
    node.data: true
    node.ingest: true
    ingest.geoip.downloader.enabled: false

    xpack.security.enabled: true
    xpack.monitoring.collection.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12

elasticsearch.pvc.yaml

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: logging-local-storage-elk-es-pv-0
  namespace: logging
  labels:
    name: logging-local-storage-elk-es-pv-0
spec:
  capacity:
    storage: 250Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: logging-local-storage
  local:
    path: /data/elk/es
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-n01

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: logging-local-storage-elk-es-pv-1
  namespace: logging
  labels:
    name: logging-local-storage-elk-es-pv-1
spec:
  capacity:
    storage: 250Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: logging-local-storage
  local:
    path: /data/elk/es
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-n02

#---
#apiVersion: v1
#kind: PersistentVolume
#metadata:
#  name: logging-local-storage-elk-es-pv-2
#  namespace: logging
#  labels:
#    name: logging-local-storage-elk-es-pv-2
#spec:
#  capacity:
#    storage: 250Gi
#  accessModes:
#    - ReadWriteOnce
#  persistentVolumeReclaimPolicy: Retain
#  storageClassName: logging-local-storage
#  local:
#    path: /data/elk/es
#  nodeAffinity:
#    required:
#      nodeSelectorTerms:
#        - matchExpressions:
#            - key: kubernetes.io/hostname
#              operator: In
#              values:
#                - k8s-m01
#
#---
#apiVersion: v1
#kind: PersistentVolume
#metadata:
#  name: logging-local-storage-elk-es-pv-3
#  namespace: logging
#  labels:
#    name: logging-local-storage-elk-es-pv-3
#spec:
#  capacity:
#    storage: 250Gi
#  accessModes:
#    - ReadWriteOnce
#  persistentVolumeReclaimPolicy: Retain
#  storageClassName: logging-local-storage
#  local:
#    path: /data/elk/es
#  nodeAffinity:
#    required:
#      nodeSelectorTerms:
#        - matchExpressions:
#            - key: kubernetes.io/hostname
#              operator: In
#              values:
#                - k8s-m02
#
#---
#apiVersion: v1
#kind: PersistentVolume
#metadata:
#  name: logging-local-storage-elk-es-pv-4
#  namespace: logging
#  labels:
#    name: logging-local-storage-elk-es-pv-4
#spec:
#  capacity:
#    storage: 250Gi
#  accessModes:
#    - ReadWriteOnce
#  persistentVolumeReclaimPolicy: Retain
#  storageClassName: logging-local-storage
#  local:
#    path: /data/elk/es
#  nodeAffinity:
#    required:
#      nodeSelectorTerms:
#        - matchExpressions:
#            - key: kubernetes.io/hostname
#              operator: In
#              values:
#                - k8s-m03

elasticsearch.statefulset.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elk-elasticsearch
  namespace: logging
  labels:
    app: elk-elasticsearch
spec:
  serviceName: elk-elasticsearch
  replicas: 2
  selector:
    matchLabels:
      app: elk-elasticsearch
  template:
    metadata:
      labels:
        app: elk-elasticsearch
    spec:
      initContainers:
      - name: elasticsearch-logging-init
        image: 10.194.24.53/k8s-component/alpine:3.18
        command: [ "/sbin/sysctl", "-w", "vm.max_map_count=262144" ]
        securityContext:
          privileged: true

      - name: elasticsearch-volume-init
        image: 10.194.24.53/k8s-component/alpine:3.18
        command:
          - chmod
          - -R
          - "777"
          - /usr/share/elasticsearch/data/
        volumeMounts:
          - name: elasticsearch-storage-data
            mountPath: /usr/share/elasticsearch/data/

      containers:
        - name: elasticsearch
          image: 10.194.24.53/k8s-component/elk/elasticsearch:7.17.9
          command:
            - /bin/bash
            - '-c'
            - |-
              #!/bin/bash
              ulimit -l unlimited
              echo ulimit
              exec su elasticsearch /usr/local/bin/docker-entrypoint.sh

          resources:
            limits: #最大
              memory: 6144Mi
            requests:
              memory: 5120Mi
          ports:
            - containerPort: 9200
              name: http
              protocol: TCP

            - containerPort: 9300
              name: transport
              protocol: TCP
          env:
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

            - name: ES_JAVA_OPTS
              value: "-Xms5120m -Xmx5120m"

            - name: TZ
              value: Asia/Shanghai

            - name: TIME_ZONE
              value: Asia/Shanghai

          volumeMounts:
            - name: elastic-certificates
              readOnly: true
              mountPath: /usr/share/elasticsearch/config/elastic-certificates.p12
              subPath: elastic-certificates.p12

            - name: config
              mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
              readOnly: true
              subPath: elasticsearch.yml

            - name: elasticsearch-storage-data
              mountPath: /usr/share/elasticsearch/data

            - name: localtime
              mountPath: /etc/localtime


      volumes:
        - name: elastic-certificates
          secret:
            secretName: elastic-certificates

        - name: localtime
          hostPath:
            path: /etc/localtime

        - name: config
          configMap:
            name: elk-elasticsearch-config

      tolerations:
        - operator: Exists

  volumeClaimTemplates:
    - metadata:
        name: elasticsearch-storage-data
        namespace: logging
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "logging-local-storage"
        resources:
          requests:
            storage: 250Gi

elasticsearch.service.yaml

---
apiVersion: v1
kind: Service
metadata:
  namespace: logging
  name: elk-elasticsearch
  labels:
    app: elk-elasticsearch-svc
spec:
  type: NodePort
  ports:
    - port: 9200
      name: http
      nodePort: 39200
    - port: 9300
      name: transport
      nodePort: 39201
  selector:
    app: elk-elasticsearch

设置ES集群密码

#执行：自动生成随机密码并打印终端
# kubectl -n logging exec -it $(kubectl -n logging get pods | grep elk-elasticsearch | sed -n 1p | awk '{print $1}') -- bin/elasticsearch-setup-passwords auto -b
# 自行配置密码
kubectl -n logging exec -it $(kubectl -n logging get pods | grep elk-elasticsearch | sed -n 1p | awk '{print $1}') -- bin/elasticsearch-setup-passwords  interactive


Changed password for user apm_system
PASSWORD apm_system = xxx

Changed password for user kibana_system
PASSWORD kibana_system = xxx

Changed password for user kibana
PASSWORD kibana = xxx

Changed password for user logstash_system
PASSWORD logstash_system = xxx

Changed password for user beats_system
PASSWORD beats_system = xxx

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = xxx

Changed password for user elastic
PASSWORD elastic = xxx

部署 kibana

kibana.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: logging
  name: kibana-config
  labels:
    app: kibana
data:
  kibana.yml: |-
    server.host: 0.0.0.0
    i18n.locale: "zh-CN"
    monitoring.ui.container.elasticsearch.enabled: true
    elasticsearch:
      hosts: http://elk-elasticsearch:9200
      username: elastic
      password: xxx

    xpack.encryptedSavedObjects.encryptionKey: min-32-byte-long-strong-encryption-key
---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logging-kibana-data-pvc
  namespace: logging
  labels:
    app: logging-kibana-data-pvc
spec:
  storageClassName: k8s-data
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: logging
  labels:
    k8s-app: kibana
    srv: srv-kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      containers:
      - name: kibana
        image: 10.194.24.53/k8s-component/elk/kibana:7.17.9
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: SERVER_PUBLICBASEURL
            value: "http://0.0.0.0:5601"
          - name: TZ
            value: Asia/Shanghai
          - name: TIME_ZONE
            value: Asia/Shanghai

        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
        volumeMounts:

          - name: kibana-config
            mountPath: /usr/share/kibana/config/kibana.yml
            readOnly: true
            subPath: kibana.yml

          - mountPath: /etc/localtime
            name: localtime

          - name: kibana-data
            mountPath: /usr/share/kibana/plugins
      volumes:
        - name: kibana-config
          configMap:
            name: kibana-config

        - hostPath:
            path: /etc/localtime
          name: localtime

        - name: kibana-data
          persistentVolumeClaim:
            claimName: logging-kibana-data-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
  namespace: logging
  labels:
    k8s-app: kibana
    kubernetes.io/name: "Kibana"
    svc: kibana-svc
spec:
  type: ClusterIP
  ports:
    - port: 5601
      protocol: TCP
      targetPort: ui
  selector:
    k8s-app: kibana

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: logging
  name: kibana-ing
spec:
  rules:
    - host: kibana.xxx.com
      http:
        paths:
          - backend:
              service:
                name: kibana-svc
                port:
                  number: 5601
            pathType: Prefix
            path: /

部署 logstash

因后续会采集Nginx日志，也提前存放了GeoLite2-City.mmdbip数据库

logstash.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
  namespace: logging
  labels:
    app: logstash
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
    config.reload.automatic: true
    config.reload.interval: 300s
  logstash.conf: |-
    input {
      beats {
        type => "beat-nginx-logs"
        port => 5044
        codec => json
        client_inactivity_timeout => 3600
      }
    }

    filter {
      if [type] == "beat-nginx-logs" {
        # nginx 日志
        if [xff] != ""{
          geoip {
            target => "geoip"
            source => "xff"
            database => "/usr/share/logstash/GeoLite2-City.mmdb"
            add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
            add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
            # 去掉显示 geoip 显示的多余信息
            remove_field => ["[geoip][latitude]", "[geoip][longitude]", "[geoip][country_code]", "[geoip][country_code2]", "[geoip][country_code3]", "[geoip][timezone]", "[geoip][continent_code]", "[geoip][region_code]"]
          }
        }else{
          geoip {
            target => "geoip"
            source => "client_ip"
            database => "/usr/share/logstash/GeoLite2-City.mmdb"
            add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
            add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
            # 去掉显示 geoip 显示的多余信息
            remove_field => ["[geoip][latitude]", "[geoip][longitude]", "[geoip][country_code]", "[geoip][country_code2]", "[geoip][country_code3]", "[geoip][timezone]", "[geoip][continent_code]", "[geoip][region_code]"]
          }
        }

        mutate {
          convert => [ "size", "integer" ]
          convert => [ "status", "integer" ]
          convert => [ "responsetime", "float" ]
          convert => [ "upstreamtime", "float" ]
          convert => [ "[geoip][coordinates]", "float" ]
          # 过滤 filebeat 没用的字段,这里过滤的字段要考虑好输出到es的，否则过滤了就没法做判断
          # remove_field => [ "ecs","agent","host","cloud","@version","input","logs_type" ]
          remove_field => [ "ecs","agent","cloud","@version","input" ]
        }
        # 根据 http_user_agent来自动处理区分用户客户端系统与版本
        useragent {
          source => "http_user_agent"
          target => "ua"
          # 过滤useragent没用的字段
          remove_field => [ "[ua][minor]","[ua][major]","[ua][build]","[ua][patch]","[ua][os_minor]","[ua][os_major]" ]
        }
      }
    }

    output {
      if [type] == "beat-nginx-logs" {
        elasticsearch {
          hosts => "elk-elasticsearch:9200"
          user => "elastic"
          password => "xxx"
          index => "logstash-nginx-log-%{+yyyy.MM.dd}"
        }
      } else {
        stdout { }
      }
    }

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elk-logstash
  namespace: logging
spec:
  selector:
    matchLabels:
      app: logstash
  replicas: 2
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
        - name: logstash
          image: 10.194.24.53/k8s-component/elk/logstash:7.17.9
          ports:
            - containerPort: 5044
              name: nginx-port

            - containerPort: 5045
              name: tcp-test-port

          volumeMounts:
            - name: config-volume
              mountPath: /usr/share/logstash/config/logstash.yml
              readOnly: true
              subPath: logstash.yml

            - name: logstash-pipeline-volume
              mountPath: /usr/share/logstash/pipeline/logstash.conf
              readOnly: true
              subPath: logstash.conf

            - mountPath: /etc/localtime
              name: localtime

            - mountPath: /usr/share/logstash/GeoLite2-City.mmdb
              name: geolite2

          env:
            - name: TZ
              value: Asia/Shanghai

            - name: TIME_ZONE
              value: Asia/Shanghai

      volumes:
        - name: config-volume
          configMap:
            name: logstash-configmap
            items:
              - key: logstash.yml
                path: logstash.yml

        - name: logstash-pipeline-volume
          configMap:
            name: logstash-configmap
            items:
              - key: logstash.conf
                path: logstash.conf

        - hostPath:
            path: /etc/localtime
          name: localtime

        - hostPath:
            path: /glj_k8s/GeoLite2-City.mmdb
          name: geolite2

---
kind: Service
apiVersion: v1
metadata:
  name: logstash-svc
  namespace: logging
  labels:
    app: logstash
spec:
  selector:
    app: logstash
  type: NodePort
  ports:
    - name: nginx-port
      port: 5044
      nodePort: 35044
    - name: tcp-test-port
      port: 5045
      nodePort: 35045

部署 filebeat

后续加入采集容器日志，暂时只是部署Nginx服务器采集日志分析

参考：https://blog.csdn.net/weixin_43334786/article/details/122435701

ELKk8s
暂无评论

发送评论 编辑评论 


				

			

						

				

					

						

							

								
							

							
						

					

				

				

					

						

							

								
							

							
						

					

				

				

					

						

							

								
							

							
							
													

					

				

			

			
			

				

					
				

			

				

											

							
							
						

																					

							
							
						

										
					
											
						

	

		
|´・ω・)ノ
ヾ(≧∇≦*)ゝ
(☆ω☆)
（╯‵□′）╯︵┴─┴
￣﹃￣
(/ω＼)
∠( ᐛ 」∠)＿
(๑•̀ㅁ•́ฅ)
→_→
୧(๑•̀⌄•́๑)૭
٩(ˊᗜˋ*)و
(ノ°ο°)ノ
(´இ皿இ｀)
⌇●﹏●⌇
(ฅ´ω`ฅ)
(╯°A°)╯︵○○○
φ(￣∇￣o)
ヾ(´･ ･｀｡)ノ"
( ง ᵒ̌皿ᵒ̌)ง⁼³₌₃
(ó﹏ò｡)
Σ(っ °Д °;)っ
( ,,´･ω･)ﾉ"(´っω･｀｡)
╮(╯▽╰)╭
o(*////▽////*)q
＞﹏＜
( ๑´•ω•) "(ㆆᴗㆆ)
😂
😀
😅
😊
🙂
🙃
😌
😍
😘
😜
😝
😏
😒
🙄
😳
😡
😔
😫
😱
😭
💩
👻
🙌
🖕
👍
👫
👬
👭
🌚
🌝
🙈
💊
😶
🙏
🍦
🍉
😣
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Image
Source: github.com/k4yt3x/flowerhd
	

	

		
颜文字
Emoji
小恐龙
花!
	


									

			

			
			
		

	






上一篇
下一篇