Prometheus, Grafana and Loki Installation (Kubeadm)

Setup Metrics Server for KubeAdm

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl edit deploy metrics-server -n kube-system

Add the following lines as part of the -args:

command:

- /metrics-server - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP

Increase the number of open files allowed on the server

sudo vi /etc/security/limits.conf

Add the following lines to the bottom of the file

* hard nofile 1000000
* soft nofile 1000000

Increase inotify limits

sudo vi /etc/sysctl.conf

Add the following lines to the end of the file

fs.inotify.max_user_instances=8192 
fs.inotify.max_user_watches=524288

Apply changes

sudo sysctl --system

Setup Persistent Folders

sudo mkdir -p /data/prometheus
sudo mkdir -p /data/grafana
sudo mkdir -p /data/loki

Change Permissions for Persistent Volume Folders

sudo chmod -R 777 /data/prometheus
sudo chmod -R 777 /data/grafana
sudo chmod -R 777 /data/loki

Create PV and PVC’s

kubectl create ns monitoring

Create Storage Class

vi storageClass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

kubectl apply -f storageClass.yaml -n monitoring
vi persistentVolumes.yaml

NOTE: Modify lines 21, 44, 67 and 91 to reflect your configuration.

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: loki-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/loki
  nodeAffinity:
    required:
       nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01  # Replace with your node name
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-alert-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/prometheus
  nodeAffinity:
    required:
       nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01  # Replace with your node name

kubectl apply -f persistentVolumes.yaml -n monitoring
vi persistentVolumeClaims.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-server
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage
  volumeMode: Filesystem
  volumeName: prometheus-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: storage-prometheus-alertmanager-0
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: local-storage
  volumeMode: Filesystem
  volumeName: prometheus-alert-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: local-storage
  volumeMode: Filesystem
  volumeName: grafana-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: storage-loki-0
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage
  volumeMode: Filesystem
  volumeName: loki-pv

kubectl apply -f persistentVolumeClaims.yaml -n monitoring

Add and Update Helm Repo’s

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
helm repo add grafana https://grafana.github.io/helm-charts 
helm repo update

Install Helm Charts

vi prometheus-values.yaml

server:
  global:
    scrape_interval: 10s     # Default is 1m - set to your desired interval
    scrape_timeout: 7s
     evaluation_interval: 10s
  persistentVolume:
    enabled: true
    existingClaim: "prometheus-server"
    storageClass: "local-storage"
    size: 10Gi
alertmanager:
  persistentVolume:
    enabled: true
    existingClaim: "storage-prometheus-alertmanager-0"
    storageClass: "local-storage"
    size: 2Gi

helm upgrade --install prometheus prometheus-community/prometheus -f prometheus-values.yaml -n monitoring
vi grafana-values.yaml

persistence:
  enabled: true
  existingClaim: "grafana"
  storageClassName: local-storage
  size: 5Gi
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.monitoring.svc.cluster.local
      access: proxy
      isDefault: true
    - name: Loki
      type: loki
      access: proxy
      url: http://loki.monitoring.svc.cluster.local:3100/

helm upgrade --install grafana grafana/grafana -f grafana-values.yaml -n monitoring
vi loki-values.yaml

deploymentMode: SingleBinary
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1  
  schemaConfig:
    configs:
      - from: "2020-10-24"
        store: boltdb-shipper
        object_store: filesystem
        schema: v11
        index:
          prefix: index_
          period: 24h
  storage:
    type: filesystem
  limits_config:
     allow_structured_metadata: false
    volume_enabled: true
singleBinary:
  replicas: 1
  persistence:
    enabled: true
    existingClaim: "storage-loki-0"
    storageClass: local-storage
    size: 10Gi
  extraArgs: []
canary:
  enabled: false
gateway:
  enabled: false
read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0

helm upgrade --install loki grafana/loki -f loki-values.yaml -n monitoring
vi promtail-values.yaml

config:
  clients:
    - url: http://loki:3100/loki/api/v1/push
  positions:
    filename: /data/positions.yaml
  scrape_configs:
    - job_name: varlogs
      static_configs:
        - targets:
            - localhost
          labels:
            job: varlogs
            __path__: /var/log/*.log
# Optional: persistent volume for positions file
persistence:
  enabled: true
  size: 2Gi

helm upgrade --install promtail grafana/promtail -f promtail-values.yaml -n monitoring

Install NGINX Ingress for Grafana

vi grafana-ingress.yaml

NOTE: Remember to make sure that the host specified in line 9 matches your configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
    - host: grafana.testmachine.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                   number: 80

kubectl apply -f grafana-ingress.yaml -n monitoring

Retrieve Grafana password

kubectl get secret grafana -o jsonpath="{.data.admin-password}" -n monitoring | base64 -d ; echo

Open the Grafana server in your browser http://grafana.testmachine.com
Username: admin
Password: <as per command above>

Add Dashboard

Download the following Grafana Dashboard example https://lumenvox-public-assets.s3.amazonaws.com/third-party/grafana-dashboards/LumenVox-KubeAdm-Grafana-Dashboard.json

Click on Home and then Dashboards
Click on + Create Dashboard select the Import Dashboard option and upload the LumenVox-KubeAdm-Grafana-Dashboard.json file you downloaded above. Select the Prometheus Datasource from the drop down menu and click Import.

You should now have a dashboard running that looks like this:

Was this article helpful?