Prometheus Operator logo Prometheus Operator

Prometheus Operator
for Kubernetes

by CoreOS

The Prometheus operator provisions and configures the popular Prometheus monitoring system and time series database on top of Kubernetes. It allows setting both Prometheus configuration like versions, persistence, retention policies as well as monitoring targets through Kubernetes custom resources and manages the lifecycle of the Prometheus instances.

You can use the operator to provision Prometheus instances to monitor any workload that exposes metrics in the Prometheus format. Additionally the operator can also provision AlertManager for alerting.

Documentation

description README

Install, update and remove

description DEPLOY

Deploy Prometheus

Install the Prometheus Operator

info_outline

Below instructions explain how to install the Prometheus operator in a repository bootstrapped using the Kubestack quickstart. But you can adapt the workflow to any kustomize directory layout easily.

  1. Vendor the base

    # Run these commands from the root of your Kubestack infra repository
    wget https://storage.googleapis.com/catalog.kubestack.com/prometheus-v0.31.1-kbst.0.zip
    unzip -d manifests/bases/ prometheus-v0.31.1-kbst.0.zip
    rm prometheus-v0.31.1-kbst.0.zip
    
  2. Include in bases

    cd manifests/overlays/common
    kustomize edit add base ../../bases/prometheus/clusterwide
    
  3. Commit and push

    cd -
    git checkout -b add-prometheus
    git add manifests/bases/prometheus manifests/overlays/common/kustomization.yaml
    git commit -m "Add prometheus v0.31.1-kbst.0 base"
    git push origin add-prometheus
    
  4. Review PR and merge

    Finally, review and merge the PR into master. Once it's been successfully applied against the Ops-Cluster set a prod-deploy tag to also apply the change against the Apps-Cluster.


Update the Prometheus Operator

info_outline

The UPDATE instructions assume the operator has been installed by following the INSTALL instructions. If not, adjust accordingly.

To update the operator delete the previously vendored base and then vendor the new version.

  1. Delete the previous vendored version

    # Run these commands from the root of your Kubestack infra repository
    rm -r manifests/bases/prometheus
    
  2. Vendor the new version

    # Run these commands from the root of your Kubestack infra repository
    wget https://storage.googleapis.com/catalog.kubestack.com/prometheus-v0.31.1-kbst.0.zip
    unzip -d manifests/bases/ prometheus-v0.31.1-kbst.0.zip
    rm prometheus-v0.31.1-kbst.0.zip
    
  3. Commit and push

    git checkout -b update-prometheus
    git add manifests/bases/prometheus
    git commit -m "Update prometheus base to v0.31.1-kbst.0"
    git push origin update-prometheus
    

Remove the Database Clusters and the Operator

info_outline

Please be advised, the removal instructions for the Prometheus operator require manual steps to cleanly remove both provisioned Prometheus instances and the operator itself from your Kubernetes cluster.

  1. List and deprovision Prometheus instances

    # Make sure your context is pointing to the correct cluster
    kubectl get prometheus --all-namespaces
    

    Make sure you want to delete the instances listed above, then delete them using kubectl -n NAMESPACE delete prometheus NAME.

  2. Delete the vendored base from your repository

    # Run these commands from the root of your Kubestack infra repository
    rm -r manifests/bases/prometheus
    
  3. Commit and push

    git checkout -b remove-prometheus
    git add manifests/bases/prometheus
    git commit -m "Remove prometheus base"
    git push origin remove-prometheus
    
  4. Delete operator

    When manifests are applied using kubectl apply, resources that have been removed from the manifests, are not removed from the clusters. So, as a last step, we need to delete the operator from our clusters, which we can do by simply deleting the namespace.

    kubectl delete namespace operator-prometheus
    

Deploy Prometheus

Using the Prometheus operator to provision a Prometheus instance and start monitoring an application requires three steps:

  1. Deploy the operator
  2. Provision a Prometheus instance
  3. Create ServiceMonitors for each service exposing metrics

The first step was to deploy the Prometheus operator following the instructions on the install tab.

Prometheus Instance Manifests

Next, we can continue with step number two and use the operator to provision our Prometheus instance.

Each Prometheus instance needs read-only access to the Kubernetes api in order to keep its monitoring targets up to date. To make this easier, the clusterwide base includes two ClusterRoles. One for the operator, called prometheus-operator, and one for the instances, aptly named prometheus-instance.

So, in order to provision a Prometheus instance we need to first create a ServiceAccount and a RoleBinding linking that service account to the pormetheus-instance cluster role. Finally, we can create a Prometheus resource that instructs the operator to provision a Prometheus instance that uses our service account. Below example does just that.

---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: example-instance
  namespace: default
  labels:
    prometheus: example-instance
spec:
  serviceAccountName: prometheus-example-instance
  serviceMonitorSelector:
    matchLabels:
      prometheus-instance: example-instance
  resources:
    requests:
      # by default the operator requests 2Gi of memory
      # adapt the line below if required to schedule pods
      memory: 2Gi

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: prometheus-example-instance
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-instance
subjects:
- kind: ServiceAccount
  name: prometheus-example-instance
  namespace: default

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-example-instance
  namespace: default

ServiceMonitors Manifest

Finally, we need to create a ServiceMonitor to instruct the operator, to configure our Prometheus instance to scrape metrics from each of the replicas of an application that exposes its metrics in the Prometheus format.

Since this step is depending on the application and where it exposes the metrics, you will have to adapt the example below.

But there are basically three things to note. First, the metadata.labels need to match the spec.serviceMonitorSelector.matchLabels from the Prometheus resource. Next, the spec.selector.matchLabels below needs to match the metadata.labels set for the service of the application you are trying to monitor. Finally, the spec.endpoints need to match the port name, configured in the application's service.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: application-example
  labels:
    # this label instructs Prometheus to include this ServiceMonitor
    # based on the `spec.serviceMonitorSelector.matchLabels` above
    prometheus-instance: example-instance
spec:
  selector:
    matchLabels:
      # this selector is how the `ServiceMonitor` finds the
      # application's service
      app: application-example
  endpoints:
  # this tells Prometheus on what port the app exposes its metrics
  - port: web

Prometheus UI

To check if Promethues has started scraping your applicaiton's metrics, you can access the Prometheus UI using kubectl port-forward and use it to run a query.

kubectl port-forward prometheus-example-instance-0 9090

Advanced Configuration

Above instructions only cover the basics. The Prometheus operator has many more features and options. For more extended documentation about Prometheus operator configuration, please refer to the official documentation.