Prometheus Pushgateway on Cloud Foundry with Basic Authentication

Authentication methods are currently not supported in Prometheus, nor its related components like the Pushgateway.

However, this can be added via reverse proxies. Pushing it as an app to Cloud Foundry with neither the go-buildpack nor the binary-buildpack will give you such functionality. The staticfile-buildpack supports configuring basic authentication.

This article describes how to push the Pushgateway to Cloud Foundry using the staticfile-buildpack with basic authentication.

Instructions

Create Working Directories

First, let’s create a workspace directory for our project and change into it. Every file we create will be placed into that directory. The rest of the tutorial is using paths relative to that directory, so you may change it to your liking.

$ mkdir -p ~/workspace/apps/pushgateway $ cd ~/workspace/apps/pushgateway

Download and Extract Pushgateway Binary

We will use the binary release of the Pushgateway, so head over to the Github release page and download the respective Linux-amd64 archive of the version compatible with your Prometheus installation. For our purpose, we’ll be using the current version 1.0.0.

$ wget https://github.com/prometheus/pushgateway/releases/download/v1.0.0/pushgateway-1.0.0.linux-amd64.tar.gz $ wget https://github.com/prometheus/pushgateway/releases/download/v1.0.0/sha256sums.txt $ sha256sum -c <(grep pushgateway-1.0.0.linux-amd64.tar.gz sha256sums.txt) pushgateway-1.0.0.linux-amd64.tar.gz: OK

If the shasum check is OK, extract the binary and remove what is not needed anymore.

On Linux (GNU tar installed):

$ tar xzf pushgateway-1.0.0.linux-amd64.tar.gz --wildcards --strip-components=1 */pushgateway $ rm pushgateway-1.0.0.linux-amd64.tar.gz sha256sums.txt

For MacOS users (BSD tar installed):

$ tar xzf pushgateway-1.0.0.linux-amd64.tar.gz --strip-components=1 */pushgateway $ rm pushgateway-1.0.0.linux-amd64.tar.gz sha256sums.txt

Add Basic Authentication

The staticfile-buildpack is designed to serve files like HTML or CSS via an integrated NGINX server. Basic authentication is set using a special file Staticfile.auth that the buildpack will read at container start and setup NGINX accordingly.

The file is in the typical .htpasswd file format and can be created using the following command (I will use pushgateway as the username, but you may choose whatever suits you):

$ echo "pushgateway:$(openssl passwd -1)" > Staticfile.auth

Set the password and confirm it and you’re good to go.

Create an Application Manifest

Everything except for the manifest.yml and files/folders listed in a .cfignore will be integrated to /home/vcap/app in the container.

NGINX is started by default via app/boot.sh on container start. But since we also want to start the pushgateway process, this needs to be changed.

Create the following manifest.yml file your workspace folder:

--- name: pushgateway disk_quota: 256M memory: 256M command: exec /home/vcap/app/start.sh buildpack: staticfile_buildpack

This will change the command executed at container start to /home/vcap/app/start.sh, which has yet to be created by us.

Create the Start Script

All you need to do is to start the pushgateway in the background and continue with the default startup.

#!/bin/bash /home/vcap/app/pushgateway --web.listen-address 127.0.0.1:9091 & /home/vcap/app/boot.sh

However, this does not redirect requests to 127.0.0.1:9091 yet. Instead, it serves all files in the workspace directory to the world, which is not what we want.

Add a Custom Location

To change this, the buildpack offers a way to customize the location block. We have to set root and location_include in a file named Staticfile:

root: htdocs location_include: includes/*.conf

I have chosen htdocs to serve as the root location, but you can use whatever you like. But if you leave it like that, you will get a staging error because the root directory is empty.

-----> Root folder /tmp/app/htdocs **ERROR** Invalid root directory: the application Staticfile specifies a root directory htdocs that does not exist

All you need to do is add a dummy index.html file

mkdir -p htdocs echo "<h1>Nothing here</h1>" > htdocs/index.html

It doesn’t matter what content you add here, because it should actually never be displayed.

The location_include directive is relative to the /home/vcap/app/nginx/conf directory. NGINX will load any *.conf file there which we will use to configure the location. But as the workspace directory is added to /home/vcap/app, we need to create additional directories.

mkdir -p nginx/conf/includes

Then add the following content to nginx/conf/includes/pushgateway.conf with your favorite editor:

proxy_pass http://127.0.0.1:9091/; proxy_set_header	Host $host; proxy_set_header	X-Real-IP $remote_addr; proxy_set_header	X-Forwarded-for $remote_addr;

Adjust Scrape Configuration

Your scrape config must be adjusted to scrape with basic authentication (see Prometheus Configuration).

If you deploy Prometheus with prometheus-boshrelease, the internal pushgateway does not have basic-auth support at the moment and there are no ops files to scrape an external pushgateway. But you can add a custom ops file like the following to the deployment to make up for it.

- type: replace path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties?/prometheus/scrape_configs/- value: job_name: pushgateway static_configs: - targets: - ((pushgateway_host)) scheme: https basic_auth: username: ((pushgateway_basicauth.username)) password: ((pushgateway_basicauth.password))

If you use credhub to store your deployment secrets (recommended), you need to set the pushgateway_basicauth user credentials and pushgateway_host.

For example using the credhub CLI (replace the paths and values with respect to your environment):

$ credhub set -n /bosh/prometheus/pushgateway_host -t value -v pushgateway.app.domain.tld $ credhub set -n /bosh/prometheus/pushgateway_basicauth -t user --username=pushgateway --password="REPLACE_ME"

In case you already have an external pushgateway setup without authentication, you can deploy this change without breaking metrics scraping. This is how we transitioned to an external pushgateway with basic authentication smoothly. That is because a pushgateway without an authentication set will just ignore the additional credentials. In the next step, you can then push the basic-auth enabled pushgateway to Cloud Foundry.

The only thing that might break is if your configuration so far does not use HTTPS, but you could also just remove the scheme line from the ops file above to use HTTP (not recommended).

Push the App

The final step is just to push the pushgateway to Cloud Foundry by simply using

$ cf push

after you logged in and chose the org/space where you want it to be deployed.

Test

Finally, let’s test pushing metrics and reading them from the pushgateway.

Should fail:

$ metrics="# TYPE test_metric gauge test_metric{job=\"test\"} 0" $ curl -i --data-binary @<(echo "$metrics") https://pushgateway.app.domain.tld/metrics/job/test HTTP/1.1 401 Unauthorized [...] $ curl -i https://pushgateway.app.domain.tld/metrics HTTP/1.1 401 Unauthorized [...]

Should work:

$ metrics="# TYPE test_metric gauge test_metric{job=\"test\"} 0" $ curl -i --user "pushgateway:REPLACE_ME" --data-binary @<(echo "$metrics") https://pushgateway.app.domain.tld/metrics/job/test HTTP/1.1 202 Accepted [...] $ curl -i --user "pushgateway:REPLACE_ME" https://pushgateway.app.domain.tld/metrics HTTP/1.1 200 OK [...]

Final Thoughts

In this tutorial, we have seen how to use staticfile-buildpack to deploy the Prometheus Pushgateway to Cloud Foundry with basic authentication. It is a rather long procedure and there were some traps on the way because the staticfile-buildpack probably isn’t directly meant for this kind of usage, but once done even updates are rather simple by simply exchanging the Pushgateway binary.

One thing, however, is not covered here and should be improved upon, and that is health monitoring/restarting of the pushgateway process in case it crashes. But this is for another time.

Originally published at https://blog.anynines.com on October 27, 2020.