Deploy Prometheus and Grafana for performance monitoring#

Also available in legacy Mattermost Enterprise Edition E20

Performance monitoring support enables admins to track system health for large Enterprise deployments through integrations with Prometheus and Grafana. These integrations support data collection from several Mattermost servers, which is particularly useful if you’re running Mattermost in high availability mode. Once you’re tracking system health, you can set up performance alerts on your Grafana dashboard.

Admins can collect and store various data points from the Mattermost application in an OpenMetrics format by deploying the Mattermost Metrics plugin, or by deploying Prometheus and Grafana.

Tip

Don’t want to deploy Prometheus and Grafana? You can also :doc: collect performance metrics using the Mattermost Metrics plugin </scale/collect-performance-metrics-using-metrics-plugin>.

Install Prometheus#

Important

While Prometheus and Grafana may be installed on the same server as Mattermost, we recommend installing these integrations on separate servers, and configure Prometheus to pull all metrics from Mattermost and other connected servers.

Download a precompiled binary for Prometheus. Binaries are provided for many popular distributions, including Darwin, Linux, and Windows. For installation instructions, see the Prometheus install guides.

The following settings are recommended in the Prometheus configuration file named prometheus.yml:

# my global config
global:
   scrape_interval:     60s # By default, scrape targets every 15 seconds.
   evaluation_interval: 60s # By default, scrape targets every 15 seconds.
   # scrape_timeout is set to the global default (10s).

   # Attach these labels to any time series or alerts when communicating with
   # external systems (federation, remote storage, Alertmanager).
   external_labels:
      monitor: 'mattermost-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   # - "first.rules"
   # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
   - job_name: 'prometheus'

   # Override the global default and scrape targets from this job every five seconds.
   # scrape_interval: 5s

   # metrics_path defaults to '/metrics'
   # scheme defaults to 'http'.

   static_configs:
      - targets: ["<hostname1>:<port>", "<hostname2>:<port>"]

Replace the <hostname1>:<port> parameter with your Mattermost host IP address and port to scrape the data. It connects to /metrics using HTTP.
In the Mattermost System Console, go to Environment > Performance Monitoring to set Enable Performance Monitoring to true, then specify the Listen Address and select Save. See our Configuration Settings documentation for details.
To test that the server is running, go to <ip>:<port>/metrics.

Note

A Mattermost Enterprise license is required to connect to /metrics using HTTP.

Finally, run vi prometheus.yml to finish configuring Prometheus. For starting the Prometheus service, read the comprehensive guides provided by Prometheus.
Once the service has started, you can access the data in <localhost>:<port>/graph. While you can use the Prometheus service to create graphs, we’ll focus on creating metric and analytics dashboards in Grafana.

Tip

For troubleshooting advice, check the Prometheus FAQ page.

Install Grafana#

Important

Download a precompiled binary for Grafana on Ubuntu or Debian. Binaries are also available for other distributions, including Redhat, Windows and Mac. For install instructions, see Grafana install guides
The Grafana package is installed as a service, so it is easy to start the server. See their install guides to learn more.
The default HTTP port is 3000 and default username and password are admin.
Add a Mattermost data source with the following settings as defined in the screenshot below

Tip

For troubleshooting advice, check the Grafana Troubleshooting page.
For user guides and tutorials, check the Grafana documentation to learn more.

Getting started#

To help you get started, you can download three sample dashboards shared in Grafana:

Mattermost Performance Monitoring v2, which contains detailed charts for performance monitoring including application, cluster, job server, and system metrics.
Mattermost Collapsed Reply Threads Metrics, which contains detailed metrics on the queries involved in our Collapsed Reply Threads feature.
Mattermost Performance KPI Metrics, which contains key metrics for monitoring performance and system health.
Mattermost Performance Monitoring (Bonus Metrics), which contains additional metrics such as emails sent or files uploaded, which may be important to monitor in some deployments.

See this Grafana guide to learn how to import Grafana dashboards either from the UI or from the HTTP API.

What’s collected?#

Mattermost provides custom metrics and standard Go metrics that can be used to monitor your system’s performance.