Skip to main content

Calls Offloader Setup and Configuration

This guide provides detailed instructions for setting up, configuring, and validating the Mattermost calls-offloader service used for call recording and transcription features.

Overview​

The calls-offloader service is a dedicated microservice that handles resource-intensive tasks for Mattermost Calls, including:

  • Call recording: Captures audio and screen sharing content from calls
  • Call transcription: Provides automated transcription of recorded calls
  • Live captions (Experimental): Real-time transcription during active calls

By offloading these tasks to a dedicated service, the main Mattermost server and RTCD service can focus on core functionality while maintaining optimal performance.

Prerequisites​

Before deploying calls-offloader, ensure you have:

  • A Mattermost Enterprise license
  • A properly configured Mattermost Calls deployment (either integrated or with RTCD)
  • A moderately powerful server with Docker installed and running
  • Sufficient storage space for recordings (see Storage Requirements)

System Requirements​

For detailed system requirements and performance recommendations, refer to the calls-offloader performance documentation.

Storage Requirements​

Call recordings can consume significant storage space. Based on average recording sizes with screen sharing on (including one audio track), storage usage by quality chosen is approximately:

  • Low: ~0.5GB/hour (or ~8MB/minute)
  • Medium: ~0.7GB/hour (or ~12MB/minute)
  • High: ~1.2GB/hour (or ~20MB/minute)

Note: Audio-only recordings consume approximately 1MB per minute per participant.

Installation and Deployment​

Bare Metal or VM Deployment​

  1. Download the latest release from the calls-offloader GitHub repository

  2. Create the necessary directories:

    sudo mkdir -p /opt/calls-offloader/data/db
    sudo useradd --system --home /opt/calls-offloader calls-offloader
    sudo chown -R calls-offloader:calls-offloader /opt/calls-offloader

  3. Create a configuration file (/opt/calls-offloader/config.toml):

    [api]
    http.listen_address = ":4545"
    http.tls.enable = false
    http.tls.cert_file = ""
    http.tls.cert_key = ""
    security.allow_self_registration = true
    security.enable_admin = false
    security.admin_secret_key = ""
    security.session_cache.expiration_minutes = 1440

    [store]
    data_source = "/opt/calls-offloader/data/db"

    [jobs]
    api_type = "docker"
    max_concurrent_jobs = 100
    failed_jobs_retention_time = "30d"
    image_registry = "mattermost"

    [logger]
    enable_console = true
    console_json = false
    console_level = "INFO"
    enable_file = true
    file_json = true
    file_level = "DEBUG"
    file_location = "/opt/calls-offloader/calls-offloader.log"
    enable_color = true

  4. Create a systemd service file (/etc/systemd/system/calls-offloader.service):

    [Unit]
    Description=Mattermost Calls Offloader Service
    After=network.target docker.service
    Requires=docker.service

    [Service]
    Type=simple
    User=calls-offloader
    WorkingDirectory=/opt/calls-offloader
    ExecStart=/opt/calls-offloader/calls-offloader --config /opt/calls-offloader/config.toml
    Restart=always
    RestartSec=10
    LimitNOFILE=65536

    [Install]
    WantedBy=multi-user.target

  5. Enable and start the service:

    sudo systemctl daemon-reload
    sudo systemctl enable calls-offloader
    sudo systemctl start calls-offloader

  6. Check the service status:

    sudo systemctl status calls-offloader

  7. Verify the service is responding:

    curl http://localhost:4545/version
    # Example output:
    # {"buildDate":"2025-03-10 19:13","buildVersion":"v0.9.2","buildHash":"a4bd418","goVersion":"go1.23.6"}

Configuration​

API Configuration​

The API section controls how the service accepts requests:

  • http.listen_address: The address and port where the service listens (default: :4545)
  • http.tls.enable: Whether to use TLS encryption for the API
  • security.allow_self_registration: Allow clients to self-register for job management
  • security.enable_admin: Enable admin functionality
  • security.admin_secret_key: Secret key for admin authentication (change from default!)

Store Configuration​

Controls persistent data storage:

  • data_source: Path to directory for storing client (i.e., connecting Mattermost nodes) IDs and credentials

Jobs Configuration​

Controls job processing behavior:

  • api_type: Job execution backend (docker or kubernetes)
  • max_concurrent_jobs: Maximum number of simultaneous recording/transcription jobs
  • failed_jobs_retention_time: How long to keep failed job data before cleanup
  • image_registry: Docker registry for job runner images (typically mattermost)

Logger Configuration​

Controls logging output:

  • enable_console: Log to console output
  • console_json: Use JSON format for console logs
  • console_level: Log level for console (DEBUG, INFO, WARN, ERROR)
  • enable_file: Log to file
  • file_location: Path to log file
  • enable_color: Use colored output for console logs

Private Network Configuration​

When the Mattermost deployment is running in a private network, additional configuration may be necessary for the jobs spawned by the calls-offloader service to reach the Mattermost server.

In such cases, you can override the site URL used by recorder jobs or transcriber jobs to connect to Mattermost by setting the following environment variables on the Mattermost server:

  • MM_CALLS_RECORDER_SITE_URL: Override the site URL used by recording jobs
  • MM_CALLS_TRANSCRIBER_SITE_URL: Override the site URL used by transcription jobs

Example configuration:

Create or edit the Mattermost environment file (/opt/mattermost/config/mattermost.environment):

MM_CALLS_RECORDER_SITE_URL="http://internal-mattermost-server:8065"
MM_CALLS_TRANSCRIBER_SITE_URL="http://internal-mattermost-server:8065"

Then ensure your Mattermost systemd service references this environment file:

[Unit]
Description=Mattermost
After=network.target

[Service]
Type=notify
EnvironmentFile=/opt/mattermost/config/mattermost.environment
ExecStart=/opt/mattermost/bin/mattermost
TimeoutStartSec=3600
KillMode=mixed
Restart=always
RestartSec=10
WorkingDirectory=/opt/mattermost
User=mattermost
Group=mattermost

[Install]
WantedBy=multi-user.target

This is particularly useful when:

  • The calls-offloader service runs in a different network segment than clients
  • Internal DNS resolution differs from external URLs
  • You need to use internal load balancer endpoints for job communication

Validation and Testing​

After deploying calls-offloader, validate the installation:

  1. Check service status:

    # For systemd
    sudo systemctl status calls-offloader

  2. Test API connectivity:

    From the calls-offloader server (localhost test):

    curl http://localhost:4545/version
    # Should return version information
    # Example: {"buildDate":"2025-03-10 19:13","buildVersion":"v0.9.2","buildHash":"a4bd418","goVersion":"go1.23.6"}

    From the Mattermost server:

    curl http://YOUR_CALLS_OFFLOADER_SERVER:4545/version
    # Should return the same version information
    # This confirms network connectivity from Mattermost to calls-offloader

    If the localhost test works but the Mattermost server test fails, check:

    • Firewall rules or SELinux policies on the calls-offloader server (port 4545 must be accessible)
    • Network connectivity between Mattermost and calls-offloader servers

  3. Verify Docker service (if using docker api_type):

    # Check that system user running calls-offloader can access Docker
    sudo -u calls-offloader docker ps

Integration with Mattermost​

Once calls-offloader is properly set up and validated, configure Mattermost to use it:

  1. Go to System Console > Plugins > Calls

  2. In the Job Service section:

    • Set Job Service URL to your calls-offloader service (e.g., http://calls-offloader-server:4545)

  3. Enable recording and transcription features as needed:

    • Enable Call Recordings: Toggle to allow call recordings
    • Enable Call Transcriptions: Toggle to allow call transcriptions
    • Enable Live Captions (Experimental): Toggle to allow real-time transcription

  4. Save the configuration

  5. Restart the Calls plugin to re-establish state:

    • Go to System Console > Plugins > Plugin Management
    • Find the Calls plugin and click Disable
    • Wait a few seconds, then click Enable

  6. Test by starting a call and starting a recording

Troubleshooting​

Common Issues​

"failed to create recording job: max concurrent jobs reached"

This error occurs when the calls-offloader service has reached its configured job limit, and it will usually result in a failure message on the Mattermost Calls plugin side (such as a timeout).

Solutions:

  • Increase max_concurrent_jobs in the configuration
  • Check if jobs are hanging and restart the service
  • Monitor system resources and scale up if needed

Jobs not processing

Check the following:

  • Verify the calls-offloader service is running: sudo systemctl status calls-offloader
  • Ensure network connectivity between Mattermost and calls-offloader
  • Check Docker daemon is running and accessible by the user running the Calls Offloader service (E.g., user: calls-offloader)
  • Verify authentication configuration matches between services
  • Review service logs for specific error messages

Docker permission issues

If using Docker API and seeing permission errors:

# Add calls-offloader user to docker group
sudo usermod -a -G docker calls-offloader
sudo systemctl restart calls-offloader

Debugging Commands​

Monitor calls-offloader job containers:

# View running job containers
docker ps --format "{{.ID}} {{.Image}}" | grep "calls"

# Follow logs for debugging
docker ps --format "{{.ID}} {{.Image}}" | grep "calls" | awk '{print $1}' | xargs -I {} docker logs -f {}

# View completed job containers
docker ps -a --filter "status=exited"

Monitor service health:

# Check service version and health
curl http://localhost:4545/version

Check service logs:

# View recent logs
sudo journalctl -u calls-offloader -f

# View log file (if file logging enabled)
tail -f /opt/calls-offloader/calls-offloader.log

Performance Monitoring​

Monitor calls-offloader performance and resource usage to ensure optimal operation. See Calls Metrics and Monitoring for details on setting up metrics and observability.

Air-Gapped Installation of calls-offloader​

This guide covers deploying calls-offloader in an environment without internet access. The process uses scripts from the calls-install-scripts repository and follows a two-phase workflow: preparing a transfer bundle on an internet-connected machine, then deploying it on the isolated target machine.

Overview​

Because calls-offloader relies on Docker images for the recorder and transcriber jobs, an air-gapped deployment requires that those images be pre-pulled and packaged alongside the calls-offloader binary before being transferred to the target environment.

The install scripts handle this in two stages:

  1. Prepare (internet-connected machine) — setup-airgap-offloader.sh pulls the required Docker images and downloads the calls-offloader binary, then packages everything into a transfer bundle and generates a ready-to-run deployment script.
  2. Deploy (air-gapped machine) — Transfer the bundle and run the generated deploy-airgap-offloader.sh script, which loads the Docker images into a local registry and installs the service.

Prerequisites​

  • An internet-connected Linux machine with Docker installed (for the preparation phase)
  • A target air-gapped Linux machine with:
    • systemd
    • Docker installed
    • Root or sudo access
  • The calls-install-scripts repository cloned on the internet-connected machine

Phase 1: Prepare the Transfer Bundle​

On the internet-connected machine, run setup-airgap-offloader.sh specifying the versions of each component to package:

./setup-airgap-offloader.sh \
  --offloader v0.9.5 \
  --recorder v0.9.0 \
  --transcriber v0.3.0 \
  --arch amd64
FlagDescriptionDefault
--offloader VERSIONcalls-offloader binary version (e.g. v0.9.5)required
--recorder VERSIONcalls-recorder Docker image versionrequired
--transcriber VERSIONcalls-transcriber Docker image versionrequired
--arch amd64|arm64Target CPU architectureamd64

The script will:

  1. Pull the calls-recorder and calls-transcriber Docker images from Docker Hub
  2. Download the calls-offloader binary from GitHub releases
  3. Save the Docker images as .tar archives
  4. Generate a deploy-airgap-offloader.sh deployment script configured for the selected versions
  5. Produce a transfer bundle containing all of the above

Phase 2: Transfer to the Air-Gapped Machine​

Copy the generated bundle to the target machine using whatever transfer mechanism is available in your environment (USB drive, secure file transfer, etc.):

scp calls-offloader-airgap-bundle.tar.gz user@airgap-host:/tmp/

On the air-gapped machine, extract the bundle:

tar -xzf calls-offloader-airgap-bundle.tar.gz
cd calls-offloader-airgap-bundle/

Phase 3: Deploy on the Air-Gapped Machine​

Run the generated deployment script with root or sudo privileges:

sudo ./deploy-airgap-offloader.sh

This script will:

  1. Configure the local Docker daemon to use a local image registry
  2. Load the packaged Docker images into that registry
  3. Install the calls-offloader binary to /usr/local/bin/
  4. Create the mattermost system user and add it to the docker group
  5. Generate and enable a systemd service unit
  6. Start the service and verify it is running

Once complete, verify the service is up:

curl http://localhost:4545/version

Connecting to Mattermost​

Configure the Mattermost Calls plugin to use the offloader service via System Console > Plugins > Calls > Job service URL, setting it to http://<offloader-host>:4545.

[!NOTE] The first time Mattermost connects to the offloader it will self-register and store its authentication key in the database, provided API_SECURITY_ALLOWSELFREGISTRATION=true is set (the default in the deployment script).

Private Network Considerations​

In air-gapped environments the recorder and transcriber containers typically need to reach the Mattermost server via an internal URL. Set the following environment variables on the Mattermost server to override the site URL used by spawned jobs:

MM_CALLS_RECORDER_SITE_URL=http://internal-mattermost-server:8065
MM_CALLS_TRANSCRIBER_SITE_URL=http://internal-mattermost-server:8065

You may also need to add the internal URL to ServiceSettings.AllowCorsFrom in the Mattermost server configuration.

[!NOTE] In particularly restrictive environments (e.g., VMs with strict network isolation), set DOCKER_NETWORK=host in the calls-offloader service environment so that job containers can reach the Mattermost server via its local address.

Custom Docker Registry​

If your air-gapped environment already has an internal Docker registry, you can point install-offloader.sh at it directly instead of using the local registry set up by the deployment script:

sudo ./install-offloader.sh \
  --binary ./calls-offloader-linux-amd64 \
  --image-registry registry.internal.example.com/mattermost \
  --arch amd64

The --image-registry flag sets the registry prefix used when the offloader pulls recorder and transcriber images for each job.

Other Calls Documentation​