Bulk Loading Data

Large quantities of data can be imported from a JSONL file into Mattermost at the command line using the bulk loading feature. This feature is most suitable for migrating data from an existing messaging system, or for pre-populating a new installation with data.

You can import the following data types:

  • Teams
  • Channels (Public & Private)
  • Users
  • Users’ Team memberships
  • Users’ Channel memberships
  • Users’ notification preferences
  • Posts (regular, non-reply posts)
  • Posts’ Replies
  • Posts’ Reactions
  • Direct Message and Group Message channels
  • Direct Messages and Group Messages

Importing additional types of posts is not yet supported.

About the bulk loading command

The bulk loading command is interruptible and idempotent
If the import is interrupted for any reason, it continues from where it left off the next time you run it. You can run the command repeatedly with the same data file, and the data is imported only once.
You can run the bulk loading command on a live system
Although you don’t need to shut down Mattermost to run the command, changes made by users of the system between runs can be overwritten if the corresponding fields exist in the data file.
Some data fields are optional
Not all fields are mandatory. If an optional field is missing from the object that is being imported, the field’s current value in the database is not changed.
The bulk loading command is not a synchronization tool
You cannot use the bulk loading command to remove any objects or their fields from the Mattermost database. The command only creates or overwrites fields.

Important

The bulk loading command runs in the CLI and operates in the security context of the CLI. This means it has full permissions to access and alter everything in the Mattermost database.

Running the bulk loading command

Before running the bulk loading command, you must first create a JSONL file that contains the data that you want to import. After you create the file, run the bulk load command in validation mode. In this mode, the data is checked for correctness, but is not written to the database. After validating, run the command in apply mode, which saves the data to the database.

To bulk load data:

1. Create the JSONL data file in your Mattermost bin directory. The file can have any name, but in this procedure it’s called data.jsonl. The format of the file is described in the Data Format section. 2. Validate that the file is correct:

  1. Change to the Mattermost bin directory.
cd /opt/mattermost/bin (the location might be different on your system)
  1. Run the following command:
sudo ./platform import bulk data.jsonl --validate
  1. Resolve any errors that are reported, and validate the file again. Do not go to the next step until you can run the validate command without errors.
  2. Run the bulk load command in apply mode:
sudo ./platform import bulk data.jsonl --apply
  1. When the bulk load command completes, clear all caches. Open the System Console, and click General > Configuration > Purge All Caches.

Data Format

The input data file must be a valid JSONL file with the following objects, each on its own line in the file. The objects must occur in the file in the order listed.

Version
Mandatory. The Version object must be the first line in the file, and must occur only once.
Team
Optional. If present, Team objects must occur after the Version object and before any Channel objects.
Channel
Optional. If present, Channel objects must occur after all Team objects and before any User objects.
User
Optional. If present, User objects must occur after the Team and Channel objects in the file and before any Post objects. Each User object defines the teams and channels that the user is a member of. If the corresponding teams and channels are not in the data file, then they must exist in the Mattermost database.
Post
Optional. If present, Post objects must occur after the last User object but before any DirectChannel objects. Each Post object defines the team, the channel, and the username of the user who posted the message. If the corresponding team, channel, or user are not in the data file, then they must exist in the Mattermost database.
DirectChannel
Optional. If present, DirectChannel objects must occur after all Post objects in the file and before any DirectPost objects.
DirectPost
Optional. If present, DirectPost objects must occur after all other objects in the file. Each DirectPost object defines the usernames of the channel members and the username of the user who posted the message. If the corresponding usernames are not in the data file, then they must exist in the Mattermost database.

With the exception of the Version object, each object has a field or a combination of fields that is used as the unique identifier of that object. The bulk loader uses the unique identifier to determine if the object being imported is a new object or an update to an existing object.

The identifiers for each object are listed in the following table:

Objects and their unique identifiers
Object Unique Identifier
Version Not Applicable
Team name
Channel name, team
User username
UserNotifyProps username
UserTeamMembership team, username
UserChannelMembership team, channel, username
Post channel, message, create_at
Reply post, message, create_at
Reaction post, emoji_name, create_at
DirectChannel members
DirectPost channel_members, user, message, create_at

The following fragment is from a file that imports two teams, each with two channels, many users, and many posts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
{ "type": "version", ... }
{ "type": "team", "team": { "name": "TeamA", ...} }
{ "type": "team", "team": { "name": "TeamB", ...} }
{ "type": "channel", "channel": { "team": "TeamA", "name": "ChannelA1", ...} }
{ "type": "channel", "channel": { "team": "TeamA", "name": "ChannelA2", ...} }
{ "type": "channel", "channel": { "team": "TeamB", "name": "ChannelB1", ...} }
{ "type": "channel", "channel": { "team": "TeamB", "name": "ChannelB1", ...} }
{ "type": "user", "user": { "username": "user001", ...} }
{ "type": "user", "user": { "username": "user002", ...} }
{ "type": "user", "user": { "username": "user003", ...} }
{ "type": "user", ... }
{ "type": "user", ... }
{ "type": "user", ... }
.
.
.
{ "type": "post", { "team": "TeamA", "name": "ChannelA1", "user": "user001", ...} }
{ "type": "post", { "team": "TeamA", "name": "ChannelA1", "user": "user001", ...} }
{ "type": "post", { "team": "TeamA", "name": "ChannelA1", "user": "user001", ...} }
.
.
.

Version object

The Version object must be the first object in the data file, and can appear only once.

Example Version object

For clarity, the object is shown using regular JSON formatting, but in the data file it cannot be spread across several lines. It must be all on one line.

{
  "type": "version",
  "version": 1
}

Fields of the Version object

Field name Type Description Validated Mandatory
type string Must be the string "version" Yes Yes
version number Must be the number 1. Yes Yes

Team object

If present, Team objects must occur after the Version object and before any Channel objects.

Example Team object

For clarity, the object is shown using regular JSON formatting, but in the data file it cannot be spread across several lines. It must be all on one line.

{
"type": "team",
"team": {
  "name": "team-name",
  "display_name": "Team Display Name",
  "type": "O",
  "description": "The Team Description",
  "allow_open_invite": true
  }
}

Fields of the Team object

Field name Type Description Validated Mandatory
name string The team name. Yes Yes
display_name string The display name for the team. Yes Yes
type string The type of team. Can have one the following values:
"O" for an open team
"I" for an invite-only team.
Yes Yes
description string The team description. Yes No
allow_open_invite bool Whether to allow open invitations. Must have one of the following values:
true
false
Yes No

Channel object

If present, Channel objects must occur after all Team objects and before any User objects.

Example Channel object

For clarity, the object is shown using regular JSON formatting, but in the data file it cannot be spread across several lines. It must be all on one line.

{
  "type": "channel",
  "channel": {
    "team": "team-name",
    "name": "channel-name",
    "display_name": "Channel Name",
    "type": "O",
    "header": "The Channel Header",
    "purpose": "The Channel Purpose",
  }
}

Fields of the Channel object

[1] Not validated, but an error occurs if no such team exists when running in apply mode.
Field name Type Description Validated Mandatory
team string The name of the team this channel belongs to. No [1] Yes
name string The name of the channel. Yes Yes
display_name string The display name for the channel. Yes yes
type string The type of channel. Can have one the following values:
"O" for a public channel.
"P" for a private channel.
Yes Yes
header string The channel header. Yes No
purpose string The channel purpose. Yes No

User object

If present, User objects must occur after the Team and Channel objects in the file and before any Post objects.

Example User object

For clarity, the object is shown using regular JSON formatting, but in the data file it cannot be spread across several lines. It must be all on one line.

{
  "type": "user",
  "user": {
    "profile_image": "avatar.png",
    "username": "username",
    "email": "email@example.com",
    "auth_service": "",
    "auth_data": "",
    "password": "passw0rd",
    "nickname": "bobuser",
    "first_name": "Bob",
    "last_name": "User",
    "position": "Senior Developer",
    "roles": "system_user",
    "locale": "pt_BR",
    "teams": [
      {
        "name": "team-name",
        "roles": "team_user team_admin",
        "channels": [
          {
            "name": "channel-name",
            "roles": "channel_user",
            "notify_props": {
              "desktop": "default",
              "mark_unread": "all"
            }
          }
        ]
      }
    ]
  }
}

Fields of the User object

Field name Type Description Validated Mandatory
profile_image string The user’s profile image. This must be an existing file path. Yes No
username string The user’s username. This is the unique identifier for the user. Yes Yes
email string The user’s email address. Yes Yes
auth_service string The authentication service to use for this user account. If not provided, it defaults to password-based authentication. Must be one of the following values:
"" or not provided - password authentication.
"gitlab" - GitLab authentication.
"ldap" - LDAP authentication (E10 and E20)
"saml" - Generic SAML based authentication (E20)
"google" - Google OAuth authentication (E20)
"office365" - Microsoft Office 365 OAuth Authentication (E20)
No No
auth_data string The authentication data if auth_service is used. The value depends on the auth_service that is specified.
The data comes from the following fields for the respective auth_services:
"" or not provided - must be omitted.
"gitlab" - The value of the Id attribute provided in the Gitlab auth data.
"ldap" - The value of the LDAP attribute specified as the "ID Attribute" in the Mattermost LDAP configuration.
"saml" - The value of the SAML Email address attribute.
"google" - The value of the OAuth Id attribute.
"office365" - The value of the OAuth Id attribute.
No No
password string A password for the user. Can be present only when password-based authentication is used. When password-based authentication is used and the password is not present, the bulk loader generates a password. Yes No
nickname string The user’s nickname. Yes No
first_name string The user’s first name. Yes No
last_name string The user’s last name. Yes No
position string The user’s position. Yes No
roles string The user’s roles. Must be one of the following values:
"system_user"
"system_admin system_user"
Yes No
locale string The user’s locale. This must be a valid locale for which Mattermost has been localised. No No
teams array The teams which the user will be made a member of. Must be an array of TeamMembership objects. Yes No
theme string The user’s theme. Formatted as a Mattermost theme string. No No
use_military_time string How times should be displayed to this user. Must be one of the following values:
"true" - Use 24 hour clock.
"false" - Use 12 hour clock.
No No
collapse_previews string Whether to collapse or expand link previews by default. Must be one of the following values:
"true" - Collapsed by default.
"false" - Expanded by default.
No No
message_display string Which style to use for displayed messages. Must be one of the following values:
"clean" - Use the standard style.
"compact" - Use the compact style.
No No
channel_display_mode string How to display channel messages. Must be one of the following values:
"full" - Use the full width of the screen.
"centered" - Use a fixed width, centered block.
No No
tutorial_step string Where to start the user tutorial. Must be one of the following values:
"1", "2" or "3" - Start from the specified tutorial step.
"999" - Skip the user tutorial.
No No

Fields of the UserNotifyProps object

This object is a member of the User object.

[1] Not validated, but an error occurs if no such team exists when running in apply mode.
Field name Type Description Validated Mandatory
desktop string Preference for sending desktop notifications. Must be one of the following values:
"all" - For all activity.
"mention" - Only for mentions.
"none" - Never.
Yes No
desktop_duration string Preference for how long desktop notifications remain on screen. Must be one of the following values:
"3" - 3 seconds.
"5" - 5 seconds.
"10" - 10 seconds.
"0" - Unlimited.
No No
desktop_sound string Preference for whether desktop notification sound is played. Must be one of the following values:
"true" - Sound is played.
"false" - Sound is not played.
Yes No
email string Preference for email notifications. Must be one of the following values:
"true" - Email notifications are sent immediately.
"false" - Email notifications are not sent.
No No
mobile string Preference for sending mobile push notifications. Must be one of the following values:
"all" - For all activity.
"mention" - Only for mentions.
"none" - Never.
Yes No
mobile_push_status string Preference for when push notifications are triggered. Must be one of the following values:
"online" - When online, away or offline.
"away" - When away or offline.
"offline" - When offline.
Yes No
channel string Whether @all, @channel and @here trigger mentions. Must be one of the following values:
"true" - Mentions are triggered.
"false" - Mentions are not triggered.
Yes No
comments string Preference for reply mention notifications. Must be one of the following values:
"any" - Trigger notifications on messages in reply threads that the user starts or participates in.
"root" - Trigger notifications on messages in threads that the user starts.
"never" - Do not trigger notifications on messages in reply threads unless the user is mentioned.
Yes No
mention_keys string Preference for custom non-case sensitive words that trigger mentions. Words must be separated by commas. No No

Fields of the UserTeamMembership object

This object is a member of the User object.

[1] Not validated, but an error occurs if no such team exists when running in apply mode.
Field name Type Description Validated Mandatory
name string The name of the team this user should be a member of. No [1] Yes
roles string The roles the user should have within this team. Must be one of the following values:
"team_user"
"team_admin team_user"
Yes No
channels array The channels within this team that the user should be made a member of. Must be an array of ChannelMembership objects. Yes No

Fields of the UserChannelMembership object

This object is a member of the TeamMembership object.

[1] Not validated, but an error occurs if the parent channel does not exist when running in apply mode.
Field name Type Description Validated Mandatory
name string The name of the channel in the parent team that this user should be a member of. No [1] Yes
roles string The roles the user should have within this channel. Must be one of the following values:
"channel_user"
"channel_user channel_admin"
Yes No
notify_props object The notify preferences for this user in this channel. Must be a ChannelNotifyProps object Yes No
favorite boolean Whether to favorite the channel. Must be one of the following values:
"true" - Yes.
"false" - No.
No No

Fields of the ChannelNotifyProps object

This object is a member of the ChannelMembership object.

Field name Type Description Validated Mandatory
desktop string Preference for sending desktop notifications. Must be one of the following values:
"default" - Global default.
"all" - For all activity.
"mention" - Only for mentions.
"none" - Never.
Yes No
mark_unread string Preference for marking channel as unread. Must be one of the following values:
"all" - For all unread messages.
"mention" - Only for mentions.
Yes No

Post object

If present, Post objects must occur after the last User object in the file, but before any DirectChannel objects.

Example Post object

For clarity, the object is shown using regular JSON formatting, but in the data file it cannot be spread across several lines. It must be all on one line.

{
  "type": "post",
  "post": {
    "team": "team-name",
    "channel": "channel-name",
    "user": "username",
    "message": "The post message",
    "create_at": 140012340013,
    "flagged_by": [
      "username1",
      "username2",
      "username3"
    ],
    "replies": [{
      "user": "username4",
      "message": "The reply message",
      "create_at": 140012352049,
    }, {
      "user": "username5",
      "message": "Other reply message",
      "create_at": 140012353057,
    }],
    "reactions": [{
      "user": "username6",
      "emoji_name": "+1",
      "create_at": 140012356032,
    }, {
      "user": "username7",
      "emoji_name": "heart",
      "create_at": 140012359034,
    }]
  }
}

Fields of the Post object

[1] Not validated, but an error occurs if the team does not exist when running in apply mode.
[2] Not validated, but an error occurs if the channel does not exist in the corresponding team when running in apply mode.
[3] Not validated, but an error occurs if the user does not exist when running in apply mode.
Field name Type Description Validated Mandatory
team string The name of the team that this post is in. No [1] Yes
channel string The name of the channel that this post is in. No [2] Yes
user string The username of the user for this post. No [3] Yes
message string The message that the post contains. Yes Yes
create_at int The timestamp for the post, in milliseconds since the Unix epoch. Yes Yes
flagged_by array Must contain a list of members who have flagged the post. No No
replies array The posts in reply to this post. Must be an array of Reply objects. Yes No
reactions array The emoji reactions to this post. Must be an array of Reaction objects. Yes No

Fields of the Reply object

This object is a member of the Post/DirectPost object.

Field name Type Description Validated Mandatory
user string The username of the user for this reply. No [3] Yes
message string The message that the reply contains. Yes Yes
create_at int The timestamp for the reply, in milliseconds since the Unix epoch. Yes Yes

Fields of the Reaction object

This object is a member of the Post/DirectPost object.

Field name Type Description Validated Mandatory
user string The username of the user for this reply. No [3] Yes
emoji_name string The emoji of the reaction. Yes Yes
create_at int The timestamp for the reply, in milliseconds since the Unix epoch. Yes Yes

DirectChannel object

A direct channel can have from two to eight users as members of the channel. If there are only two members, Mattermost treats it as a Direct Message channel. If there are three or more members, Mattermost treats it as a Group Message channel.

Example DirectChannel object

For clarity, the object is shown using regular JSON formatting, but in the data file it cannot be spread across several lines. It must be all on one line.

{
  "type": "direct_channel",
  "direct_channel": {
    "members": [
      "username1",
      "username2",
      "username3"
    ],
    "header": "The Channel Header",
    "favorited_by": [
      "username1",
      "username2",
      "username3"
    ]
  }
}

Fields of the DirectChannel object

[1] Not validated, but an error occurs if one or more of the users don't exist when running in apply mode.
Field name Type Description Validated Mandatory
members array Must contain a list of members, with a minimum of two usernames and a maximum of eight usernames. No [1] Yes
header string The channel header. Yes No
favorited_by array Must contain a list of members who have favorited the channel. No No

DirectPost object

DirectPost objects must occur after all other objects in the file.

Example DirectPost object

For clarity, the object is shown using regular JSON formatting, but in the data file it cannot be spread across several lines. It must be all on one line.

{
  "type": "direct_post",
  "direct_post": {
    "channel_members": [
      "username1",
      "username2",
      "username3",
    ],
    "user": "username2",
    "message": "Hello Group Channel",
    "create_at": 140012340013,
    "flagged_by": [
      "username1",
      "username2",
      "username3"
    ],
    "replies": [{
      "user": "username4",
      "message": "The reply message",
      "create_at": 140012352049,
    }, {
      "user": "username5",
      "message": "Other reply message",
      "create_at": 140012353057,
    }],
    "reactions": [{
      "user": "username6",
      "emoji_name": "+1",
      "create_at": 140012356032,
    }, {
      "user": "username7",
      "emoji_name": "heart",
      "create_at": 140012359034,
    }]
  }
}

Fields of the DirectPost object

[1] Not validated, but an error occurs if no channels contain an identical list when running in apply mode.
[2] Not validated, but an error occurs if the user does not exist when running in apply mode.
Field name Type Description Validated Mandatory
channel_members array Must contain a list of members, with a minimum of two usernames and a maximum of eight usernames. No [1] Yes
user string The username of the user for this post. No [2] Yes
message string The message that the post contains. Yes Yes
create_at int The timestamp for the post, in milliseconds since the Unix epoch. Yes Yes
flagged_by array Must contain a list of members who have flagged the post. No No
replies array The posts in reply to this direct post. Must be an array of Reply objects. Yes No
reactions array The emoji reactions to this direct post. Must be an array of Reaction objects. Yes No