Feature Release Process

Mattermost core team works on a monthly release process, with a new version shipping on the 16th of each month in binary form.

This document outlines the development process for the Mattermost core team, which draws from what we find works best for us from Agile, Scrum and Software Development Lifecycle approaches.

Recommended pre-reading:

Release Timeline

Notes:

  • All cut-off dates are based on 10am (San Francisco Time) on the day stated.
  • T-minus counts are measured in “working days” (weekdays other than major holidays concurrent in US and Canada) prior to release day.

A. (Code complete date of previous release) Beginning of release

Pre-work for the current release begins at the code complete date of the previous release. See “Code Complete” section below for details.

B. (T-minus 18 working days) Cut-off for submitting features

No pull requests for major features should be submitted to the current release after this date. In special cases, exceptions can be made by the Release Manager.

  1. Release Manager:
    • Post this checklist in Release Checklist channel
    • Follow that major feature PR reviews are prioritized and post a list of outstanding feature PRs in the Release Discussion channel
    • Check that major features for platform have a corresponding ticket for RN, and vice versa, where applicable
    • Check that all major features are behind a feature flag
    • Confirm with PMs that each Enterprise feature is in the correct pricing SKU
    • Review any features that are currently in beta and confirm with PMs if there are any to be promoted
    • Confirm all config settings and new features have diagnostics

C. (T-minus 15 working days) Cut-off for merging features

No pull requests for major features should be merged to the current release after this date. In special cases, exceptions can be made by the Release Manager.

  1. (Team) Major Feature Complete Meeting (10:00am San Francisco time):
    • Discuss worst bug currently on master
    • Discuss release highlights for marketing
    • Review status of remaining feature PRs to be merged
  2. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete
    • Queue a list of MVP candidates in alphabetical order to the Platform Meeting channel. See example
    • Draft Changelog in a WIP PR with updates for highlights, feature additions, known issues, compatibility updates for deprecated features, config.json, database changes, API changes, and WebSocket event changes; see example
    • Review software requirements are up-to-date based on these guidelines. If not, update documentation accordingly, and note changes in the Changelog.
    • Update Upgrade Guide with any special notes for upgrading to the new version
    • Ask PMs if there are any notable breaking changes or deprecated features in the release
    • Review submitted NOTICE.txt PRs for any new libraries added from dev, and ensure they are cherry-picked to feature release branch.
    • Start posting a daily Zero Bug Balance query (posted until zero bugs or day of release)
    • Post a reminder in the French Localization channel about the due date for translations - Example. Follow that translations are prioritized at https://translate.mattermost.com/projects/mattermost/
  3. Dev:
    • Cut release branch both for server and mobile
      • Merge database upgrade before cutting the branch
      • Point translation server to release branch after cutting
    • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release
      • After the cut-off, any PRs that include significant code changes, require approval of the release manager before merging
  4. Marketing:
    • Prepare bullet points and release headline for release announcement. Release manager to review the outline (benefits and order of major features) with PMs before sending to Justin to work on
    • Decide which sections of the release announcement will have an accompanying screenshot / photo

D. (T-minus 14 working days) Feature testing

  1. QA:
    • Prioritize testing merged PRs and resolved tickets for this release
    • Ensure that new features are also properly tested on mobile apps
    • Prioritize updating tests in the Release Testing spreadsheet and in Selenium IDE
    • Identify most-affected areas and queue Selenium tests to be updated and run

E. (T-minus 13 working days) Judgment Day

Day when Leads and PMs decide which major features are included in the release, and which are postponed.

  1. (Team) Judgment Day Meeting (10:00am San Francisco time):
    • Discuss worst bug on master
    • Finalize which major features will be in or out for the release
      • Discuss reverting feature(s) if 5 or more bugs found
    • Begin daily triage of tickets
      • Also start to triage tickets in the backlog
  2. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete
    • Update Changelog PR based on what’s in/out of the release
    • Create meta issue for release in GitHub (see example)
    • Confirm date of marketing announcement for the release date with Marketing, and update release channel header if needed
      • If release day falls on a Friday, the blog post goes out on the Friday and the emailed newsletter goes out the following Tuesday.
    • Post a reminder to devs in the Release Discussion channel of the the code complete date with the ZBB count see example
    • Based on results of Team Meeting discussion, create tickets for features that need to be turned on or off for the release
    • Ask release PM to review the JIRA tickets remaining in the current release fix version and push those that won’t make it to the next fix version
  3. Leads:
    • Finalize roadmap for next release, and identify planned marketing bullet points
  4. Marketing:
    • Start drafting blog post, tweet, and email for the release announcement

F. (T-minus 12 working days) Code complete

Stabilization period begins when all features for release have been committed. During this period, only bugs can be committed to the release branch. Non-bug pull requests are tagged for next version. Exceptions can be made by the Release Manager during triage. Review the Release Features & Bugs Quality Gate Guidelines

  1. (Team) Code Complete Meeting (10:00am San Francisco time):
    • Team review of Changelog
    • Last check of tickets that need to be merged before RC1
  2. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete
    • Review all Severity 1 bugs (data loss or security) to consider adding to Hotfix list
    • Draft Mattermost Security Updates, but do not post until 30 days after official release.
      • Add a placeholder text saying “Details on the security update will be posted here on X date, as per our Responsible Disclosure Policy”
    • If there are any breaking compatibility changes in the release, open an issue in the GitLab Omnibus to make sure GitLab is aware. Post a link to the issue in the Release Discussion channel
  3. Dev:
    • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release
  4. QA:
    • Identify any new teammates who will be joining release testing, DM them an intro to the testing process and timeframe, send them the hardware/software survey

G. (T-minus 9 working days) Release Candidate Cut

  1. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete
    • Update the GitHub meta issue:
      • Include a link to the changelog on the documentation branch
      • Post comments to the meta issue with approved fixes for the next RCs
      • Update download links and testing server links to the latest RCs
    • After build is cut, tweet announcement that RC1 is ready (see example)
  2. Logistics:
    • Mail out contributor and security researcher mugs
      • Space out the ordering of mugs over the next 3 weeks to prevent mistakes being made by the supplier. i.e. If there are 12 contributors to order mugs for, place an order every 2nd or 3rd day over the next 3 weeks.
    • Update Team page with new contributors
    • Provide release manager with a list of contributors for Changelog draft
  3. QA:
    • Confirm up to date with testing merged PRs and resolved tickets
    • Confirm up to date with test updates and known issues in release testing spreadsheet
    • Assign release testing areas to team members
    • After RC1 is cut: Update rctesting and CI server invite links in Release Testing spreadsheet
    • After RC1 is cut: Lock Selenium server to RC1
  4. Build:
    • Review all TODO notes, including one for uncommenting upgrade code
    • Confirm all PRs in /enterprise repo have been merged.
    • Update Redux before each RC and Final build
    • Master is tagged and branched and “Release Candidate 1″ is cut (e.g. 3.5.0-RC1) according to the Release Candidate Checklist in mattermost/process
    • After branching, the database version in sql_upgrade.go on master is set to the next scheduled release version (e.g. 3.6.0)
    • CI servers are updated to the release branch
    • Translation server is locked to the release branch
    • Run daily automated upgrade tests to avoid catching upgrade bugs late
  5. Docs:
    • Submit Changelog PR for team review
    • Submit any remaining documentation PRs for product updates in the release
    • Check that a redirect page has been set up in mattermost.com for any links added to the System Console
    • Submit documentation for API changes and WebSocket event changes to API documentation
    • Confirm changes to config.json in compatibility section of Changelog are written back to settings documentation
    • Confirm all new diagnostics are documented in the telemetry docs (https://docs.mattermost.com/administration/telemetry.html)

H. (T-minus 8 working days) Release Candidate Testing

  1. (Team) Daily Release Update Meeting
    • Triage Jira tickets
    • Decide when to cut next RC or final
  2. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete
    • Post list of tickets to be fixed to the Release Discussion channel (see example)
    • Update Changelog “Known Issues” section with any significant issues that were found and not fixed for the final release
  3. QA:
    • Update Release Discussion header with links to RC instances and testing spreadsheet (template)
    • Post release testing instructions to Release Discussion channel (template)
    • DM reminders to team members who are not QA or devs, or who are new to release testing
    • Post “Bug Hunter Coin” message to Reception channel (see example)
    • Begin running all Selenium IDE tests
    • At end of day, post reminders about release testing in Release Discussion and Announcements channels, DM any team members who have zero test cells marked Done
  4. Dev:
    • Make PRs for bug fixes to the release branch
    • Review PRs made from release branch and merge changes into the release branch as required and merge the release branch back into master once per day
    • Run daily automated upgrade tests to avoid catching upgrade bugs late
  5. Build:
    • Verify with Release Manager before cutting any new RCs (approved fixes should be merged)
    • Push next RC to acceptance and announce in Town Square with new RC link
    • Check CI servers running on release branch
  6. Marketing:
    • Finish draft of blog post for mattermost.com and all art work (screenshots, GIFs and twitter banners) used for the blog post
      • Upgrade should be recommended if there are security fixes in this version, with a note thanking the security researcher
    • Send blog post for release manager and PMs to review

I. (T-minus 7 working days) Release Candidate Testing Finished

  1. (Team) Daily Release Update Meeting
    • Confirm testing assigned in the release testing spreadsheet is complete
    • Continue to triage Jira tickets
    • If no blocking issues are found, PM, Dev and QA sign off on the release and plan to cut final
  2. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete
    • Verify that the final translations PR for the release is submitted
    • Confirm Changelog reflects any changes since it was merged (including known issues and contributors from all repositories)
      • If there is a security fix, confirm the Changelog recommends upgrade, with a note mentioning the security level and thanking the security researcher (security process doc for example)
  3. Marketing:
    • Send blog post for mattermost.com and all related art work for marketing lead to review
    • Find www-gitlab-com merge request for latest GitLab release blog post and make request for adding GitLab Mattermost update (see example request, example update). Post to Release Discussion channel with link to request.
  4. QA:
    • Midday: Post at-channel reminders about testing, and DM team members whose tests are not marked Done
    • Find QA or other teammates to help finish unfinished tests if needed
    • End of day: Verify all release tests are finished
    • Go through all tabs of testing spreadsheet and verify all comments and questions have been filed in JIRA as needed
    • Verify all JIRA tickets other than newly filed bugs have been tested, verified, and closed
    • As bug fixes are merged and RCs are cut, verify fixes on new RCs and post in Release Channel after testing
    • As RCs are cut, update selenium.mattermost.com to latest RC

J. (T-minus 2 working days) Release Build Cut

The final release is cut - RC cuts and bug fixes should be completed by this date. Only urgent and critical issues are considered for fixing.

  1. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete
    • Work with a developer to submit GitLab MR following this process and test the upgrade once the GitLab MR is merged and included in their RC.
    • Close GitHub meta ticket for the release
    • Add the download links, SHA-256 hash and GPG signature to the version archive
    • Merge changelog PR after review is complete
    • Update the Mattermost server download page with the links to the EE and TE bits
      • Test the download links before and after updating the page
    • Check Security Issues spreadsheet and confirm disclosure text
    • Check the security researcher was added to the Responsible Disclosure Policy page
    • Confirm link to security updates appears in blog post if there are security updates in this release, with a note thanking the security researcher
    • Update deprecated feature list in mattermost.com with new and scheduled deprecations
  2. QA:
    • Verify all PRs and tickets for the release have been tested / closed
    • Verify Selenium server was put on final RC and build passed
    • Verify smoke tests on platform and RN apps all passed
    • Post QA approval in Release Discussion channel
  3. Build:
    • Update Redux before each RC and Final build
    • Tags a new release (e.g. 1.1.0) and runs an official build which should be essentially identical to the last RC
    • Posts SHA key, md5 sum and GPG signatures of the final build to release channel
    • Post in Release Discussion with links to the EE and Team Edition bits
  4. Dev:
    • Verify the hashes (SHA-1, SHA-256 and md5) and GPG signatures are correct for both Team Edition and Enterprise Edition.
    • Test upgrade from previous version to current version, following the Upgrade Guide with database upgrades on both MySQL and Postgres
    • Test upgrade from Team Edition to Enterprise edition based on the Upgrade Guide
    • Review any changes made to install guides, and test if necessary
    • Ensure Security Policies page has been updated
    • Update dependancies after release branch is cut in mattermost-server, mattermost-webapp, desktop, mattermost-mobile and mattermost-redux
  5. Logistics:
    • Update MVP page with the most valued professional of the release and order the contributor’s coaster
  6. Docs:
    • Finalize docs
      • If reviews are not complete, hold a 30 minute doc review meeting with PMs and anyone else who has changed or reviewed docs this release and wants to join
      • Merge the docs release branch to master and verify all changes on docs.mattermost.com once the build is up
      • Submit a correction PR for any incorrect formatting or other errors missed during the initial review
  7. Marketing:
    • Receive sign off on final version of MailChimp email blast and Twitter announcement and schedule for 08:00 PST on the date of marketing announcement
    • Finalize blog post for mattermost.com, test on mobile view, and set timer for 08:00 PST on the day of release
    • Update feature lists on https://about.mattermost.com/pricing/ and https://about.mattermost.com/features/ with relevant new features
    • Add links to Admin guide in the release blog post where needed

K. (T-minus 0 working days) Release Day

  1. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete, if not alert the release manager
    • Schedule a release retrospective meeting, to be held within 5 days from the release
    • Prepare and post release metrics
    • Review and update company roadmap with which major features made it into the release
    • Post the MVP winner announcement in the Contributors channel
    • Post key dates for the next release in the Release Discussion channel and remove links to RC candidates and testing spreadsheet from the header
      • Make sure that statutory holidays for Canada and US are accounted for in the release dates
    • Check for any UserVoice feature suggestions that were completed in the current release
      • Find the release tweet and insert a link to the tweet next to the feature that shipped with the release.
    • For the next release, create the following team meetings. If they conflict with existing meetings, check with meeting owner to reschedule or reschedule the release meeting
      • Feature Complete Meeting on T-15 at 10:00am San Francisco time
      • Judgment Day Meeting on T-13 at 10:00am San Francisco time
      • Code Complete Meeting on T-12 at 10:00am San Francisco time
      • Release Triage and Update Meeting each weekday starting at T-13 and ending at T-2 at 9:30am San Francisco time for PM, QA and release dev.
    • Prepare tickets for the next release, with a corresponding vX.X prefix, and put the tickets in the appropriate sprints as follows:
    • Confirm that mattermost-docker has been updated to the latest version (contact the maintainer via direct message on pre-release if necessary)
    • Contact owners of community installers or submit PRs to update install version number
      • For Puppet, Heroku and Ansible Playbook, post to Installers and Images channel announcing the new release. See example.
      • For Chef Cookbook, open a new issue to announce the new release. See example.
      • For Yunohost, open a new pull request to update the version. See example.
      • For OpenShift, open a new issue to update the version. See example.
  2. Docs:
    • Create a new branch on docs for the next release - vX.X-documentation
      • Submit a PR for changelog against the vX.X-documentation branch and add a Work in Progress label for it
      • Submit a PR to change version number in docs/source/conf.py against the vX.X-documentation branch
  3. Logistics:
    • Close the release in Jira both for webapp and mobile (releases page)
      • If there are many unresolved tickets in the current release, ask the release manager to review the ticket queue
      • Otherwise, release the fix version (Actions > […] > Release)
  4. Build:
    • Put CI servers and translation server back onto master, and post in Release Discussion channel once done
    • Update ci-linux-mysql-prev to the previous release version
  5. Dev:
    • Confirm oss.mattermost.com is updated to final build
    • Cut release branch for Bug Fix release
    • Update existing tickets or create new ones for the next release
  6. Marketing:
    • Turn on CrazyEgg for blog post page
    • Confirm marketing has been posted (animated GIFs, screenshots, mail announcement, tweets, blog posts)
    • Update @mattermosthq Twitter profile with the next release date
    • Prepare retweet of GitLab release tweet (see example here)

L. (T-plus 5 working days) Release Updates

  1. Release Manager:
    • Post this checklist in Release Checklist channel
    • Verify all items in the last posted release checklist are complete
    • Check that dependancies have been upgraded
    • Update Security Research Hall of Fame on the Responsible Disclosure Policy page
    • Update ‘latest version of Mattermost’ supported in the Airtable Integrations Directory on the Mattermost Apps and Integrations page
  2. Leads:

M. (T-plus 30 days) Update Mattermost Security Page

  1. Release Manager:
    • Post Mattermost Security Updates after reviewing with security lead
      • If a dot release is shipping with security fixes, do not post new details until T-plus 10 working days from the dot release ship date
    • Update Security Issues spreadsheet with issue number from posted update (e.g. v3.2.0.1)

Release Numbering

Mattermost numbers its stable releases based on the following format:[Version Number].[Major Build Number].[Minor Build Number]

Version Number:

  • Indicates a major system release (e.g. 1.x.x, 2.x.x)

Major Build Number:

  • Indicates significant new functionality, (e.g. 0.5.x, 0.6.x, 0.7.x)

Minor Build Number:

  • Indicates a bug fix or security release (e.g. 1.2.5, 1.2.6)