Deploying for production

In this guide, we’ll cover the key steps to get Bugsink up and running in your production environment. Our goal is to make the process as simple and hassle-free as possible. If you want something even simpler, the local installation guide might be more suitable.

The recommended way to deploy Bugsink is on a single virtualized server. This minimizes complexity and increases robustness and performance. It also scales well: a single cheap server can handle up to 2.5 million events per month.

The components you’ll set up are:

Prerequisites

This guide assumes Ubuntu 24.04 LTS as the operating system. Feel free to use another Linux system, though you may need to substitute commands here and there.

This is what you should know and have before you start:

  • You know your way around the command line
  • You have root ssh access to a fresh system with the single purpose of running Bugsink.
  • DNS is already set up, i.e. an A-record for your hostname is set up to point to the IP address of your server.

Python, pip and venv

Bugsink is a Python application, so you’ll need Python 3.8 or later. You’ll also need pip. Python comes pre-installed on Ubuntu, but to make sure it is and you have the correct version, run:

python3 --version

and similarly for pip:

pip3 --version

and for venv:

python3 -m venv --version

If any of these commands fail, you can install the necessary packages by running:

apt update
apt upgrade
apt install python3 python3-pip python3-venv -y

Set up a non-root user

It’s a good practice to use a non-root user to run the Bugsink server. You can create a new user by running, as root:

adduser bugsink --disabled-password --gecos ""

Switch to the new user by running:

su - bugsink

You should now be in /home/bugsink. This is where we’ll put Bugsink’s codebase, the configuration for Bugsink and the database.

Set up a virtual environment and activate it

To avoid conflicts between Bugsink’s dependencies and other Python packages on your system, it’s a good practice to use a virtual environment. Run the following commands to create a one and activate it:

python3 -m venv venv
. venv/bin/activate

After running these commands, you should see the name of the virtual environment in parentheses at the start of your shell prompt, i.e. (venv).

Install Bugsink and its dependencies

You can install Bugsink using pip:

python3 -m pip install Bugsink --upgrade

You should see output indicating that Bugsink and its dependencies are being installed. After the installation is complete, you can verify that Bugsink is installed by running:

bugsink-show-version

This should print the version of Bugsink that was installed.

Configuration

Bugsink relies on a configuration file to determine how it should run. You can create a configuration file that’s suitable for production by running the following command (replace YOURHOST with the hostname of your server):

bugsink-create-conf --template=recommended --host=YOURHOST

This will create a file bugsink_conf.py in the current directory, i.e. in /home/bugsink/. Open this file in your favorite editor.

nano bugsink_conf.py

The file is based on a template which matches the current guide, so you will not need to change much. The site-specific settings you may want to change are:

  • BASE_URL to match the URL where you want to access Bugsink.
  • SITE_NAME a display-name for your site if you want to distinguish it from other Bugsink instances.
  • DEFAULT_FROM_EMAIL to match the email address from which Bugsink will send emails.
  • EMAIL_HOST and associated variables to match the SMTP server you want to use to send emails.
  • TIME_ZONE to match your timezone (if you want to see times in your local timezone rather than UTC).

Also noteworthy are the settings that control the “openness” of your particular installation, i.e. whether users can sign up themselves or need to be invited and whether anybody can start a new team or whether this is managed. Look for:

  • USER_REGISTRATION-prefixed settings
  • SINGLE_USER to allow only a single user (or not)
  • SINGLE_TEAM to allow only a single team (or not)
  • TEAM_CREATION to allow team creation by anybody (or not)

Initialize the database

Bugsink uses a database to store the data it collects. When set up with Snappea, it additionally uses a separate database as a message queue. You can initialize both these databases by running:

bugsink-manage migrate
bugsink-manage migrate snappea --database=snappea

This will create two new SQLite database in the location specified in the configuration file (by default: /home/bugsink) and set up the necessary tables. You may verify their presence by running:

ls *.sqlite3

Create the first user

Create a superuser to manage the Bugsink installation. Run the following command and follow the prompts. To stay consistent with usernames in the rest of the system, you may want to use your email-address as a username:

bugsink-manage createsuperuser

This will create a new user account with administrative privileges.

Check your configuration

Run the following commands to check that the steps you’ve taken so far have been successful:

bugsink-manage check_migrations
bugsink-manage check --deploy --fail-level WARNING

Gunicorn, the WSGI server

We will run Bugsink using Gunicorn, a WSGI server. Gunicorn was already installed as part of the Bugsink dependencies, so we just need to run it.

Rather than running Gunicorn directly, we will use a systemd service to manage the process.

Exit back to the root user by running (as the bugsink user):

exit

Create a file: /etc/systemd/system/gunicorn.service with the following contents:

[Unit]
Description=gunicorn daemon
After=network.target

[Service]
Restart=always
Type=notify
User=bugsink
Group=bugsink

Environment="PYTHONUNBUFFERED=1"
RuntimeDirectory=gunicorn
WorkingDirectory=/home/bugsink
ExecStart=/home/bugsink/venv/bin/gunicorn \
    --bind="127.0.0.1:8000" \
    --workers=10 \
    --timeout=6 \
    --access-logfile - \
    --max-requests=1000 \
    --max-requests-jitter=100 \
    bugsink.wsgi
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
TimeoutStopSec=5

[Install]
WantedBy=multi-user.target

A few notes on the configuration:

  • --bind="127.0.0.1:8000" sets the address and port on which Gunicorn will listen. We will set up Nginx to forward requests to this address and port.
  • --workers=10 sets the number of worker processes to 10. With Bugsink running (far) below 100MiB per worker, this should easily fit into a 2GiB server, while still providing very good throughput. Our general recommendation for reliability is to stay well below 50% of available memory, even if this comes at some cost in throughput: no need to fly too close to the sun.
  • --timeout=6 sets the timeout for a worker to 6 seconds. Bugsink is easily able to handle its requests in less than a second, and the DB timeout is set to 5 seconds, so this should be more than enough.
  • access-logfile is set up to forward the access-log to the systemd journal.
  • max-requests and max-requests-jitter are set up for occasional restarts of the workers to avoid memory leaks.

Enable and start the service (enabling means it will also start on boot):

systemctl enable --now gunicorn.service

Inspect the status of gunicorn using

systemctl status gunicorn.service

To test whether gunicorn actually listens on the socket, and whether everything can be reached, use:

curl http://localhost:8000/accounts/login/ --header "Host YOURHOST"

(Replacing YOURHOST with the hostname of your server). This should dump a bunch of html on screen.

Nginx

We will first set up Nginx to run on port 80. Once this is running correctly, we can use certbot to set up SSL automatically.

Setting up Nginx

Install nginx:

apt install nginx

To verify that nginx is running, and that your DNS record is pointing to your server, you may enter the hostname of your server in your browser. You should see the default nginx welcome page (“Welcome to nginx!”).

Remove the default configuration file:

rm /etc/nginx/sites-enabled/default

At /etc/nginx/sites-available/bugsink, create a configuration file for your site, with the following contents:

server {
    server_name                 YOURHOST;
    listen                      80;
    location / {
        proxy_pass              http://127.0.0.1:8000;
        proxy_set_header        Host $host;
    }
}

(YOURHOST should be replaced with the full hostname of your server).

Create a link from sites-enabled to sites-available:

ln -s /etc/nginx/sites-available/bugsink /etc/nginx/sites-enabled

Check the configuration file for syntax errors:

service nginx configtest

If there are no errors, restart nginx:

systemctl restart nginx

You should now be able to access Bugsink by going to http://YOURHOST in your browser.

Congratulations! Resist the temptation to log in though… set up SSL first to avoid sending your unencrypted password over the internet.

Setting up SSL

To set up SSL, we will use certbot, a tool that automates the process of obtaining and renewing SSL certificates. Certbot can be installed as a snap package:

snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

Next, we actually run certbot to obtain the certificate.

  • We will use the --nginx plugin to automatically configure Nginx to use the SSL certificate.
  • --no-redirect is used to avoid redirecting all HTTP traffic to HTTPS. We will set this up ourselves in the next step.

Run certbot like this and follow the prompts:

certbot --nginx --rsa-key-size 4096 --no-redirect

If you wish, you can take a look at the configuration file in /etc/nginx/sites-available/bugsink to see how certbot has modified it.

After this step you should be able to access Bugsink using HTTPS. Open your browser and enter the hostname of your server, but with https instead of http. You should see the Bugsink interface, and the browser should indicate that the connection is secure.

Final Nginx configuration

A few final touches to the Nginx configuration are necessary to ensure that all traffic is encrypted and that the server only responds to requests for the correct hostname:

  • Set up automatic redirects from HTTP to HTTPS to ensure that all traffic is encrypted
  • Use HSTS to ensure that browsers only connect to your site over HTTPS
  • Avoid host spoofing by setting up a catch-all server_name directive

(Note that you cannot skip to this step without first setting up a port-80 configuration file as described above, because certbot will not be able to verify your domain otherwise).

Simply copy the configuration below and replace YOURHOST with the hostname of your server to ensure that your Nginx configuration is secure:

# Disable nginx version number in headers (unnecessary information
# for potential attackers)
server_tokens off;

# Catch-all server block to avoid host spoofing, i.e. to ensure
# that the server only responds to requests for the correct
# hostname
server {
    listen 80 default_server;
    return 444;
}

# Redirect HTTP to HTTPS
server {
    server_name YOURHOST;
    listen 80;
    return 307 https://$host$request_uri;
}

server {
    server_name YOURHOST;

    # match the configuration for MAX_ENVELOPE_COMPRESSED_SIZE.
    # Note that Nginx's "M" means MiB.
    client_max_body_size 20M;

    access_log /var/log/nginx/bugsink.access.log;
    error_log /var/log/nginx/bugsink.error.log;


    location / {
        # Pass the request to Gunicorn via proxy_pass.
        proxy_pass http://127.0.0.1:8000;

        # Set the Host header to the original host.
        proxy_set_header Host $host;

        # Set the X-Forwarded-Proto header to the original scheme;
        # because Django/Bugsink is behind a proxy, it needs to
        # know the original scheme to know whether the current
        # request is secure or not. This directive corresponds to
        # the setting "SECURE_PROXY_SSL_HEADER" in your
        # bugsink_conf.py file.
        proxy_set_header X-Forwarded-Proto $scheme;

        # Because the server is behind a proxy, it needs to know
        # the original IP address of the client. This information
        # is passed on in the X-Forwarded-For header, and picked
        # up by Bugsink because of the setting
        # "USE_X_FORWARDED_FOR"
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # Set up HSTS with a long max-age (1 year) and the
        # "preload" directive, which tells browsers to include
        # your site in their HSTS preload list. This means that
        # browsers will only connect to your site over HTTPS, even
        # if the user types in the URL without the "https://"
        # prefix.
        add_header Strict-Transport-Security "max-age=31536000; preload" always;
    }

    # This whole block is auto-generated by Certbot;
    # Alternatively, use the block below from the previous version
    # of the configuration file, i.e.  the version of the file
    # right after you ran certbot:
    listen 443 ssl;  # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/YOURHOST/fullchain.pem;  # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/YOURHOST/privkey.pem;  # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf;  # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;  # managed by Certbot
}

Make sure to check the configuration file for syntax errors, and restart nginx after making changes:

service nginx configtest
systemctl restart nginx

Snappea

We will use Snappea to handle background tasks, i.e. tasks that are not immediately necessary to complete a request and are potentially time-consuming. We will set up Snappea to run as a systemd service.

Add a file /etc/systemd/system/snappea.service with the following contents:

[Unit]
Description=snappea daemon

[Service]
Restart=always
User=bugsink
Group=bugsink

Environment="PYTHONUNBUFFERED=1"
RuntimeDirectory=gunicorn
WorkingDirectory=/home/bugsink
ExecStart=/home/bugsink/venv/bin/bugsink-runsnappea
KillMode=mixed
TimeoutStopSec=5
RuntimeMaxSec=1d

[Install]
WantedBy=multi-user.target

Enable and start the service by running:

systemctl enable --now snappea.service

You may check whether this was successful by running:

systemctl status snappea.service

To ensure that Snappea is actually picking up tasks, you may additionally do the following:

# log in as bugsink user
su - bugsink

# activate the virtual environment
. venv/bin/activate

# run the following command to add a test-task to the queue
bugsink-manage checksnappea

# exit back to root
exit

The snappea journal should then show that the task was picked up and executed. Check by running:

journalctl -u snappea.service

This should show a log entry indicating that the task was picked up and executed, i.e. something like:

Starting 000-001 for "snappea.example_tasks.fast_task" with (), {}
Worker done in 0.000s

Congratulations!

With snappea set up you’re ready to actually start using Bugsink.

Log in with the superuser credentials you set up earlier and configure your first project.

Onc that’s set up, you can start using Bugsink to collect crash reports for your applications.