Django Deployment Guide with Vagrant and Fabric

Although Django is a great all-in-one tool for creating websites in a very short time, its deployment can be a pain since there are a lot of steps you need to take and a lot of bits and pieces you need to fiddle with. If you don't take the easy path of using a service such as Heroku, especially looking at their limitations; you need to prepare your server by looking at a lot of guides and best practices etc. Of course you may not care about the best practices or the security of your server and want to put together a testing environment with Django's development server, but for a production environment you will need to go through some of the phases such as:

  • Making your server secure (install and configure firewall, enable SSH login, disable root login etc.)
  • Installing and configuring the Django server such as Gunicorn
  • Installing and configuring the proxy server such as Nginx
  • Setting up the virtual environment
  • If necessary, setting up some sort of automated deployment with tools such as Git, Fabric, Chef, Puppet etc.

If you do some research on the internet to find a good deployment guide for all these steps, you will be disappointed. Most of the Django deployment guides are outdated or include various bad practices, or just are not explanatory. In the end, you will need to look at some individual guides for each tool you decide to use. This makes the deployment process of Django to be tiresome.

Since I had the problems above, I decided to write a guide about how I carry out the deployment steps. I actually use a scribble of notes that I prepared for myself beforehand to write this post. Therefore, I don't guarantee the readability and clarity of this guide. If you get stuck at some step, please leave a comment below so that I can fix or update some of the parts of this post.

Throughout the guide, I will give the commands for each steps which include some parts you need to edit. Please change these according to your project:

  • <<project name>>: change to your Django project name
  • <<your name>>: change to your name with surname
  • <<your email>>: change to your e-mail address
  • <<repo address>>: change to git repository address
  • <<db user password>>: database password
  • <<secret key>>: Django app secret key
  • <<site address>>: Website's address such as berkersonmez.com

This guide considers the usage of following tools and environments:

  • Ubuntu (14.04, some sections include instructions for 16.04)
  • Django 1.10 or later
  • Vagrant
  • Gunicorn
  • Nginx
  • Git / Github
  • Fabric
  • Vagrant
  • Virtualenv
  • Pip
  • Python 3 or later
  • PostgreSQL

Preparation

We need to separate our "settings.py" file into three files. Create a settings folder instead of the "settings.py" file, add a "__init__.py" file that containsfrom .dev import * to this folder. Add these setting files to this folder:

  • "dev.py" with development database setting and following:
    from .base import *
    
    DEBUG=True
    
    try:
       from .local import *
    except ImportError:
       pass
  • "local.py" with secret key setting
  • "base.py" with all other settings

You can add "dev.py", "base.py" and "__init__.py" to the version control.

You should be using Vagrant in your development environment to run the project. I assume that you already have a running Vagrant setup and know how to login to your Vagrant instance using SSH and run commands on it.

Also, before starting to take steps below, please ensure that you secured your deployment server. You must have a user with sudo access.

Step 1: Copy project to the remote server

Our project will be accessed by the user named with our project's name. In other words, automated deployment will be carried out by a user named "<<project name>>". Thus the project files will reside on this user's home directory. We need to copy the project into the folder "/home/<<project name>>".  To do that, first install Git to the server:

sudo mkdir /home/<<project name>>
cd /home/<<project name>>
sudo apt-get update
sudo apt-get install git
git config --global user.name "<<your name>>"
git config --global user.email "<<your email>>"

Generate and install your SSH key into the server and Github: https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/ OR you can choose to clone the repository over the HTTPS address. Doing latter will cause the requirement of entering Github username and password.

Clone the git repository that contains the project into the target folder:

sudo git clone <<repo address>>

Step 2: Install Python3, Setup Tools, Pillow, Postgresql and other required libraries

Install required libraries with following commands:

sudo apt-get install build-essential python3-dev python-setuptools
sudo apt-get install libjpeg-dev libtiff-dev zlib1g-dev libfreetype6-dev liblcms2-dev
sudo apt-get install postgresql libpq-dev

Step 3: Create Postgresql user and database

In this step, we will create our project's OS user and also the user for Postgresql. We then create the database of the project and grant the user access to the database.

sudo adduser <<project name>>
sudo chown -R <<project name>>:<<project name>> /home/<<project name>>
sudo su - postgres -c "createuser -s <<project name>>"
sudo su - <<project name>> -c "createdb <<project name>>"
sudo su - postgres
psql
ALTER USER <<project name>> WITH PASSWORD '<<db user password>>';
GRANT ALL PRIVILEGES ON DATABASE <<project name>> TO <<project name>>;
\q
logout

Step 4: Install and configure Virtualenv

Virtualenv is very important for Python projects since it is required to isolate the project's environment and requirements from other projects running on the server.  We install and configure virtualenv:

sudo easy_install -U pip
sudo pip install virtualenv virtualenvwrapper stevedore virtualenv-clone
sudo /usr/local/bin/virtualenv /home/<<project name>>/.virtualenvs/<<project name>> \
    --python=/usr/bin/python3
sudo chown -R <<project name>>:<<project name>> /home/<<project name>>
sudo echo home/<<project name>> > /home/<<project name>>/.virtualenvs/<<project name>>/.project

Step 5: Configure Django settings file for production environment

Since we don't include our database settings and secret key in version control, we need to create settings files for these settings manually in our production environment. You should apply the following steps with our newly created "<<project name>>" user.

Inside your Django settings folder, edit "__init__.py" as follows:

from .prod import *

Create "prod.py" with following content:

from .base import *
 
DEBUG = False
 
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': '<<project name>>',
        'USER': '<<project name>>',
        'PASSWORD': '<<db user password>>',
        'HOST': '',  # Set to empty string for localhost.
        'PORT': '',  # Set to empty string for default.
        'CONN_MAX_AGE': 600,  # number of seconds database connections should persist for
    }
}
 
try:
   from .local import *
except ImportError:
   pass

Create "local.py" with following content:

SECRET_KEY = '<<secret key>>'

Step 6: Configure Gunicorn

We will configure Gunicorn to bind to our Django project socket file. This step has some changes according to your Ubuntu version, refer to the warning to see changes for Ubuntu 16.04.

Make sure your "requirements.txt" file is similar to this:

Django==1.9.7
psycopg2>=2.6
Pillow==2.7.0
django-ckeditor>=5.0.3
gunicorn==19.6.0

Activate the virtual environment and install the requirements:

sudo su - <<project name>>
source /home/<<project name>>/.virtualenvs/<<project name>>/bin/activate
pip install -r /home/<<project name>>/requirements.txt
deactivate
exit

Create gunicorn service entry:

sudo nano /etc/init/gunicorn.conf

Write following contents and save:

description "Gunicorn application server handling <<project name>>"
 
start on runlevel [2345]
stop on runlevel [!2345]
 
respawn
setuid <<project name>>
setgid www-data
chdir /home/<<project name>>
 
exec .virtualenvs/<<project name>>/bin/gunicorn --workers 5 --bind unix:/home/<<project name>>/<<project name>>.sock <<project name>>.wsgi:application

Activate gunicorn service:

sudo service gunicorn start

Warning: In Ubuntu 16.04, init service is changed to "systemd" so you need to apply different steps to configure gunicorn. If so, apply following steps instead:

sudo nano /etc/systemd/system/gunicorn.service
[Unit]
Description=gunicorn daemon
After=network.target

[Service]
User=<<project name>>
Group=www-data
WorkingDirectory=/home/<<project name>>/<<project name>>
ExecStart=/home/<<project name>>/.virtualenvs/<<project name>>/bin/gunicorn --workers 3 --bind unix:/home/<<project name>>/<<project name>>/<<project name>>.sock <<project name>>.wsgi:application

[Install]
WantedBy=multi-user.target

sudo systemctl start gunicorn
sudo systemctl enable gunicorn

Step 7: Configure Nginx

Edit /etc/nginx/sites-available/<<project name>> file to include following server settings:

server {
    listen 80;
    server_name <<site address>>;
 
    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/<<project name>>/<<project name>>;
    }
 
    location / {
        include proxy_params;
        proxy_pass http://unix:/home/<<project name>>/<<project name>>.sock;
    }
}

Test nginx and start if there is no error:

sudo nginx -t
sudo service nginx restart

Warning: Make sure that the group "www-data" and user "www-data" has read/write permissions for the socket file.

Step 8: Carry out manage.py tasks

Using project's user, activate the virtual environment and run following tasks to apply initial Django management tasks. You may need to run python3.5 instead of python3 command if you use Python 3.5:

python3 manage.py migrate
python3 manage.py collectstatic
python3 manage.py compilemessages
python3 manage.py createsuperuser

Restart gunicorn and check if the site is working.

Step 9: Setup Fabric configuration

Add the project user to the sudo group:

sudo adduser <<project name>> sudo

Make sure <<project user>> is owner of ".git/FETCH_HEAD".

To carry out the deployment with Fabric, we need to configure SSH login for our project's user. Create a SSH key for our "<<project name>>" user. Add the public key to the user's ".ssh" folder on the server. Move the private key to "vagrant_data" folder in your development environment and rename the keyfile to "ssh_key". The "vagrant_data" should be at the project root with the same level as the "fabfile.py" file. Do not add the key file to the repository.

Create "fabfile.py" file in your project directory and add it to the repository. The contents of the "fabfile.py" should be as following:

from fabric.api import run, env, cd, sudo, prefix, local
from contextlib import contextmanager as _contextmanager
 
env.activate = 'source /home/<<project name>>/.virtualenvs/<<project name>>/bin/activate'
env.directory = '/home/<<project name>>'
env.key_filename = 'vagrant_data/ssh_key'
 
 
@_contextmanager
def virtualenv():
    with prefix(env.activate):
        yield
 
 
def deploy():
    with cd(env.directory):
 
        run('git pull origin master')
 
        with virtualenv():
            sudo('pip install -r ' + env.directory + '/requirements.txt')
            run('python3 manage.py migrate')
            run('python3 manage.py collectstatic')
            # run('python3 manage.py compilemessages')
 
        sudo('service gunicorn restart')

According to this configuration, when we run the deploy command:

  • Project user will change directory to the project directory.
  • Git repository will be updated using the pull command.
  • Virtual environment will be activated.
  • Pip requirements will be installed if necessary.
  • Database migrations will be applied.
  • Static files will be prepared.
  • If you uncomment the "compilemessages" line, the localization messages will be updated.
  • Gunicorn service will be restarted 

Warning: If you are using Ubunto 16.04 or later, you need to change the last line to sudo('systemctl restart gunicorn')

You can add new commands to the Fabric file according to your needs, please refer to the Fabric documentation to understand how Fabric works.

Add following to the requirements.txt and install requirements to the Vagrant box (login to your Vagrant box through SSH and run pip install -r requirements.txt from the project root folder):

Fabric3==1.11.1.post1

Also instead of adding fabric to your project "requirements.txt" file, you may choose to install it manually using pip install command since it is not required to be installed to the server.

Step 10: Automated deployment

You will need to apply the following steps when you update the project and want to deploy to the server.

SSH into the Vagrant box and run fab deploy. You will need to supply the host connection string to Fabric when you run the command. Write it as "<<project name>>@<<server ip>>:22":

source .virtualenvs/<<project name>>/bin/activate
fab deploy
No hosts found. Please specify (single) host string for connection: <<project name>>@<<server ip>>:22
[<<project name>>@xx.xxx.xxx.xx:22] run: git pull origin master
[<<project name>>@xx.xxx.xxx.xx:22] Login password for '<<project name>>':

As seen above, it will ask for the passphrase for the private key in your "vagrant_data" folder.  Enter the passphrase. If asked for the Github credentials, enter them. An example run for "quenchless" application (one of the projects of mine) can be seen below:

(quenchless) vagrant@vagrant-ubuntu-trusty-32:~/quenchless$ fab deploy
No hosts found. Please specify (single) host string for connection: quenchless@46.101.214.89:22
[quenchless@46.101.214.89:22] run: git pull origin master
[quenchless@46.101.214.89:22] Login password for 'quenchless':
[quenchless@46.101.214.89:22] out: Username for 'https://github.com': berkersonmez
[quenchless@46.101.214.89:22] out: Password for 'https://berkersonmez@github.com':
[quenchless@46.101.214.89:22] out: From https://github.com/berkersonmez/quenchless
[quenchless@46.101.214.89:22] out:  * branch            master     -> FETCH_HEAD
[quenchless@46.101.214.89:22] out: Already up-to-date.
[quenchless@46.101.214.89:22] out:
 
[quenchless@46.101.214.89:22] run: pip install -r /home/quenchless/requirements.txt
[quenchless@46.101.214.89:22] out: Requirement already satisfied (use --upgrade to upgrade): Django==1.9.7 in ./.virtualenvs/quenchless/lib/python3.4/site-packages (from -r /home/quenchless/requirements.txt (line 1))
[quenchless@46.101.214.89:22] out: Requirement already satisfied (use --upgrade to upgrade): psycopg2>=2.6 in ./.virtualenvs/quenchless/lib/python3.4/site-packages (from -r /home/quenchless/requirements.txt (line 2))
[quenchless@46.101.214.89:22] out: Requirement already satisfied (use --upgrade to upgrade): Pillow==2.7.0 in ./.virtualenvs/quenchless/lib/python3.4/site-packages (from -r /home/quenchless/requirements.txt (line 3))
[quenchless@46.101.214.89:22] out: Requirement already satisfied (use --upgrade to upgrade): django-ckeditor>=5.0.3 in ./.virtualenvs/quenchless/lib/python3.4/site-packages (from -r /home/quenchless/requirements.txt (line 4))
[quenchless@46.101.214.89:22] out: Requirement already satisfied (use --upgrade to upgrade): gunicorn==19.6.0 in ./.virtualenvs/quenchless/lib/python3.4/site-packages (from -r /home/quenchless/requirements.txt (line 5))
[quenchless@46.101.214.89:22] out:
 
[quenchless@46.101.214.89:22] run: python3 manage.py migrate
[quenchless@46.101.214.89:22] out: Operations to perform:
[quenchless@46.101.214.89:22] out:   Apply all migrations: berkersonmez, sessions, auth, admin, contenttypes
[quenchless@46.101.214.89:22] out: Running migrations:
[quenchless@46.101.214.89:22] out:   No migrations to apply.
[quenchless@46.101.214.89:22] out:
 
[quenchless@46.101.214.89:22] run: python3 manage.py collectstatic
[quenchless@46.101.214.89:22] out:
[quenchless@46.101.214.89:22] out: You have requested to collect static files at the destination
[quenchless@46.101.214.89:22] out: location as specified in your settings:
[quenchless@46.101.214.89:22] out:
[quenchless@46.101.214.89:22] out:     /home/quenchless/quenchless/static
[quenchless@46.101.214.89:22] out:
[quenchless@46.101.214.89:22] out: This will overwrite existing files!
[quenchless@46.101.214.89:22] out: Are you sure you want to do this?
[quenchless@46.101.214.89:22] out:
[quenchless@46.101.214.89:22] out: Type 'yes' to continue, or 'no' to cancel: yes
[quenchless@46.101.214.89:22] out:
[quenchless@46.101.214.89:22] out: 0 static files copied to '/home/quenchless/quenchless/static', 1427 unmodified.
[quenchless@46.101.214.89:22] out:
 
[quenchless@46.101.214.89:22] sudo: service gunicorn restart
[quenchless@46.101.214.89:22] out: sudo password:
[quenchless@46.101.214.89:22] out: Sorry, try again.
[quenchless@46.101.214.89:22] out: sudo password:
 
[quenchless@46.101.214.89:22] out: gunicorn stop/waiting
[quenchless@46.101.214.89:22] out: gunicorn start/running, process 20367
[quenchless@46.101.214.89:22] out:
 
 
Done.
Disconnecting from 46.101.214.89... done.

Congratulations, the application has been deployed!

Troubleshooting

400 Bad Request when opening the page

Make sure that Django settings file has the ALLOWED_HOSTS setting with the domain you are trying to access the project from:

ALLOWED_HOSTS = ("yourdomain.com",)