How to upgrade old version of postgres to latest in 2027?

Hi Noonari.

I was working on everything-personal project. I ran into a hiccup when I ran the `uv run manage.py migrate`. I was on `django 6.1.2`. And it turns out that from Django version 6.1 onwards, django will not be supporting postgres 14.XX or below along with any mysql below mysql 8.4. 

I had two options. One take a step back to django version 5.2LTS or stay on the current version and take on the challenge of upgrading the postgres version. Ofcourse, I lack foresight, I chose later. I added another challenge to this. I will not use AI to do this. Claude could have helped me do all this without a hiccup and I would have been done and dusted in like less than 30 minutes. But it is me. You know me. How I was. So Here is it.

I started from postgres official documentation. Why you ask? I did that because, every tom dick and harry of a search engine website has added AI explanation/Summary. I learnt about pg_upgrade. Here is what the docs taught me. pg_upgrade upgrades a cluster in place. It needs the old Postgres and the new Postgres installed side by side, and it moves your data files from one to the other. Powerful, but a bit raw. You have to point it at both binary folders, both data directories and both configs, and one wrong path is a bad afternoon.

Then I found out Ubuntu ships a friendlier wrapper called pg_upgradecluster. It figures out all the paths for you. By default it does a dump and restore into the new cluster, which is slower than pg_upgrade on huge databases but perfectly fine for my 39 MB of personal projects. If you want the real pg_upgrade behind it, there is a --method=upgrade flag. Good to know for later.

Now the actual work.

1. Look around first.

   pg_lsclusters

A cluster is one running Postgres server with its own port and data folder. I had 14 on port 5432. Remember this, you will need it.

2. Backup. Before anything else. The docs repeat this like a broken record, so I listened.

   cd /tmp && sudo -u postgres pg_dumpall > ~/pg14_backup.sql

The cd /tmp is there because the postgres user cannot enter my home folder and complains about it. The complaint is harmless.

3. Check the backup is real.

   ls -lh ~/pg14_backup.sql
   grep -c "^CREATE DATABASE" ~/pg14_backup.sql
   tail -n 3 ~/pg14_backup.sql

39 MB, 12 databases, and the last line said PostgreSQL database cluster dump complete. Good.

4. Add the official PostgreSQL repo and install 18. Ubuntu’s own repo lags behind.

   sudo apt install -y postgresql-common
   sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
   sudo apt install -y postgresql-18
   pg_lsclusters

Surprise. The installer quietly created an empty 18 main cluster on port 5433 because 14 was already on 5432. I also found a forgotten empty 16 main cluster hiding on 5434. Always run pg_lsclusters before you trust your memory.

5. Get rid of the empty 18 cluster so the upgrade can use the name main.

   sudo pg_dropcluster 18 main --stop

6. The upgrade itself. One command.

   sudo pg_upgradecluster -v 18 14 main

It builds a fresh 18 cluster, copies every database and user over, gives it port 5432, and parks the old 14 on another port, switched off. My Django settings did not need a single change.

7. Verify. Do not skip this.

   pg_lsclusters
   sudo -u postgres psql -c "SELECT version();"
   sudo -u postgres psql -l
   uv run manage.py migrate

PostgreSQL 18.6, all my databases listed, and migrate ran with no version error. That was the moment.

8. Clean up, but only after step 7 passes.

   sudo pg_dropcluster 14 main
   sudo apt remove -y postgresql-14

I kept the backup file anyway. It costs nothing.

Two things that scared me for no reason:

• collation version mismatch warnings showed up on the old clusters. Ubuntu updated its system libraries after those clusters were created. The upgrade builds the new cluster against the current libraries, so it does not carry over.
• apt suggested apt autoremove, and the list had ffmpeg and ImageMagick libraries in it. Nothing to do with Postgres. I removed only the old postgresql-client-xx packages by hand.

What I took away from it: the “scary” upgrade was really three ideas. A cluster is just a server with a port. A backup you have checked is the reason you can stay calm. And the old cluster stays on disk until you decide to delete it, so there is a way back until the very last step.

With this today’s lecture ends. Remember Noonari, the docs are slower than a chatbot, but they stick. Stay sharp and stay hungry.

Deploying High-Performance Django on Coolify

Part 1: Initial Setup in Coolify Dashboard

Step 1: Accessing Coolify

1. Open your browser and navigate to your Coolify instance (e.g., https://coolify.yourdomain.com)
2. Log in with your credentials
3. You'll see the main dashboard with "Projects" and "Resources" sections

Step 2: Creating a New Project

1. Click the "New Project" button in the top right corner
2. Fill in the form:
   • Name: "Django-High-Performance"
   • Description: "High-performance Django application with PostgreSQL"
   • Click "Create Project"

Step 3: Setting up PostgreSQL Database

1. In your project, click "New Resource"
2. Select "Database" from the resource types
3. Choose "PostgreSQL"
4. Fill in the configuration:

   Name: postgres-main
   Version: 15
   Username: django_user
   Password: [generate a secure password]
   Database Name: django_db
   Port: 5432

5. Under "Advanced Settings":

   Memory Limit: 12288 (12GB in MB)
   CPU Limit: 4

6. Click "Create Database"
7. After creation, click on the database resource
8. Go to "Settings" tab
9. Under "Configuration", add these settings:

   max_connections = 200
   shared_buffers = 4GB
   effective_cache_size = 12GB
   maintenance_work_mem = 1GB

10. Click "Save Configuration" and "Restart Database"

Step 4: Setting up PgBouncer

1. Click "New Resource"
2. Select "Service" → "Docker"
3. Configuration:

   Name: pgbouncer
   Image: edoburu/pgbouncer:1.18.0
   Port: 6432

4. Click "Add Volume" and add:

   Host Path: /data/pgbouncer/pgbouncer.ini
   Container Path: /etc/pgbouncer/pgbouncer.ini

5. Add another volume:

   Host Path: /data/pgbouncer/userlist.txt
   Container Path: /etc/pgbouncer/userlist.txt

6. Under "Environment Variables", add:

   DB_HOST=postgres-main
   DB_PORT=5432
   DB_USER=django_user
   DB_PASSWORD=[your-postgres-password]

7. Click "Create Service"

Step 5: Setting up Redis

1. Click "New Resource"
2. Select "Database" → "Redis"
3. Configure:

   Name: redis-cache
   Version: 7
   Password: [generate secure password]
   Memory Limit: 4096 (4GB in MB)
   CPU Limit: 2

4. Click "Create Redis"

Part 2: Preparing Your Application

Step 1: Project Files Setup

Create these files in your Django project:

Dockerfile:

FROM python:3.11-slim

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1

WORKDIR /app

RUN apt-get update && apt-get install -y \
    build-essential \
    postgresql-client \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt \
    gunicorn[gevent] \
    redis \
    hiredis \
    django-redis \
    psycopg2-binary

COPY . .

RUN python manage.py collectstatic --noinput

RUN useradd -m myuser
RUN chown -R myuser:myuser /app
USER myuser

CMD ["gunicorn", "--config", "gunicorn.conf.py", "core.wsgi:application"]

gunicorn.conf.py:

import multiprocessing

workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'gevent'
worker_connections = 1000
timeout = 30
keepalive = 2
max_requests = 1000
max_requests_jitter = 200
preload_app = True
bind = "0.0.0.0:8000"

# Logging
accesslog = '-'
errorlog = '-'
loglevel = 'info'

Step 2: Update Django Settings

Update settings.py:

import os
from pathlib import Path

BASE_DIR = Path(__file__).resolve().parent.parent

SECRET_KEY = os.getenv('DJANGO_SECRET_KEY')
DEBUG = os.getenv('DJANGO_DEBUG', 'False') == 'True'
ALLOWED_HOSTS = os.getenv('DJANGO_ALLOWED_HOSTS', '').split(',')

# Database
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': os.getenv('POSTGRES_DB'),
        'USER': os.getenv('POSTGRES_USER'),
        'PASSWORD': os.getenv('POSTGRES_PASSWORD'),
        'HOST': os.getenv('POSTGRES_HOST'),
        'PORT': os.getenv('POSTGRES_PORT'),
        'CONN_MAX_AGE': 0,
    }
}

# Cache
CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': os.getenv('REDIS_URL'),
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'PARSER_CLASS': 'redis.connection.HiredisParser',
            'CONNECTION_POOL_CLASS_KWARGS': {
                'max_connections': 50,
                'timeout': 20,
            }
        }
    }
}

# Session
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
SESSION_CACHE_ALIAS = 'default'

Part 3: Deploying the Application

Step 1: Creating the Application Service

1. In Coolify dashboard, click "New Resource"
2. Select "Application" → "Docker"
3. Select your Git repository
4. Configure Build:

   Branch: main (or your preferred branch)
   Dockerfile Path: ./Dockerfile
   Port: 8000

5. Under "Advanced Settings":

   Memory Limit: 8192 (8GB in MB)
   CPU Limit: 8
   Instance Count: 2

Step 2: Environment Variables

1. Click on "Environment Variables"
2. Add the following (click "Add" after each):

   # Django
   DJANGO_SECRET_KEY=[generate a secure key]
   DJANGO_DEBUG=False
   DJANGO_ALLOWED_HOSTS=your-domain.com
   DJANGO_CSRF_TRUSTED_ORIGINS=https://your-domain.com
   
   # Database
   POSTGRES_DB=django_db
   POSTGRES_USER=django_user
   POSTGRES_PASSWORD=[your-postgres-password]
   POSTGRES_HOST=pgbouncer
   POSTGRES_PORT=6432
   
   # Redis
   REDIS_URL=redis://:[redis-password]@redis-cache:6379/0
   
   # Port
   PORT=8000

Step 3: Network Configuration

1. Go to "Network" tab
2. Click "Add to Network"
3. Select the network containing your database and Redis
4. Click "Add"

Step 4: Initial Deployment

1. Click "Deploy" button
2. Watch the build logs (click "View Logs")
3. Wait for the build to complete

Step 5: Database Migration

1. After successful deployment, click "Terminal"
2. Run:

   python manage.py migrate
   python manage.py createsuperuser

Part 4: Post-Deployment Configuration

Step 1: Domain Setup

1. Go to "Settings" tab
2. Under "Domains", click "Add Domain"
3. Enter your domain
4. Enable "HTTPS/SSL"
5. Choose "Let's Encrypt"
6. Click "Add Domain"

Step 2: Monitoring Setup

1. Go to "Monitoring" tab
2. Enable "Resource Monitoring"
3. Configure alerts:
   • Click "Add Alert"
   • Set CPU Usage > 80%
   • Add email notification
4. Repeat for:
   • Memory Usage > 80%
   • Error Rate > 1%

Common Issues and Solutions

Problem 1: Database Connection Failed

If you see "could not connect to database" errors:

1. Check network connectivity:

   # In application terminal
   nc -zv pgbouncer 6432

2. Verify environment variables:

   # In application terminal
   env | grep POSTGRES

3. Check PgBouncer logs:
   • Go to PgBouncer service
   • Click "Logs"

Problem 2: Redis Connection Issues

If cache errors occur:

1. Verify Redis connection:

   # In Django shell
   from django.core.cache import cache
   cache.set('test', 'working')
   cache.get('test')

2. Check Redis URL format
3. Verify network connectivity

Problem 3: Static Files Not Loading

1. Check STATIC_ROOT setting
2. Run collectstatic again:

   python manage.py collectstatic --clear --noinput

3. Verify whitenoise configuration

Performance Monitoring

Step 1: Database Monitoring

Run these queries in PostgreSQL:

-- Current connections
SELECT count(*) FROM pg_stat_activity;

-- Connection states
SELECT state, count(*) 
FROM pg_stat_activity 
GROUP BY state;

-- Slow queries
SELECT pid, age(clock_timestamp(), query_start), usename, query 
FROM pg_stat_activity 
WHERE state != 'idle' 
  AND query NOT ILIKE '%pg_stat_activity%' 
ORDER BY query_start desc;

Step 2: Application Monitoring

1. Check Gunicorn workers:

   # In application terminal
   ps aux | grep gunicorn

2. Monitor request latency:
   • Go to "Metrics" tab
   • Look for "Request Duration"
   • Check "95th percentile" latency

How to setup an SSH Key and connect to ubuntu server

1. Create SSH Key using
   ssh-keygen
   
   
2. Copy .pub key to server
   ssh-copy-id username@remote_host
   
   
3. Validate previous steps by connecting to server
   ssh username@remote_host
   
   
4. disable password authentication
5. enable firewall

How to run Django Server from VSCode

 Hi Noonari, 

Here is how  I did it.

Step 1. Install following plugins

• Python (by microsoft)
• Live Server

Step 2. Set python version to your virtual env by following these steps in bullets in order

1. Press `CTRL+SHIFT+P`
   This will open a command pallet in vscode.
2. Set the python version by selecting if it is visible in the dropdowns or by typing it like
   `Python > <path-to-venv>/bin/python`
3. create launch.json (normally it is automatically created when you press run button so first press run and see if it runs or not and then check or create launch.json)
4. add this as contents of launch.json
   
   {

       // Use IntelliSense to learn about possible attributes.

       // Hover to view descriptions of existing attributes.

       // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387

       "version": "0.2.0",

       "configurations": [

           {

               "name": "Python: Django",

               "type": "python",

               "request": "launch",

      

               "program": "${workspaceFolder}/monolith/manage.py",

               "args": [

                   "runserver",

                   "--noreload"

               ],

               "django": true,

               "justMyCode": true

           }

       ]

   }

Step 3. Run and test.

Beautifying your Terminal in Linux

 Hi Noonari, long time no see.

So this is about how you can increase your productivity by reducing clutter in your terminal. It may not be up to your test in future but trust me when I am writing this for you, this had mind blowing productivity results. 

1. So here's the deal. You are deep in the repositories. And here you have to write some crazy long command on terminal. What would happen, it will jump to new line and destroy your readability. Here is how you can reduce the path of entire set of directories to just current directory.
               add PROMPT_DIRTRIM=1 to your .bashrc file.
   
   
   
   
2. you also need to install fish. As this is going to be amazing for your future self. Then fish_config to open browser based configuration of fish.
   
   
   
3. You need to find something in terminal for example a command that you ran but you do not know what it was. What you would normally do is press CTRL+r and keep on pressing them until you find it, right? Now you do not have to do this. But just press CTRL+r once and scroll through the commands with arrow keys using this
   
     git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf && ~/.fzf/install

With this today's lecture ends. Now remember Noonari, you will keep on receiving more of these for you to use in future. I will keep on typing that I can for you here. Stay sharp and stay hungry. Thats how you got this much of success in career.

How to increase swap size in Ubuntu? (works for all Ubuntu Flavors)

Here we will see how we can increase the SwapSize  in Ubuntu!

1. Open Terminal
2. Enter "sudo swapon --show"
1. This is show you the details of current swap size
3. Enter "df -h"
1. This is to see how much disk space you have.
4. Now Turn the swap off using "sudo swapoff -a"
1. This will allow us to write to swap file without being interrupted by File in Use error
5. This is the important part and requires some knowledge if you want to understand what is going on under the hood. But high level understanding here is as follows:
1. If have xyz GB of ram, your swap file size will be XYZ GB + 2GB at the minimum. In my case, I had 16GB of Ram on Laptop, but current swap file was of 2GB only. When created automatically using the command that follows, it made 27GB in SwapFile.
2. Enter "sudo dd if=/dev/zero of=/swapfile bs=5M count=5120 status=progress"
3. Wait for the entire process to end.
6. Now lets give this file root only permissions: "sudo chmod 600 /swapfile"
7. Now mark this file as swap space file: "sudo mkswap /swapfile"
8.  Now make this file permanent so that restart of system doesn't revert it to previous state: 
1. "echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab"
9. Now turn the swap back on using "sudo swapon /swapfile"
10. Use HTOP to see change in swap size.

What is difference between queryset and get_queryset?

Overriding queryset and get_queryset have the same effect. I would slightly favour setting get_queryset because it's more verbose.

When you set queryset, the queryset is created only once, when you start your server. On the other hand, the get_queryset method is called for every request.

That means that get_queryset is useful if you want to adjust the query dynamically. For example, you could return objects that belong to the current user:

class IndexView(generic.ListView):
    def get_queryset(self):
        """Returns Polls that belong to the current user"""
        return Poll.active.filter(user=self.request.user).order_by('-pub_date')[:5]

Another example where get_queryset is useful is when you want to filter based on a callable, for example, return today's polls:

class IndexView(generic.ListView):
    def get_queryset(self):
        """Returns Polls that were created today"""
        return Poll.active.filter(pub_date=date.today())

If you tried to do the same thing by setting queryset, then date.today() would only be called once, when the view was loaded, and the view would display incorrect results after a while.

class IndexView(generic.ListView):
    # don't do this!
    queryset = Poll.active.filter(pub_date=date.today())