source: server/databases/upgradedb.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
   "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
  <!ENTITY % general-entities SYSTEM "../../general.ent">
  %general-entities;
]>

<sect1 id="upgradedb" xreflabel="Upgrade Database">
  <?dbhtml filename="upgradedb.html"?>


  <title>Important Notes About Upgrading Database Server Software</title>

    <note><para>This section is about reinstalling database software
    when an existing database is in use.  It is not applicable for
    initial installations or if there is no existing database for
    the package being updated, but users should read through it
    to become aware of issues that can arise in the future.</para></note>

    <para>Let's start this chapter with a dramatic screenshot of an error that
    really happened.  This error will not occur if you are installing database
    software for the first time:</para>

<screen>$ sudo systemctl status postgresql
-- postgresql.service - PostgreSQL database server
     Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2021-10-26 17:11:53 CDT; 2min 49s ago
    Process: 17336 ExecStart=/usr/bin/pg_ctl -s -D ${PGROOT}/data start -w -t 120 (code=exited, status=1/FAILURE)
        CPU: 7ms

Oct 26 17:11:53 SVRNAME systemd[1]: Starting PostgreSQL database server...
Oct 26 17:11:53 SRVNAME postgres[17338]: 2021-10-26 17:11:53.420 CDT [17338] FATAL:
                database files are incompatible with server
Oct 26 17:11:53 SRVNAME postgres[17338]: 2021-10-26 17:11:53.420 CDT [17338] DETAIL:
                The data directory was initialized by PostgreSQL version 13,
                which is not compatible with this version 14.0.
Oct 26 17:11:53 SRVNAME postgres[17336]: pg_ctl: could not start server
Oct 26 17:11:53 SRVNAME postgres[17336]: Examine the log output.
Oct 26 17:11:53 SRVNAME systemd[1]: postgresql.service: Control process exited, code=exited, status=1/FAILURE
Oct 26 17:11:53 SRVNAME systemd[1]: postgresql.service: Failed with result 'exit-code'.
Oct 26 17:11:53 SRVNAME systemd[1]: Failed to start PostgreSQL database server.</screen>

  <para>
    To avoid situations like this (i.e., your database server
    software refuses to start), read the following discussion of the best way to
    upgrade a DBMS (Database Management System).
  </para>

  <para>
    The root cause of the error shown above was an upgrade
    of the server software to a newer major version which left the
    data files untouched. In this case, the administrator was able to recover
    the DBMS without any loss of data.
  </para>

  <para>
    Even if you are doing an initial DBMS install, read through this
    section. It provides information about implementing backup
    and restore procedures (or at least a strategy for creating
    them) which will satisfy your needs and guarantee the safety
    of your data.
  </para>

  <sect2>
    <title>Upgrade Database Server Packages</title>

    <para>
      Database systems work on files which hold the database metadata and the
      data itself. The internal structure of these files is optimized for use
      by the server software. When such server software is upgraded,
      the new software may utilize a different file format than
      had previously been used. Sometimes the new software can
      work with the old format as well as the new one&mdash;but without the
      performance improvements the new format provides.
      Other times, the new server software will
      reformat the data files automatically after the upgrade.
    </para>

    <para>
      Unfortunately, the most likely case is that the new server software
      complains about out of date file formats and exits.  When this happens,
      and you have overwritten the old server software, you may end up
      with a broken system and lost data.
    </para>

    <para>
       Changes in data file formats usually happen at major version changes, but
       they can also occur at other times.  Before upgrading any DBMS
       software, check the documentation to see if this upgrade makes changes which
       require reformatting the database.
    </para>

    <para>
      Of course, if you have databases with content that is not easily
      rebuilt, it is always a good idea to create backups of the database from
      time to time. Before upgrading the server software, you should run
      another backup.
    </para>

    <sect3>
      <title>Upgrade by Backup and Restore</title>

      <note>
        <para>
          A backup is useless if there is no verified process
          to restore the data from this backup. When running a
          database server, you should not only create backups; you
          should also verify that the restore process
          really works. The time to test the restore procedure
          is <emphasis>before</emphasis> you urgently need
          to recover lost data.
        </para>
      </note>

      <para>
        Most database server software provides some basic
        tools to create backups of your data. Usually the backups created with
        those tools can be read by newer versions of the software (via a
        restore tool). Using older restore tools with newer backup data is
        a bad idea; you should <emphasis>never</emphasis> blindly assume that
        it will work. It might, but usually it doesn't.
      </para>

      <para>
        The easiest way to upgrade your database files is to
      </para>

      <itemizedlist>
        <listitem>
          <para>Create a full database backup using the old tools.</para>

          <para>This step creates an offline copy of the database files&mdash;for
          long term archiving, for disaster recovery, or as
          preparation for an upgrade. This offline backup consists of either (1) a full
          one-to-one copy of the current database files, or (2) a full backup of the
          database files from a certain point in time, plus all the journal data (that is
          Oracle&reg; terminology, it is called "Continuous Archiving" or
          "write ahead log (WAL)" in Postgresql) describing the
          changes made after that point in time. This second form takes less time to create
          (if the DB software provides this type of journaling) because you only have
          to save the data that have changed since the last full backup was created.</para>

          <para>When upgrading database server software, a full backup
          (which can be used for subsequent incremental backups) should be
          created; but if there is a lot of data, an incremental backup will
          suffice. The best strategy for you depends on
          the amount of data stored in your database (is it a few hundred table
          rows, or is it hundreds of terabytes?). A full backup in the latter case
          can't be done quickly.  To
          fully protect your data, create a backup of the old
          programs (and/or their sources) and save it, along with the data
          files, to be certain there is a fallback solution if
          the new software cannot read the old data.</para>
          </listitem>

          <listitem>
            <para>Upgrade the server software</para>

            <para>In this step, instructions to build the database server
            software are executed just as they are shown in subsequent sections
            talking about the DBMs like MariaDB or Postgresql. That is, build
            the software as usual using BLFS instructions.</para>
          </listitem>

          <listitem>
            <para>Restore the database by using the new tools.</para>

            <para>To restore the data, the tools of the newly installed server
            software should be used. During the restoration process, the new
            tools will create and/or upgrade the data files in the format the
            new software requires. It is assumed that newer software is capable of
            reading old data.</para>
          </listitem>
          </itemizedlist>

      <para>
        Since you already have a backup procedure in place (and you
        have tested your restore procedure, right?), this might
        be the easiest way to upgrade as you can use your well known
        processes to upgrade just as you always do&mdash;at least in terms
        of the backup and restore.
      </para>

    </sect3>

    <sect3>
      <title>Upgrade the Database Files by Using System Tools</title>

      <para>
        Some database systems (for instance Postgresql) provide
        a tool which can reformat (upgrade) the existing database
        files to the new format.
        If you need to restore from a backup (for example, running
        the upgrade tool failed) you will have to reinstall the old software
        to recover your data.
      </para>

      <para>
        Even though the reformatting tools might work as advertised,
        you should create a full backup before running them. A failure
        could cause serious damage to the database.
      </para>

    </sect3>

  </sect2>

  <sect2>
    <title>Notes for Specific DBMS</title>

    <sect3>
      <title>PostgreSQL</title>

      <!-- might add some advanced notes for the DBMS here -->

      <para>Upstream documentation for Backup/Restore:
        <ulink url="https://www.postgresql.org/docs/current/backup.html"/>
      </para>

    </sect3>

    <sect3>
      <title>MariaDB</title>

      <!-- might add some advanced notes for the DBMS here -->

      <para>Upstream documentation for Backup/Restore:
        <ulink url="https://mariadb.com/kb/en/backup-and-restore-overview/"/>
      </para>

    </sect3>

    <sect3>
      <title>Sqlite</title>

      <!-- might add some advanced notes for the DBMS here -->

      <para>Do not underestimate <application>Sqlite</application>. It is a
      feature-rich DBMS. The main difference from the two big players above is
      that Sqlite does not provide access via a network API. Sqlite databases
      are always stored on the machine running the program which
      uses the database. The manipulation of data content is done via API calls
      to library functions directly within the program.</para>

      <para>In the upstream documentation you may find the following
      useful:</para>

      <para>Documentation of the sqlite3 command line tool:
        <ulink url="https://www.sqlite.org/cli.html"/>
      </para>

      <para>Documentation of backup API calls:
        <ulink url="https://www.sqlite.org/backup.html"/>
      </para>

      <para>Unfortunately, there is no dedicated chapter in the
      upstream documentation talking about backup/restore, but
      there are several articles about it on the
      Internet. Here is an example.</para>

      <para>Documentation for Backup/Restore:
        <ulink url="https://database.guide/backup-sqlite-database/"/>
      </para>

    </sect3>

    <sect3>
      <title>LMDB</title>

      <!-- might add some advanced notes for the DBMS here -->

      <para>Like <application>Sqlite</application>, this
      software acts on local database files; there is no
      network interface.</para>

      <para>The relevant resources to back up/restore a LMDB database
      are the man pages for <filename>mdb_dump</filename> and its
      counterpart <filename>mdb_load</filename>.</para>

    </sect3>
  </sect2>

</sect1>