The Updater
===========

The process of updating for administrators is described in
:doc:`../administration/upgrading`. Here, we will describe the developer side
of the update process.

Motivation
----------

Some updates do not only change the code of Fietsboek, but adapt the data that
is stored. New features for example might require new columns in the database,
or other adaptions might move data around (commit 7a60619d_ for example moves
the GPX data out of the database and into the data directory).

.. _7a60619d: https://gitlab.com/dunj3/fietsboek/-/commit/7a60619d3f6fd523d42f50753436f3b7e7d72ca4

This process is commonly called *data migration*. In Fietsboek, we use Alembic_
to do the heavy lifting of SQL migrations. Alembic can automatically create
migration scripts and provides an abstraction over the different database
backends.

.. _Alembic: https://alembic.sqlalchemy.org/en/latest/

With the image uploads, and more concretely with commit 5a205756_, Fietsboek
has been extended to store data on disk. This has resulted in `issue 20`_,
which raises the question about how to deal with update scripts now: Alembic
handles the SQL updates well, but now we have a second place to run migrations
for.

.. _5a205756: https://gitlab.com/dunj3/fietsboek/-/commit/5a2057560a5703a59408009e40b51de5a0c18600
.. _issue 20: https://gitlab.com/dunj3/fietsboek/-/issues/20

This has resulted in the ``fietsupdate`` tool, which implements a similar logic
as Alembic, but for general purpose Python code instead of SQL. It also acts as
the user interface for administrators, providing some abstractions over the
Alembic CLI.

Update scripts
--------------

An update script contains the logic that ``fietsupdate`` applies when going
from one version to the next. Such a script looks like the this:

.. code:: python

    """Revision upgrade script v0.8.0

    Date created: 2023-06-05 21:00:37.464583
    """
    from fietsboek.updater.script import UpdateScript

    update_id = 'v0.8.0'
    previous = [
        'v0.7.0',
    ]
    alembic_revision = '3149aa2d0114'


    class Up(UpdateScript):
        def pre_alembic(self, config):
            pass

        def post_alembic(self, config):
            pass


    class Down(UpdateScript):
        def pre_alembic(self, config):
            pass

        def post_alembic(self, config):
            pass

* Each script imports the :class:`~fietsboek.updater.script.UpdateScript` class
  as a base.
* Each update has a unique ID, which is used on the command line to identify
  the correct script. For most scripts, this is a randomly generated string,
  but some specific revisions have a more readable ID.
* Each update (except for the initial one) has references to its previous
  versions. As such, the commits form a DAG. This allows ``fietsupdate`` to
  check which updates have been applied, and which still need to be applied.
* Each update has a reference to the underlying alembic version. When
  ``fietsupdate`` applies the update, it will make sure that the database will
  have the given alembic version after.
* Finally, each update has the actual update logic, in four sections:
  Migrations that are run *before* the alembic update and *after* the alembic
  update, each for the upgrade and the downgrade direction.

Creating Alembic versions
-------------------------

There is nothing special about how we use Alembic to generate SQL revisions. As
such, we refer to the `Alembic documentation`__.

.. __: https://alembic.sqlalchemy.org/en/latest/cookbook.html#create-revision-migrations

In general, a command like this should work::

    alembic -c development.ini revision --autogenerate

Creating Fietsupdate versions
-----------------------------

The ``fietsupdate`` has a hidden command to generate a revision from the
template, and it will populate it with the current alembic revision::

    fietsupdate revision -c development.ini

When to create revisions
------------------------

If you change Fietsboek code or documentation that does not change the way that
data is stored, you do *not* need to bother with Alembic or Fietsupdate.

If you change the database schema, or the way that the data is stored in the
existing schema, you need to create an Alembic revision.

If you change the data that is stored in the data directory, you need to create
a Fietsupdate script.

If a new Fietsboek version is released, a new Fietsupdate migration is created
with the version as revision ID, to ensure that ``fietsupdate update v0.1.2``
will work. This script is usually empty, and only contains a reference to the
relevant previous migration ID and the correct alembic revision.

Further notes
-------------

When running Fietsboek from ``master``, it might happen that the repository is
in a state in which Alembic revisions exist, but no Fietsupdate revision that
refers to them. In this case, you must run

::

    alembic -c development.ini upgrade head

Before creating revisions, make sure that your repository and database are in
the correct state. Otherwise your revision might refer to an outdated revision.
In particular, you should ensure that you have applied all previous updates.

While both Alembic and Fietsupdate have support for "branches" and "merges", it
is still preferable to use this feature as little as possible. If possible,
rebase the changes to give them a linear order.

We cannot guarantee that migration scripts will always continue to work (e.g.
if they depend on functionality which is later removed from Fietsboek,
dependencies that are removed, ...). For this reason, big update jumps *might*
be hard to support. However, all scripts should at least stay loadable (that
means their metadata is readable, there are no syntax errors), and for empty
databases they should work (to allow bootstrapping of fresh instances).