add docs about the updater

3 files changed, 160 insertions, 0 deletions

diff --git a/doc/administration/installation.rst b/doc/administration/installation.rst
index c21af59..cf8e83a 100644
--- a/doc/administration/installation.rst
+++ b/doc/administration/installation.rst
@@ -113,6 +113,14 @@ other update tasks. You can use it to set up the initial database schema:
 
     .venv/bin/fietsupdate update -c production.ini
 
+.. note::
+
+    If you install Fietsboek via Git, and you stick to the ``master`` branch
+    instead of a specific version, you must also run ``.venv/bin/alembic -c
+    production.ini upgrade head``.
+
+    See :doc:`../developer/updater` ("Furhter notes") for more information.
+
 Adding an Administrator User
 ----------------------------
 
diff --git a/doc/developer.rst b/doc/developer.rst
index 02876dc..bcd27e5 100644
--- a/doc/developer.rst
+++ b/doc/developer.rst
@@ -9,6 +9,7 @@ Developer Guide
     developer/language-pack
     developer/js-css
     developer/authentication
+    developer/updater
     Python Packages <developer/module/modules>
 
 This guide contains information for developers that want to modify or extend
diff --git a/doc/developer/updater.rst b/doc/developer/updater.rst
new file mode 100644
index 0000000..f8fa59f
--- /dev/null
+++ b/doc/developer/updater.rst
@@ -0,0 +1,151 @@
+The Updater
+===========
+
+The process of updating for administrators is described in
+:doc:`../administration/upgrading`. Here, we will describe the developer side
+of the update process.
+
+Motivation
+----------
+
+Some updates do not only change the code of Fietsboek, but adapt the data that
+is stored. New features for example might require new columns in the database,
+or other adaptions might move data around (commit 7a60619d_ for example moves
+the GPX data out of the database and into the data directory).
+
+.. _7a60619d: https://gitlab.com/dunj3/fietsboek/-/commit/7a60619d3f6fd523d42f50753436f3b7e7d72ca4
+
+This process is commonly called *data migration*. In Fietsboek, we use Alembic_
+to do the heavy lifting of SQL migrations. Alembic can automatically create
+migration scripts and provides an abstraction over the different database
+backends.
+
+.. _Alembic: https://alembic.sqlalchemy.org/en/latest/
+
+With the image uploads, and more concretely with commit 5a205756_, Fietsboek
+has been extended to store data on disk. This has resulted in `issue 20`_,
+which raises the question about how to deal with update scripts now: Alembic
+handles the SQL updates well, but now we have a second place to run migrations
+for.
+
+.. _5a205756: https://gitlab.com/dunj3/fietsboek/-/commit/5a2057560a5703a59408009e40b51de5a0c18600
+.. _issue 20: https://gitlab.com/dunj3/fietsboek/-/issues/20
+
+This has resulted in the ``fietsupdate`` tool, which implements a similar logic
+as Alembic, but for general purpose Python code instead of SQL. It also acts as
+the user interface for administrators, providing some abstractions over the
+Alembic CLI.
+
+Update scripts
+--------------
+
+An update script contains the logic that ``fietsupdate`` applies when going
+from one version to the next. Such a script looks like the this:
+
+.. code:: python
+
+    """Revision upgrade script v0.8.0
+
+    Date created: 2023-06-05 21:00:37.464583
+    """
+    from fietsboek.updater.script import UpdateScript
+
+    update_id = 'v0.8.0'
+    previous = [
+        'v0.7.0',
+    ]
+    alembic_revision = '3149aa2d0114'
+
+
+    class Up(UpdateScript):
+        def pre_alembic(self, config):
+            pass
+
+        def post_alembic(self, config):
+            pass
+
+
+    class Down(UpdateScript):
+        def pre_alembic(self, config):
+            pass
+
+        def post_alembic(self, config):
+            pass
+
+* Each script imports the :class:`~fietsboek.updater.script.UpdateScript` class
+  as a base.
+* Each update has a unique ID, which is used on the command line to identify
+  the correct script. For most scripts, this is a randomly generated string,
+  but some specific revisions have a more readable ID.
+* Each update (except for the initial one) has references to its previous
+  versions. As such, the commits form a DAG. This allows ``fietsupdate`` to
+  check which updates have been applied, and which still need to be applied.
+* Each update has a reference to the underlying alembic version. When
+  ``fietsupdate`` applies the update, it will make sure that the database will
+  have the given alembic version after.
+* Finally, each update has the actual update logic, in four sections:
+  Migrations that are run *before* the alembic update and *after* the alembic
+  update, each for the upgrade and the downgrade direction.
+
+Creating Alembic versions
+-------------------------
+
+There is nothing special about how we use Alembic to generate SQL revisions. As
+such, we refer to the `Alembic documentation`__.
+
+.. __: https://alembic.sqlalchemy.org/en/latest/cookbook.html#create-revision-migrations
+
+In general, a command like this should work::
+
+    alembic -c development.ini revision --autogenerate
+
+Creating Fietsupdate versions
+-----------------------------
+
+The ``fietsupdate`` has a hidden command to generate a revision from the
+template, and it will populate it with the current alembic revision::
+
+    fietsupdate revision -c development.ini
+
+When to create revisions
+------------------------
+
+If you change Fietsboek code or documentation that does not change the way that
+data is stored, you do *not* need to bother with Alembic or Fietsupdate.
+
+If you change the database schema, or the way that the data is stored in the
+existing schema, you need to create an Alembic revision.
+
+If you change the data that is stored in the data directory, you need to create
+a Fietsupdate script.
+
+If a new Fietsboek version is released, a new Fietsupdate migration is created
+with the version as revision ID, to ensure that ``fietsupdate update v0.1.2``
+will work. This script is usually empty, and only contains a reference to the
+relevant previous migration ID and the correct alembic revision.
+
+Further notes
+-------------
+
+When running Fietsboek from ``master``, it might happen that the repository is
+in a state in which Alembic revisions exist, but no Fietsupdate revision that
+refers to them. In this case, you must run
+
+::
+
+    alembic -c development.ini upgrade head
+
+Before creating revisions, make sure that your repository and database are in
+the correct state. Otherwise your revision might refer to an outdated revision.
+In particular, you should ensure that you have applied all previous updates.
+
+While both Alembic and Fietsupdate have support for "branches" and "merges", it
+is still preferable to use this feature as little as possible. If possible,
+rebase the changes to give them a linear order.
+
+We cannot guarantee that migration scripts will always continue to work (e.g.
+if they depend on functionality which is later removed from Fietsboek,
+dependencies that are removed, ...). For this reason, big update jumps *might*
+be hard to support. However, all scripts should at least stay loadable (that
+means their metadata is readable, there are no syntax errors), and for empty
+databases they should work (to allow bootstrapping of fresh instances).