diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/administration/backup.rst | 12 | ||||
| -rw-r--r-- | doc/administration/configuration.rst | 26 | ||||
| -rw-r--r-- | doc/administration/cronjobs.rst | 8 | ||||
| -rw-r--r-- | doc/administration/custom-pages.rst | 4 | ||||
| -rw-r--r-- | doc/administration/installation.rst | 46 | ||||
| -rw-r--r-- | doc/administration/maintenance-mode.rst | 14 | ||||
| -rw-r--r-- | doc/administration/upgrading.rst | 15 | ||||
| -rw-r--r-- | doc/developer.rst | 1 | ||||
| -rw-r--r-- | doc/developer/updater.rst | 151 | ||||
| -rw-r--r-- | doc/index.rst | 2 | ||||
| -rw-r--r-- | doc/man/fietsctl.rst | 2 |
11 files changed, 243 insertions, 38 deletions
diff --git a/doc/administration/backup.rst b/doc/administration/backup.rst index fb379bc..0643728 100644 --- a/doc/administration/backup.rst +++ b/doc/administration/backup.rst @@ -15,13 +15,19 @@ case data is lost due unforseen reasons. Backup ------ -The Fietsboek data lives in two places: Most of it is saved in the database, -while track images and other "big files" are saved in the data directory. +The Fietsboek data lives in two places: Track and account metadata is saved in +the database, while GPX files, images and other "big files" are saved in the +data directory. You should refer to the manual of your DBMS to see what the procedure for a backup is. Using the preferred way for your DBMS ensures that your backup represents a consistent state of the database. +.. note:: + + Check :doc:`maintenance-mode` to see how the maintenance mode can be + enabled, which should disable most database accesses during the process. + After backing up the database, you should back up the data directory with a tool of your choice, for example by using ``tar``, ``rsync`` or a proper backup tool like ``borg``: @@ -82,7 +88,7 @@ specific version: .. code:: bash - .venv/bin/pip install "fietsboek==vX.Y.Z" + .venv/bin/pip install "fietsboek==X.Y.Z" Next, we run the data migrations: diff --git a/doc/administration/configuration.rst b/doc/administration/configuration.rst index 4d33b7f..24c5fdb 100644 --- a/doc/administration/configuration.rst +++ b/doc/administration/configuration.rst @@ -2,7 +2,7 @@ Configuration ============= The main configuration of Fietsboek is done via ``.ini``-files. By default, -three such files exist: +three such files exist, and one will be loaded at a time: * ``production.ini`` contains the configuration for the production environment. It turns off debugging features (as they are a security risk!) and should @@ -93,8 +93,8 @@ Database Settings Fietsboek uses three different databases: A SQL database for persistent data (like user accounts), a file storage on the -disk for big files (like images), and a redis server for ephemeral data (like -cached tiles). +disk for big files (like GPX files and images), and a redis server for +ephemeral data (like cached tiles). Set ``sqlalchemy.url`` to the URL of the SQL database. See the `SQLAlchemy documentation @@ -103,8 +103,8 @@ information on available URL formats. Make sure to install the driver necessary to communicate with your database (e.g. ``psycopg2`` for PostreSQL)! Set ``fietsboek.data_dir`` to the directory for data uploads. This directory -must be writable by the Fietsboek process, as Fietsboek will save track images -in there. +must be writable by the Fietsboek process, as Fietsboek will save track data in +there. Set ``redis.url`` to the URL of the redis instance. See the `redis module documentation @@ -133,7 +133,7 @@ accepted: standard output. Should not be used in production, as no emails would ever arrive. * ``smtp://host:port`` use the given SMTP server (without transport encryption!) -* ``smtp+ssl://host:port`` use the given SMTP server over a SSL connection +* ``smtp+ssl://host:port`` use the given SMTP server over a SSL connection. * ``smtp+starttls://host:port`` use the given SMTP server and the STARTTLS command to start an encrypted channel. @@ -192,11 +192,11 @@ You can add custom tile layers in the following way: # public fietsboek.tile_layer.ID.access = public -``ID`` must be an alphanumerical identifier. +``ID`` must be an alphanumerical identifier of your choosing. By default, Fietsboek will proxy all tile requests through the Fietsboek -instance. While this can slow down the user experience, it has the following -benefits: +instance. While this can slow down the user experience and increase the load on +your server, it has the following benefits: * Your users' IPs stay private and protected, as no third party is contacted. The tile servers will only see the IP from the Fietsboek server. @@ -239,15 +239,15 @@ default behavior. .. note:: - The ``hittekaart.autogenerate`` has no effect on the ``fietsctl + The ``hittekaart.autogenerate`` setting has no effect on the ``fietsctl hittekaart`` command. You can always use ``fietsctl`` to generate heat maps - for specific users! + manually! .. warning:: Depending on the geospatial area that a user covers with their tracks, an - overlay map can get relatively large (~100 MiB for mine). Keep that in mind - when hosting a larger number of users. + overlay map can get relatively large (in my case ~100 MiB). Keep that in + mind when hosting a larger number of users. An example configuration excerpt can look like this: diff --git a/doc/administration/cronjobs.rst b/doc/administration/cronjobs.rst index a7d1a95..c2939c8 100644 --- a/doc/administration/cronjobs.rst +++ b/doc/administration/cronjobs.rst @@ -9,14 +9,20 @@ keep the instance running smoothly: * Tracks that are not present in the cache should have their cache entry populated by pre-calculating their length/uphill/downhill/… values. This makes accessing those values faster when a user requests the track's details. +* If enabled, heatmaps/tilehunts should be generated. Fietsboek comes with a script to deal with those jobs: ``fietscron``. You can run it like this:: fietscron -c path_to_config.ini +The script is designed to be "smart", meaning that it splits the long tasks +(such as heatmap generation) over multiple runs, and it should work without +additional configuration. + You can use any mechanism you like to run this command at pre-defined intervals -(e.g. once an hour). You can find information about this in various sources: +(e.g. once an hour). You can find information about this in various sources, +depending on your system setup: * https://wiki.archlinux.org/title/Systemd/Timers * https://wiki.archlinux.org/title/Cron diff --git a/doc/administration/custom-pages.rst b/doc/administration/custom-pages.rst index db92c49..fbc3121 100644 --- a/doc/administration/custom-pages.rst +++ b/doc/administration/custom-pages.rst @@ -8,8 +8,8 @@ information or similar things on a website. Or you simply want to add some information about your instance of Fietsboek, links to other sites, ... Such pages can not be provided by Fietsboek out of the box, as they are very -customized to the particular installation. However, Fietsboek provides a way to -include custom static pages. Those pages have the benefit of being embedded in +tailored to the particular site. However, Fietsboek provides a way to include +custom static pages. Those pages have the benefit of being embedded in Fietsboek's menu and layout, so they don't look out of place and can easily be found. diff --git a/doc/administration/installation.rst b/doc/administration/installation.rst index fd0def5..cf8e83a 100644 --- a/doc/administration/installation.rst +++ b/doc/administration/installation.rst @@ -50,17 +50,41 @@ Installing the Python Modules ----------------------------- The Python package manager takes care of installing all dependencies, all you -need to do is tell it to install Fietsboek: +need to do is tell it to install Fietsboek. How exactly depends on your way of +acquiring Fietsboek. + +Via PIP +^^^^^^^ + +Fietsboek is available in the Python Package Index as `fietsboek +<https://pypi.org/project/fietsboek/>`__ and can be installed directly via +``pip``: .. code:: bash - # Assuming that we are in the Fietsboek folder - .venv/bin/pip install . - # Alternatively - .venv/bin/pip path/to/fietsboek/repository + .venv/bin/pip install fietsboek + +Via Git +^^^^^^^ + +If you prefer to work with the source version (e.g. for development purposes), +you should use Git to clone the repository from one of its mirrors: -Optionally, you can install ``lxml``, which provides a faster XML parser to -process the GPX files: +* Main on Gitlab: https://gitlab.com/dunj3/fietsboek +* Mirror on Codeberg: https://codeberg.org/dunj3/fietsboek + +.. code:: bash + + git clone https://gitlab.com/dunj3/fietsboek + .venv/bin/pip install ./fietsboek + # For development, you can use -e for an editable installation: + .venv/bin/pip install -e ./fietsboek + +Optional: Install lxml +^^^^^^^^^^^^^^^^^^^^^^ + +In any case, you can optionally install ``lxml``, which provides a faster XML +parser to process the GPX files: .. code:: bash @@ -89,6 +113,14 @@ other update tasks. You can use it to set up the initial database schema: .venv/bin/fietsupdate update -c production.ini +.. note:: + + If you install Fietsboek via Git, and you stick to the ``master`` branch + instead of a specific version, you must also run ``.venv/bin/alembic -c + production.ini upgrade head``. + + See :doc:`../developer/updater` ("Furhter notes") for more information. + Adding an Administrator User ---------------------------- diff --git a/doc/administration/maintenance-mode.rst b/doc/administration/maintenance-mode.rst index d02e27a..30a273c 100644 --- a/doc/administration/maintenance-mode.rst +++ b/doc/administration/maintenance-mode.rst @@ -7,7 +7,7 @@ with the data. Deactivating your instance avoids any "race issues" or accesses to the database and such, which might be in inconsistent states. To help with this situation, fietsboek provides a *maintenance mode*. This -allows you to deactivate fietsboek without having to fiddle with your webserver +allows you to deactivate fietsboek without having to change with your webserver settings. In the maintenance mode, fietsboek will reply to any HTTP request with a ``503 @@ -16,8 +16,8 @@ Service Unavailable`` response, and won't process the requests further. .. note:: The maintenance mode is implemented early in fietsboek, however if you have - custom HTTP proxies or other WSGI middleware between fietsboek and the - outside world, the maintenance mode will not affect those. + custom HTTP proxies, caches or other WSGI middleware between fietsboek and + the outside world, the maintenance mode will not affect those. In those cases, it might be better to deactivate the whole pipeline, depending on the setup and the middleware used — at least if those also @@ -25,17 +25,17 @@ Service Unavailable`` response, and won't process the requests further. In addition to not replying to HTTP requests, ``fietscron`` will also respect the maintenance mode and do nothing. This is to avoid accidental interference -with long tasks like updating, such that a cronjob doesn't access data in an -invalid state. +with long tasks like updating, such that a badly-timed cronjob doesn't access +data in an invalid state. -Unlike ``fietscron`` and the HTTP requests, ``fietsctl`` and ``fietsupdate`` +Unlike ``fietscron`` and the web interface, ``fietsctl`` and ``fietsupdate`` will **continue to work**. Controlling the Maintenance Mode -------------------------------- You can enable and disable the maintenance mode using the ``fietsctl`` script. -To enable the mode, simply pass a reason (that will be shown to users) to +To enable maintenance, simply pass a reason (that will be shown to users) to ``fietsctl maintenance-mode``:: fietsctl -c production.ini maintenance-mode "Updating the instance" diff --git a/doc/administration/upgrading.rst b/doc/administration/upgrading.rst index 2b69e4c..c5e899d 100644 --- a/doc/administration/upgrading.rst +++ b/doc/administration/upgrading.rst @@ -44,10 +44,17 @@ new additional dependencies: .. code:: bash - .venv/bin/pip install . + .venv/bin/pip install path/to/fietsboek/source -This step is similar to the command that you used when installing Fietsboek -initially. +If you downloaded Fietsboek from PyPI, you may need to tell ``pip`` explicitely +to upgrade. Otherwise it will simply say that the package is already installed: + +.. code:: bash + + .venv/bin/pip install --upgrade fietsboek + +In general, this step is similar to the command that you used when installing +Fietsboek initially. Note that removed dependencies are not automatically removed. It might be worth to start with a clean virtual environment from time to time to clear out old @@ -120,7 +127,7 @@ Restoring Data: Downgrading If you do not have old backups of your data, you can use ``fietsupdate`` to convert your data from the new format to the old one. Please note that the upgrade process can not always be undone: If an update deletes data, even the -downgrade cannot fix it. In such cases, the only way to restore the data is +downgrade cannot undo it. In such cases, the only way to restore the data is through a backup. To use ``fietsupdate`` to downgrade your data, run the following command, diff --git a/doc/developer.rst b/doc/developer.rst index 02876dc..bcd27e5 100644 --- a/doc/developer.rst +++ b/doc/developer.rst @@ -9,6 +9,7 @@ Developer Guide developer/language-pack developer/js-css developer/authentication + developer/updater Python Packages <developer/module/modules> This guide contains information for developers that want to modify or extend diff --git a/doc/developer/updater.rst b/doc/developer/updater.rst new file mode 100644 index 0000000..f8fa59f --- /dev/null +++ b/doc/developer/updater.rst @@ -0,0 +1,151 @@ +The Updater +=========== + +The process of updating for administrators is described in +:doc:`../administration/upgrading`. Here, we will describe the developer side +of the update process. + +Motivation +---------- + +Some updates do not only change the code of Fietsboek, but adapt the data that +is stored. New features for example might require new columns in the database, +or other adaptions might move data around (commit 7a60619d_ for example moves +the GPX data out of the database and into the data directory). + +.. _7a60619d: https://gitlab.com/dunj3/fietsboek/-/commit/7a60619d3f6fd523d42f50753436f3b7e7d72ca4 + +This process is commonly called *data migration*. In Fietsboek, we use Alembic_ +to do the heavy lifting of SQL migrations. Alembic can automatically create +migration scripts and provides an abstraction over the different database +backends. + +.. _Alembic: https://alembic.sqlalchemy.org/en/latest/ + +With the image uploads, and more concretely with commit 5a205756_, Fietsboek +has been extended to store data on disk. This has resulted in `issue 20`_, +which raises the question about how to deal with update scripts now: Alembic +handles the SQL updates well, but now we have a second place to run migrations +for. + +.. _5a205756: https://gitlab.com/dunj3/fietsboek/-/commit/5a2057560a5703a59408009e40b51de5a0c18600 +.. _issue 20: https://gitlab.com/dunj3/fietsboek/-/issues/20 + +This has resulted in the ``fietsupdate`` tool, which implements a similar logic +as Alembic, but for general purpose Python code instead of SQL. It also acts as +the user interface for administrators, providing some abstractions over the +Alembic CLI. + +Update scripts +-------------- + +An update script contains the logic that ``fietsupdate`` applies when going +from one version to the next. Such a script looks like the this: + +.. code:: python + + """Revision upgrade script v0.8.0 + + Date created: 2023-06-05 21:00:37.464583 + """ + from fietsboek.updater.script import UpdateScript + + update_id = 'v0.8.0' + previous = [ + 'v0.7.0', + ] + alembic_revision = '3149aa2d0114' + + + class Up(UpdateScript): + def pre_alembic(self, config): + pass + + def post_alembic(self, config): + pass + + + class Down(UpdateScript): + def pre_alembic(self, config): + pass + + def post_alembic(self, config): + pass + +* Each script imports the :class:`~fietsboek.updater.script.UpdateScript` class + as a base. +* Each update has a unique ID, which is used on the command line to identify + the correct script. For most scripts, this is a randomly generated string, + but some specific revisions have a more readable ID. +* Each update (except for the initial one) has references to its previous + versions. As such, the commits form a DAG. This allows ``fietsupdate`` to + check which updates have been applied, and which still need to be applied. +* Each update has a reference to the underlying alembic version. When + ``fietsupdate`` applies the update, it will make sure that the database will + have the given alembic version after. +* Finally, each update has the actual update logic, in four sections: + Migrations that are run *before* the alembic update and *after* the alembic + update, each for the upgrade and the downgrade direction. + +Creating Alembic versions +------------------------- + +There is nothing special about how we use Alembic to generate SQL revisions. As +such, we refer to the `Alembic documentation`__. + +.. __: https://alembic.sqlalchemy.org/en/latest/cookbook.html#create-revision-migrations + +In general, a command like this should work:: + + alembic -c development.ini revision --autogenerate + +Creating Fietsupdate versions +----------------------------- + +The ``fietsupdate`` has a hidden command to generate a revision from the +template, and it will populate it with the current alembic revision:: + + fietsupdate revision -c development.ini + +When to create revisions +------------------------ + +If you change Fietsboek code or documentation that does not change the way that +data is stored, you do *not* need to bother with Alembic or Fietsupdate. + +If you change the database schema, or the way that the data is stored in the +existing schema, you need to create an Alembic revision. + +If you change the data that is stored in the data directory, you need to create +a Fietsupdate script. + +If a new Fietsboek version is released, a new Fietsupdate migration is created +with the version as revision ID, to ensure that ``fietsupdate update v0.1.2`` +will work. This script is usually empty, and only contains a reference to the +relevant previous migration ID and the correct alembic revision. + +Further notes +------------- + +When running Fietsboek from ``master``, it might happen that the repository is +in a state in which Alembic revisions exist, but no Fietsupdate revision that +refers to them. In this case, you must run + +:: + + alembic -c development.ini upgrade head + +Before creating revisions, make sure that your repository and database are in +the correct state. Otherwise your revision might refer to an outdated revision. +In particular, you should ensure that you have applied all previous updates. + +While both Alembic and Fietsupdate have support for "branches" and "merges", it +is still preferable to use this feature as little as possible. If possible, +rebase the changes to give them a linear order. + +We cannot guarantee that migration scripts will always continue to work (e.g. +if they depend on functionality which is later removed from Fietsboek, +dependencies that are removed, ...). For this reason, big update jumps *might* +be hard to support. However, all scripts should at least stay loadable (that +means their metadata is readable, there are no syntax errors), and for empty +databases they should work (to allow bootstrapping of fresh instances). diff --git a/doc/index.rst b/doc/index.rst index 73dce3f..e3c765b 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -19,6 +19,8 @@ Welcome to Fietsboek's documentation! :maxdepth: 1 changelog + Website <https://fietsboek.org> + Repository <https://gitlab.com/dunj3/fietsboek> Fietsboek is a self-hostable sharing site for GPX track recordings with social features. The goal is to have an application like `MyTourbook diff --git a/doc/man/fietsctl.rst b/doc/man/fietsctl.rst index 5319a42..1d30b20 100644 --- a/doc/man/fietsctl.rst +++ b/doc/man/fietsctl.rst @@ -24,7 +24,7 @@ to manage the state and content of a Fietsboek instance from the command line. .. warning:: The ``fietsctl`` script does not do any access checking and does not - require and logins or passwords. You must use the permissions of your + require any logins or passwords. You must use the permissions of your system and database server to restrict the access to ``fietsctl`` by restricting access to the data stores directly. |