Merge remote-tracking branch 'refs/remotes/origin/main' into dev/para…

…llel-sensor-search

# Conflicts:
#	flexmeasures/data/models/generic_assets.py
#	flexmeasures/utils/config_defaults.py

.github/PULL_REQUEST_TEMPLATE.md

@@ -2,6 +2,15 @@
 
 Summary of the changes introduced in this PR. Try to use bullet points as much as possible.
 
+- [ ] ...
+- [ ] Added changelog item in `documentation/changelog.rst`
+
+<!--
+Note regarding our changelog:
+- The 'New features' section targets API / CLI / UI users.
+- The 'Infrastructure / Support' section targets plugin developers and hosts.
+-->
+
 ## Look & Feel
 
 This section can contain example pictures for UI, Input/Output for CLI, Request / Response for API endpoint, etc.
@@ -15,7 +24,7 @@ it can be used to set some example data to be used in a new UI feature.
 
 ## Further Improvements
 
-Potential improvements to be done in the same PR or follow up Issues/Discussions/PRs.
+Potential improvements to be done in the same PR or follow-up Issues/Discussions/PRs.
 
 ## Related Items
 

.github/workflows/lint-and-test.yml

@@ -23,7 +23,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        py_version: [ "3.8", "3.9", "3.10", "3.11" ]
+        py_version: [ "3.8", "3.9", "3.10", "3.11", "3.12" ]
         include:
           - python-version: "3.9"
             coverage: yes

.pre-commit-config.yaml

@@ -1,11 +1,11 @@
 repos:
 -   repo: https://github.com/pycqa/flake8
-    rev: 6.0.0  # New version tags can be found here: https://github.com/pycqa/flake8/tags
+    rev: 7.1.1  # New version tags can be found here: https://github.com/pycqa/flake8/tags
     hooks:
     - id: flake8
       name: flake8 (code linting)
 -   repo: https://github.com/psf/black
-    rev: 22.10.0  # New version tags can be found here: https://github.com/psf/black/tags
+    rev: 24.8.0  # New version tags can be found here: https://github.com/psf/black/tags
     hooks:
     - id: black
       name: black (code formatting)

.readthedocs.yaml

@@ -8,8 +8,8 @@ sphinx:
   configuration: documentation/conf.py
 
 # Optionally build your docs in additional formats such as PDF
-formats:
-  - pdf
+#formats:
+#  - pdf  # stopped working (e.g. https://readthedocs.org/projects/flexmeasures/builds/26604114/, also not required)
 
 build:
   os: ubuntu-20.04
@@ -26,4 +26,4 @@ python:
 
 
 sphinx:
-  fail_on_warning: true
+  fail_on_warning: true

Makefile

@@ -29,8 +29,7 @@ update-docs:
 
 update-docs-pdf:
 	@echo "NOTE: PDF documentation requires packages (on Debian: latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended)"
-	@echo "NOTE: Currently, the docs require some pictures which are not in the git repo atm. Ask the devs."
-	make install-sphinx-tools
+	make install-docs-dependencies
 
 	export GEN_CODE_DOCS=${gen_code_docs}; cd documentation; make clean; make latexpdf; make latexpdf; cd ..  # make latexpdf can require two passes
 
@@ -43,26 +42,27 @@ install-for-dev:
 	make ensure-deps-folder
 	pip-sync requirements/${PYV}/app.txt requirements/${PYV}/dev.txt requirements/${PYV}/test.txt
 	make install-flexmeasures
+# Locally install HiGHS on macOS
+	@if [ "$(shell uname)" = "Darwin" ]; then \
+		make install-highs-macos; \
+	fi
 
 install-for-test:
 	make install-pip-tools
 # Pass pinned=no if you want to test against latest stable packages, default is our pinned dependency set
 ifneq ($(pinned), no)
 	pip-sync requirements/${PYV}/app.txt requirements/${PYV}/test.txt
 else
-	# cutting off the -c inter-layer dependency (that's pip-tools specific)
-	tail -n +3 requirements/test.in >> temp-test.in
-	pip install --upgrade -r requirements/app.in -r temp-test.in
-	rm temp-test.in
+	pip install --upgrade -r requirements/app.in -r requirements/test.in
 endif
 	make install-flexmeasures
-# Locally install HiGS on macOS
-	if [ "$(shell uname)" = "Darwin" ]; then \
+# Locally install HiGHS on macOS
+	@if [ "$(shell uname)" = "Darwin" ]; then \
 		make install-highs-macos; \
 	fi
 
 $(HIGHS_DIR):
-	if [ ! -d $(HIGHS_DIR) ]; then \
+	@if [ ! -d $(HIGHS_DIR) ]; then \
 		git clone https://github.com/ERGO-Code/HiGHS.git $(HIGHS_DIR); \
 	fi
 	brew install cmake;

NOTICE

@@ -1,3 +1,3 @@
 FlexMeasures
-Copyright 2018-2024 Seita Energy Flexibility
+Copyright 2018-2025 Seita Energy Flexibility
 Apache 2.0 license

ci/update-packages.sh

@@ -12,7 +12,7 @@
 set -e
 set -x
 
-PYTHON_VERSIONS=(3.8 3.9 3.10 3.11)
+PYTHON_VERSIONS=(3.8 3.9 3.10 3.11 3.12)
 
 # check if we will upgrade or just freeze
 UPDATE_CMD=freeze-deps

documentation/api/change_log.rst

@@ -5,7 +5,24 @@ API change log
 
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL (e.g. `/api/v3_0`), allowing developers to upgrade at their own pace.
 
-v3.0-20 | 2024-09-XX
+v3.0-22 | 2024-12-27
+""""""""""""""""""""
+
+- Allow using numeric values for ``flex-model`` fields accepting dimensionless quantities.
+
+v3.0-21 | 2024-12-16
+""""""""""""""""""""
+
+- Introduce new fields for defining capacity contracts and peak contracts in the ``flex-context``, used for scheduling against multiple contractual commitments simultaneously:
+
+  - ``site-consumption-breach-price``: if set, the ``site-consumption-capacity`` is used as a soft constraint, and breaching it is penalized according to this per-kW price. The price is applied both to the largest breach in the planning window and to each breach that occurs.
+  - ``site-production-breach-price``: if set, the ``site-production-capacity`` is used as a soft constraint, and breaching it is penalized according to this per-kW price. The price is applied both to the largest breach in the planning window and to each breach that occurs.
+  - ``site-peak-consumption-price``: consumption peaks above the ``site-peak-consumption`` are penalized against this per-kW price.
+  - ``site-peak-production-price``: production peaks above the ``site-peak-production`` are penalized against this per-kW price.
+  - ``site-peak-consumption``: current peak consumption; costs from peaks below it are considered sunk costs.
+  - ``site-peak-production``: current peak production; costs from peaks below it are considered sunk costs.
+
+v3.0-20 | 2024-09-18
 """"""""""""""""""""
 
 -  Introduce (optional) pagination to the endpoint `/assets` (GET), also adding the `all_accessible` option to allow querying all accessible accounts in one go.

documentation/api/notation.rst

@@ -47,7 +47,12 @@ Unless stated otherwise, values of such fields can take one of the following for
          ]
      }
 
-  Note the two distinct ways of specifying a time period (``"end"`` + ``"duration"`` also works).
+  Note the two distinct ways of specifying a time period (``"end"`` in combination with ``"duration"`` also works).
+
+  .. note:: In case a field defines partially overlapping time periods, FlexMeasures automatically resolves this.
+            By default, time periods that are defined earlier in the list take precedence.
+            Fields that deviate from this policy will note so explicitly.
+            (For example, for fields dealing with capacities, the minimum is selected instead.)
 
 - A reference to a sensor that records a variable quantity, which allows cross-referencing to dynamic contexts that are already recorded as sensor data in FlexMeasures. For instance, a site's contracted consumption capacity that changes over time.
 

documentation/changelog.rst

@@ -4,15 +4,80 @@ FlexMeasures Changelog
 **********************
 
 
-v0.23.0 | August XX, 2024
+v0.24.0 | October XX, 2024
 ============================
 
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
+
+New features
+-------------
+* Allow scheduling against energy contracts, capacity contracts and peak contracts simultaneously, using various new ``flex-context`` fields [see `PR #1144 <https://github.com/FlexMeasures/flexmeasures/pull/1144>`_]
+* Allow using numeric values for ``flex-model`` fields accepting dimensionless quantities [see `PR #1144 <https://github.com/FlexMeasures/flexmeasures/pull/1299>`_]
+* The data chart on the asset page splits up its color-coded sensor legend when showing more than 7 sensors, becoming a legend per subplot [see `PR #1176 <https://github.com/FlexMeasures/flexmeasures/pull/1176>`_ and `PR #1193 <https://github.com/FlexMeasures/flexmeasures/pull/1193>`_]
+* Speed up loading the users page, by making the pagination backend-based and adding support for that in the API [see `PR #1160 <https://github.com/FlexMeasures/flexmeasures/pull/1160>`_]
+* X-axis labels in CLI plots show datetime values in a readable and informative format [see `PR #1172 <https://github.com/FlexMeasures/flexmeasures/pull/1172>`_]
+* Enhanced API for listing sensors: Added filtering and pagination on sensor index endpoint and created new endpoint to get all sensors under an asset [see `PR #1191 <https://github.com/FlexMeasures/flexmeasures/pull/1191>`_ and `PR #1219 <https://github.com/FlexMeasures/flexmeasures/pull/1219>`_]
+* Speed up loading the accounts page by making the pagination backend-based and adding support for that in the API [see `PR #1196 <https://github.com/FlexMeasures/flexmeasures/pull/1196>`_]
+* Speed up loading the account detail page by by switching to server-side pagination for assets, replacing client-side pagination [see `PR #1202 <https://github.com/FlexMeasures/flexmeasures/pull/1202>`_]
+* Added form to edit an asset's sensors_to_show data directly from the UI [see `PR #1212 <https://github.com/FlexMeasures/flexmeasures/pull/1212>`_]
+* Speed up loading on audit log tables page by switching to server-side pagination, replacing client-side pagination [see `PR #1274 <https://github.com/FlexMeasures/flexmeasures/pull/1274>`_ and `PR #1272 <https://github.com/FlexMeasures/flexmeasures/pull/1272>`_]
+* Simplify and globalize UI messages, using Toast [see `PR #1207 <https://github.com/FlexMeasures/flexmeasures/pull/1207>`_]
+* Power sensors created through the CLI no longer require a capacity attribute to be set [see `PR #1234 <https://github.com/FlexMeasures/flexmeasures/pull/1234>`_]
+
+Infrastructure / Support
+----------------------
+* The breadcrumbs on asset and sensor pages can now be customized [see `PR #1257 <https://github.com/FlexMeasures/flexmeasures/pull/1257>`_]
+* The monitoring command to check for users who have been absent too long now can be used to keep data volume low and be more effective [see `PR #1268 <https://github.com/FlexMeasures/flexmeasures/pull/1268>`_]
+* Speed up status page by choosing for a faster query (only latest belief needed) [see `PR #1142 <https://github.com/FlexMeasures/flexmeasures/pull/1142>`_]
+* Speed up user page (for users with many assets) [see `PR #1286 <https://github.com/FlexMeasures/flexmeasures/pull/1286>`_]
+* For MacOS developers, install HiGHS solver automatically [see `PR #1187 <https://github.com/FlexMeasures/flexmeasures/pull/1187>`_]
+* Migrate data for the ``sensors_to_show`` asset attribute to a dedicated column in the database table for assets [see `PR #1200 <https://github.com/FlexMeasures/flexmeasures/pull/1200>`_ and `PR #1282 <https://github.com/FlexMeasures/flexmeasures/pull/1282>`_]
+* Add support for installing FlexMeasures under Python 3.12 [see `PR #1233 <https://github.com/FlexMeasures/flexmeasures/pull/1233>`_]
+* Better error handling in UI, for example, in case of a forgotten ``flexmeasures db upgrade`` [see `PR #1302 <https://github.com/FlexMeasures/flexmeasures/pull/1302>`_]
+
+Bugfixes
+-----------
+* Fix table sorting on the assets, accounts and users page (regression from `PR #988 <https://github.com/FlexMeasures/flexmeasures/pull/988>`_) [see `PR #1239 <https://github.com/FlexMeasures/flexmeasures/pull/1239>`_, `PR #1242 <https://github.com/FlexMeasures/flexmeasures/pull/1242>`_, `PR #1248 <https://github.com/FlexMeasures/flexmeasures/pull/1248>`_, `PR #1247 <https://github.com/FlexMeasures/flexmeasures/pull/1247>`_ and `PR #1272 <https://github.com/FlexMeasures/flexmeasures/pull/1272>`_]
+* Fix `flexmeasures delete measurements` and `flexmeasures delete prognoses` which had an error in counting affected rows after SQLAlchemy2.0 upgrade [see `PR #1095 <https://github.com/FlexMeasures/flexmeasures/pull/1095>`_ and `PR #1303 <https://github.com/FlexMeasures/flexmeasures/pull/1303>`_]
+
+* Fix asset count on the user page, which showed 0 assets after (de)activating a user or resetting their password [see `PR #1251 <https://github.com/FlexMeasures/flexmeasures/pull/1251>`_]
+* The UI footer now stays at the bottom even on pages with little content [see `PR #1204 <https://github.com/FlexMeasures/flexmeasures/pull/1204>`_]
+* Correct stroke dash (based on source type) for forecasts made by forecasters included in FlexMeasures [see `PR #1211 <https://www.github.com/FlexMeasures/flexmeasures/pull/1211>`_]
+* Show the correct :abbr:`UTC (Coordinated Universal Time)` offset for the data's time span as shown under sensor stats in the UI [see `PR #1213 <https://github.com/FlexMeasures/flexmeasures/pull/1213>`_]
+* Fixed issue where audit log buttons are visible to users without the necessary permissions. [see `PR #1228 <https://github.com/FlexMeasures/flexmeasures/pull/1228>`_]
+* Fix issue with displaying ``deactivate user`` and ``reset password`` buttons for non admin users [see `PR #1220 <https://github.com/FlexMeasures/flexmeasures/pull/1220>`_]
+* Resolve overlapping time series segments in several flex-model and flex-context fields in a way befitting their nature [see `PR #1223 <https://github.com/FlexMeasures/flexmeasures/pull/1223>`_]
+
+
+v0.23.2 | December 16, 2024
+============================
+
+Bugfixes
+-----------
+* Correct source filters on reporter input, and support piping additional transformations in the ``PandasReporter`` [see `PR #1284 <https://github.com/FlexMeasures/flexmeasures/pull/1284>`_]
+
+
+v0.23.1 | November 12, 2024
+============================
+
+Bugfixes
+-----------
+* Correct unit conversion of reporter output to output sensor [see `PR #1238 <https://github.com/FlexMeasures/flexmeasures/pull/1238>`_]
+
+
+v0.23.0 | September 18, 2024
+============================
+
+.. note:: Read more on these features on `the FlexMeasures blog <https://flexmeasures.io/023-data-insights-and-white-labelling/>`_.
+
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
+
 New features
 -------------
-* Added support for adding custom titles the graphs on the asset page. This works via an extension to the sensors_to_show format. [see `PR #1125 <https://github.com/FlexMeasures/flexmeasures/pull/1125>`_]
 * New chart type on sensor page: histogram [see `PR #1143 <https://github.com/FlexMeasures/flexmeasures/pull/1143>`_]
 * Add basic sensor info to sensor page [see `PR #1115 <https://github.com/FlexMeasures/flexmeasures/pull/1115>`_]
 * Add `Statistics` table on the sensor page and also add `api/v3_0/sensors/<id>/stats` endpoint to get sensor statistics [see `PR #1116 <https://github.com/FlexMeasures/flexmeasures/pull/1116>`_]
+* Support adding custom titles to the graphs on the asset page, by extending the ``sensors_to_show`` format [see `PR #1125 <https://github.com/FlexMeasures/flexmeasures/pull/1125>`_ and `PR #1177 <https://github.com/FlexMeasures/flexmeasures/pull/1177>`_]
 * Support zoom-in action on the asset and sensor charts [see `PR #1130 <https://github.com/FlexMeasures/flexmeasures/pull/1130>`_]
 * Speed up loading the assets page, by making the pagination backend-based and adding support for that in the API, and by enabling to query all accounts one can see in a single call (for admins and consultants) [see `PR #988 <https://github.com/FlexMeasures/flexmeasures/pull/988>`_]
 * Added Primary and Secondary colors to account for white-labelled UI themes [see `PR #1137 <https://github.com/FlexMeasures/flexmeasures/pull/1137>`_]
@@ -34,13 +99,17 @@ Infrastructure / Support
 
 Bugfixes
 -----------
+* Fix string length exceeding the 255-character limit in the `event` field of `AssetAuditLog` by truncating long updates and logging each field or attribute change individually. [see `PR #1162 <https://github.com/FlexMeasures/flexmeasures/pull/1162>`_]
 * Fix image carousel on the login page [see `PR #1154 <https://github.com/FlexMeasures/flexmeasures/pull/1154>`_]
 * Fix styling for User and Documentation menu items [see `PR #1140 <https://github.com/FlexMeasures/flexmeasures/pull/1140>`_]
 * Fix styling of sensor page, especially the graph chart dropdown [see `PR #1148 <https://github.com/FlexMeasures/flexmeasures/pull/1148>`_]
 * Fix posting a single instantaneous belief [see `PR #1129 <https://github.com/FlexMeasures/flexmeasures/pull/1129>`_]
 * Allow reassigning a public asset to private ownership using the ``flexmeasures edit transfer-ownership`` CLI command [see `PR #1123 <https://github.com/FlexMeasures/flexmeasures/pull/1123>`_]
 * Fix missing value on spring :abbr:`DST (Daylight Saving Time)` transition for ``PandasReporter`` using daily sensor as input [see `PR #1122 <https://github.com/FlexMeasures/flexmeasures/pull/1122>`_]
-* Fix issue with account creation failing when the --logo-url flag is omitted. [see related PRs `PR #1167 <https://github.com/FlexMeasures/flexmeasures/pull/1167>`_ and `PR #1145 <https://github.com/FlexMeasures/flexmeasures/pull/1145>`_]
+* Fix date range persistence on session across different pages [see `PR #1165 <https://github.com/FlexMeasures/flexmeasures/pull/1165>`_]
+* Fix issue with account creation failing when the ``--logo-url`` option is omitted. [see related PRs `PR #1167 <https://github.com/FlexMeasures/flexmeasures/pull/1167>`_ and `PR #1145 <https://github.com/FlexMeasures/flexmeasures/pull/1145>`_]
+* Fix ordering of audit logs (asset, account) and job list on status page [see `PR #1179 <https://github.com/FlexMeasures/flexmeasures/pull/1179>_` and `PR #1183 <https://github.com/FlexMeasures/flexmeasures/pull/1183>`_]
+
 
 v0.22.0 | June 29, 2024
 ============================

documentation/cli/change_log.rst

@@ -4,14 +4,18 @@
 FlexMeasures CLI Changelog
 **********************
 
-since v.0.22.0 | June 29, 2024
+since v0.24.0 | October XX, 2024
 =================================
 
+* ``flexmeasures show beliefs`` shows datetime values on x-axis labels.
+* ``flexmeasures add sensor`` no longer requires the ``capacity_in_mw`` attribute to be set for power sensors.
 
-* Add ``--resolution`` option to ``flexmeasures show chart`` to produce charts in different time resolutions.
+since v0.22.0 | June 29, 2024
+=================================
 
+* Add ``--resolution`` option to ``flexmeasures show chart`` to produce charts in different time resolutions.
 
-since v.0.21.0 | April 16, 2024
+since v0.21.0 | April 16, 2024
 =================================
 
 * Include started, deferred and scheduled jobs in the overview printed by the CLI command ``flexmeasures jobs show-queues``.

documentation/cli/commands.rst

@@ -92,6 +92,15 @@ of which some are referred to in this documentation.
 ================================================= =======================================
 
 
+``monitor`` - Monitoring
+--------------
+
+================================================= =======================================
+``flexmeasures monitor latest-run``               Check if the given task's last successful execution happened less than the allowed time ago.
+``flexmeasures monitor last-seen``                Check if given users last contact (via a request) happened less than the allowed time ago.
+================================================= =======================================
+
+
 ``jobs`` - Job queueing
 --------------