Merge with upstream, fix test for YMD path provider

.codecov.yml

@@ -8,7 +8,7 @@ coverage:
         threshold: 1%
     patch:
       default:
-        target: auto
+        target: 90
         threshold: 1%
 github_checks:
   annotations: false

.github/workflows/_release.yml

@@ -23,7 +23,7 @@ jobs:
       - name: Create GitHub Release
         # We pin to the SHA, not the tag, for security reasons.
         # https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions
-        uses: softprops/action-gh-release@9d7c94cfd0a1f3ed45544c887983e9fa900f0564 # v2.0.4
+        uses: softprops/action-gh-release@69320dbe05506a9a39fc8ae11030b214ec2d1f87 # v2.0.5
         with:
           prerelease: ${{ contains(github.ref_name, 'a') || contains(github.ref_name, 'b') || contains(github.ref_name, 'rc') }}
           files: "*"

docs/explanations/design-goals.rst

@@ -32,7 +32,7 @@ Parity with Malcolm
 
 Ophyd-async should provide the same building blocks for defining flyscans scans as malcolm_. It should support PandA and Zebra as timing masters by default, but also provide easy helpers for developers to write support for their own devices.
 
-It should enable `motor trajectory scanning <motortraj_>` and `multiple triggering rates<detectorsync_>` based around a base rate, and pausing/resuming scans. Scans should be modelled using scanspec_, which serves as a universal language for defining trajectory and time-resolved scans, and converted to the underlying format of the given motion controller. It should also be possible to define an `outer scan <outerscan_>`.
+It should enable motor trajectory scanning and multiple triggering rates based around a base rate, and pausing/resuming scans. Scans should be modelled using scanspec_, which serves as a universal language for defining trajectory and time-resolved scans, and converted to the underlying format of the given motion controller. It should also be possible to define an outer scan .
 
 
 Improved Trajectory Calculation

docs/explanations/flyscanning.rst

@@ -1,41 +1,7 @@
 Flyscanning
 ===========
 
-Flyscanning (also known as hardware triggered scanning, asynchronous acquisition, and hardware synchronized scanning) is the practice of accelerating the rate of data collection by handing control over to an external hardware system that can control and synchronize the triggering of detectors with other signals and/or commands. Flyscans take many forms.
-
-.. _detectorsync_:
-
-Detector Synchronization
-------------------------
-
-.. figure:: ../images/simple-hardware-scan.png
-    :alt: hardware-triggered setup
-    :width: 300
-
-A triggering system can send pulses to two or more detectors to make them expose simultaneously, or at different multiples of the same base rate (e.g. 200Hz and 400Hz).
-
-.. _motortraj_:
-
-Motor Trajectory Scanning
--------------------------
-
-.. figure:: ../images/hardware-triggered-scan.png
-    :alt: trajectory scanning setup
-
-The triggering system can be configured to trigger the detectors at the same time as the motion controller commands the motors to go to certain points, or even exactly when they reach those points, using the readback values. This can be achieved on the scale of microseconds/nanoseconds, in comparison to traditional soft scans controlled via a network, which normally synchronize on the scale of seconds.
-
-.. _outerscan_:
-
-Outer Scanning
---------------
-
-Outer scans are flyscans nested inside soft scans. 
-
-.. figure:: ../images/outer-scan.png
-    :alt: hardware-triggered setup
-
-In the example above a 2D grid scan in ``x`` and ``y`` is repeated in a third dimension: ``z``. Given that ``z`` only needs to move for every 1 in every 25 points, it could be synchronized via software rather than hardware without significantly affecting scan time (and saving the effort/expense of wiring it into a triggering system). It then becomes the responsibility of the software to move ``z``, hand control to the external hardware, wait for one grid's worth of points, take control back, and repeat. 
-
+See the documents in the [bluesky cookbook](http://blueskyproject.io/bluesky-cookbook/glossary/flyscanning.html)
 
 Hardware
 --------

src/ophyd_async/core/detector.py

@@ -332,7 +332,7 @@ async def collect_asset_docs(
         # Collect stream datum documents for all indices written.
         # The index is optional, and provided for fly scans, however this needs to be
         # retrieved for step scans.
-        if not index:
+        if index is None:
             index = await self.writer.get_indices_written()
         async for doc in self.writer.collect_stream_docs(index):
             yield doc

src/ophyd_async/core/flyer.py

@@ -39,7 +39,7 @@ class HardwareTriggeredFlyable(
     def __init__(
         self,
         trigger_logic: TriggerLogic[T],
-        configuration_signals: Sequence[SignalR],
+        configuration_signals: Sequence[SignalR] = (),
         name: str = "",
     ):
         self._trigger_logic = trigger_logic

src/ophyd_async/epics/areadetector/writers/hdf_writer.py

@@ -60,7 +60,7 @@ async def open(self, multiplier: int = 1) -> Dict[str, DataKey]:
 
         assert (
             await self.hdf.file_path_exists.get_value()
-        ), f"File path {self.hdf.file_path.get_value()} for hdf plugin does not exist"
+        ), f"File path {info.root / info.resource_dir} for hdf plugin does not exist"
 
         # Overwrite num_capture to go forever
         await self.hdf.num_capture.set(0)

src/ophyd_async/panda/writers/_hdf_writer.py

@@ -151,7 +151,7 @@ async def open(self, multiplier: int = 1) -> Dict[str, DataKey]:
                 else split_path[-2]
             )
 
-            for suffix in str(capture_signal.capture_type).split(" "):
+            for suffix in capture_signal.capture_type.split(" "):
                 self._datasets.append(
                     _HDFDataset(
                         name,

src/ophyd_async/planstubs/__init__.py → src/ophyd_async/plan_stubs/__init__.py

@@ -1,9 +1,11 @@
 from .ensure_connected import ensure_connected
-from .prepare_trigger_and_dets import (
+from .fly import (
     prepare_static_seq_table_flyer_and_detectors_with_same_trigger,
+    time_resolved_fly_and_collect_with_static_seq_table,
 )
 
 __all__ = [
+    "time_resolved_fly_and_collect_with_static_seq_table",
     "prepare_static_seq_table_flyer_and_detectors_with_same_trigger",
     "ensure_connected",
 ]

src/ophyd_async/plan_stubs/fly.py

@@ -0,0 +1,130 @@
+from typing import List
+
+import bluesky.plan_stubs as bps
+from bluesky.utils import short_uid
+
+from ophyd_async.core.detector import DetectorTrigger, StandardDetector, TriggerInfo
+from ophyd_async.core.flyer import HardwareTriggeredFlyable
+from ophyd_async.core.utils import in_micros
+from ophyd_async.panda._table import SeqTable, SeqTableRow, seq_table_from_rows
+from ophyd_async.panda._trigger import SeqTableInfo
+
+
+def time_resolved_fly_and_collect_with_static_seq_table(
+    stream_name: str,
+    detectors: List[StandardDetector],
+    flyer: HardwareTriggeredFlyable[SeqTableInfo],
+    number_of_frames: int,
+    exposure: int,
+    shutter_time: float,
+    repeats: int = 1,
+    period: float = 0.0,
+):
+    """Run a scan wth a flyer and multiple detectors.
+
+    The standard basic flow for a flyscan:
+
+    - Set up the flyer with a static sequence table and detectors with a trigger
+    - Declare the stream and kickoff the scan
+    - Collect while completing
+
+    This needs to be used in a plan that instantates detectors and a flyer,
+    stages/unstages the devices, and opens and closes the run.
+
+    """
+    # Set up scan and prepare trigger
+    deadtime = max(det.controller.get_deadtime(exposure) for det in detectors)
+    yield from prepare_static_seq_table_flyer_and_detectors_with_same_trigger(
+        flyer,
+        detectors,
+        number_of_frames=number_of_frames,
+        exposure=exposure,
+        deadtime=deadtime,
+        shutter_time=shutter_time,
+        repeats=repeats,
+        period=period,
+    )
+    yield from bps.declare_stream(*detectors, name=stream_name, collect=True)
+    yield from bps.kickoff(flyer, wait=True)
+    for detector in detectors:
+        yield from bps.kickoff(detector)
+
+    # collect_while_completing
+    group = short_uid(label="complete")
+
+    yield from bps.complete(flyer, wait=False, group=group)
+    for detector in detectors:
+        yield from bps.complete(detector, wait=False, group=group)
+
+    done = False
+    while not done:
+        try:
+            yield from bps.wait(group=group, timeout=0.5)
+        except TimeoutError:
+            pass
+        else:
+            done = True
+        yield from bps.collect(
+            *detectors,
+            return_payload=False,
+            name=stream_name,
+        )
+    yield from bps.wait(group=group)
+
+
+def prepare_static_seq_table_flyer_and_detectors_with_same_trigger(
+    flyer: HardwareTriggeredFlyable[SeqTableInfo],
+    detectors: List[StandardDetector],
+    number_of_frames: int,
+    exposure: float,
+    deadtime: float,
+    shutter_time: float,
+    repeats: int = 1,
+    period: float = 0.0,
+):
+    """Prepare a hardware triggered flyable and one or more detectors.
+
+    Prepare a hardware triggered flyable and one or more detectors with the
+    same trigger. This method constructs TriggerInfo and a static sequence
+    table from required parameters. The table is required to prepare the flyer,
+    and the TriggerInfo is required to prepare the detector(s).
+
+    This prepares all supplied detectors with the same trigger.
+
+    """
+    trigger_info = TriggerInfo(
+        num=number_of_frames * repeats,
+        trigger=DetectorTrigger.constant_gate,
+        deadtime=deadtime,
+        livetime=exposure,
+    )
+
+    trigger_time = number_of_frames * (exposure + deadtime)
+    pre_delay = max(period - 2 * shutter_time - trigger_time, 0)
+
+    table: SeqTable = seq_table_from_rows(
+        # Wait for pre-delay then open shutter
+        SeqTableRow(
+            time1=in_micros(pre_delay),
+            time2=in_micros(shutter_time),
+            outa2=True,
+        ),
+        # Keeping shutter open, do N triggers
+        SeqTableRow(
+            repeats=number_of_frames,
+            time1=in_micros(exposure),
+            outa1=True,
+            outb1=True,
+            time2=in_micros(deadtime),
+            outa2=True,
+        ),
+        # Add the shutter close
+        SeqTableRow(time2=in_micros(shutter_time)),
+    )
+
+    table_info = SeqTableInfo(table, repeats)
+
+    for det in detectors:
+        yield from bps.prepare(det, trigger_info, wait=False, group="prep")
+    yield from bps.prepare(flyer, table_info, wait=False, group="prep")
+    yield from bps.wait(group="prep")

tests/core/test_device.py

@@ -15,9 +15,7 @@
 )
 from ophyd_async.core.soft_signal_backend import SoftSignalBackend
 from ophyd_async.epics.motion import motor
-from ophyd_async.planstubs.ensure_connected import (
-    ensure_connected,
-)
+from ophyd_async.plan_stubs.ensure_connected import ensure_connected
 from ophyd_async.sim.demo.sim_motor import SimMotor