Does SteppablePipeline.Clean() have be invoked from a compiled cmdlet for clean{} to run reliably?

[alx9r]

Links

• SteppablePipeline.Clean Method

Summary

The remarks for SteppablePipeline.Clean() state to invoke .Clean() from the clean{} block of a PowerShell function. Those remarks do not state when to invoke .Clean() from a compiled Cmdlet if at all. Because there is not really a Cmdlet method that corresponds to clean{}, it's not obvious when to invoke .Clean() from a compiled Cmdlet.

FWIW, I would call this an omission rather than a "bug".

Details

Below is a pattern that often arises when authoring commands involving user script blocks and .Net objects where implementation directly from PowerShell is difficult.

using System;
using System.Management.Automation;
using System.Collections.Generic;

[Cmdlet(VerbsLifecycle.Invoke, "Steppable")]
public sealed class InvokeSteppableCommand : PSCmdlet, IDisposable
{
    private SteppablePipeline steppable;

    [Parameter(ValueFromPipeline = true)]
    public object InputObject { get; set; }

    [Parameter(Mandatory = true, Position = 0)]
    public ScriptBlock ScriptBlock { get; set; }

    [Parameter()]
    public List<string> Log {get;set;}
    protected override void BeginProcessing()
    {
        steppable = ScriptBlock
            .Create(string.Format("param($__sb) . $__sb"))
            .GetSteppablePipeline(
                CommandOrigin.Internal,
                new object[] { ScriptBlock });

        steppable.Begin(this);
    }

    protected override void ProcessRecord(){ steppable.Process(InputObject); }
    protected override void EndProcessing() { steppable.End(); }
    public void Dispose() {
        Log.Add("Dispose()");
        steppable.Dispose();
    }
}

This cmdlet, for example, synthesizes a pipeline stopping cancellation token uses this pattern. I use this pattern to implement Use-Transaction (i.e. for sqlite) and Use-Object in a manner that is reliable despite the complications described in #24679, # 23786, and #24658.

The use of SteppablePipeline is the only technique I know of that results in idiomatic control of flow when invoking a user script block from a compiled Cmdlet. Specifically, the code

$log = [System.Collections.Generic.List[string]]::new()

1..2 |
    . {
        begin   { $log.Add('upstream begin{}'      )}
        process { $log.Add("upstream process{} $_" )}
        end     { $log.Add('upstream end{}'        )}
        clean   { $log.Add('upstream clean{}'      )}
    } |
    Invoke-Steppable -Log $log {
        begin   { $log.Add('command begin{}'      )}
        process { $log.Add("command process{} $_" )}
        end     { $log.Add('command end{}'        )}
        clean   { $log.Add('command clean{}'      )}
    } |
    . {
        begin   { $log.Add('downstream begin{}'      )}
        process { $log.Add("downstream process{} $_" )}
        end     { $log.Add('downstream end{}'        )}
        clean   { $log.Add('downstream clean{}'      )}
    }

$log

traces nearly identical flow of control as when Invoke-Steppable is replaced by . as shown in the following table:

Invoke-Steppable	dot-source .
upstream begin{}	upstream begin{}
command begin{}	command begin{}
downstream begin{}	downstream begin{}
upstream process{} 1	upstream process{} 1
upstream process{} 2	upstream process{} 2
upstream end{}	upstream end{}
command end{}	command end{}
command clean{}
downstream end{}	downstream end{}
upstream clean{}	upstream clean{}
	command clean{}
downstream clean{}	downstream clean{}
Dispose()

Only the clean{} block is called at a different point. (I can imagine some scenarios where that might have consequences, but those scenarios all seem avoidable.)

I did not expect clean{} to be invoked at all by Invoke-Steppable because the documentation for SteppablePipeline.Clean() suggests that .Clean() would have to be invoked in order for clean{} to run:

The way we handle 'Clean' blocks in a steppable pipeline makes sure that: 1. The 'Clean' blocks get to run if any exception is thrown from 'Begin/Process/End'. 2. The 'Clean' blocks get to run if 'End' finished successfully. However, this is not enough for a steppable pipeline, because the function, where the steppable pipeline gets used, may fail (think about a proxy function). And that may lead to the situation where "no exception was thrown from the steppable pipeline" but "the steppable pipeline didn't run to the end". In that case, 'Clean' won't run unless it's triggered explicitly on the steppable pipeline. This method allows a user to do that from the 'Clean' block of the proxy function.

Based on these remarks I expected to have to call .Clean() in order for clean{} to run.

For commands written in PowerShell the remarks state that SteppablePipeline.Clean() should be called from the clean{} block. Compiled Cmdlets, however, do not have a method that obviously corresponds to clean{}, and it's not obvious to me whether calling .Clean() is necessary for compiled Cmdlets.

I have tried to summarize the contenders for where .Clean() could be called from a compiled Cmdlet in the following table:

Method	Synchronous	Notes
EndProcessing()	yes	Doesn't run on upstream exception.1.
StopProcessing()	no	Only runs when stopping the pipeline from outside the runspace.
Dispose()		This refers to IDisposable.Dispose() of a Cmdlet that implements IDisposable (like Invoke-Steppable above).`

Notes

^1. An exception thrown upstream from the a compiled Cmdlet prevents the EndProcessing() from being called. This is the same control of flow when a script block is substituted for the Cmdlet. That is shown in "repro 1" below.

repro 1

$VerbosePreference = 'Continue'
1..2 |
    . {
        process {
            $_
            if ($_ -eq 2) { throw}
        }
    } |
    . {
        begin   { Write-Verbose "begin"      }
        process { Write-Verbose "process $_" }
        end     { Write-Verbose "end"        }
    }

outputs

VERBOSE: begin
VERBOSE: process 1
VERBOSE: process 2
Exception: C:\repro.ps1:6
Line |
   6 |              if ($_ -eq 2) { throw}
     |                              ~~~~~
     | ScriptHalted

Note that end{} never runs.

The table above really leaves only Dispose() as a viable contender for calling clean{}. But it seems like clean{} is already called when SteppablePipeline.Dispose() is called.

Is calling .Clean() required for clean{} to run reliably?

Suggested Fix

Once an answer to the underlying question is determined, I think it should be added to the SteppablePipeline.Clean() method's remarks section.

[daxian-dbw]

The way we handle 'Clean' blocks in a steppable pipeline makes sure that:

1. The 'Clean' blocks get to run if any exception is thrown from 'Begin/Process/End' of the command that is hosted in the steppable pipeline.
2. The 'Clean' blocks get to run if 'End' of the command that is hosted in the steppable pipeline finished successfully.

Hope the bold part makes it's easier to understand :)
So, for a proxy function, even if it doesn't call .Clean() in its clean { } block, the clean block of the proxied function will run in these two conditions, as shown by your Invoke-Steppable command.

However, this is not enough for a steppable pipeline, because the function, where the steppable pipeline gets used, may fail (think about a proxy function). And that may lead to the situation where "no exception was thrown from the steppable pipeline" but "the steppable pipeline didn't run to the end". In that case, 'Clean' won't run unless it's triggered explicitly on the steppable pipeline. This method allows a user to do that from the 'Clean' block of the proxy function.

But it's possible that the proxy function fails, say, before calling steppable.End(). So, for the steppable pipeline, it runs begin and process, but never runs end. In that case, the clean block of the proxied command will not run. For example,

public sealed class InvokeSteppableCommand : PSCmdlet, IDisposable
{
    ...

    protected override void ProcessRecord(){ steppable.Process(InputObject); }
    protected override void EndProcessing() { throw new WhateverException(); steppable.End(); }

    ...
}

The cmdlet will fail when throwing out the WhateverException, and thus the steppable.End() is not called, and thus the clean block of the ScriptBlock will not run. To ensure the clean block of ScriptBlock also gets to run in this case, for a proxy function, you will want to call steppable.clean() in the proxy function's clean { } block; for a cmdlet, you may want to call steppable.clean() in the finally block that captures the WhateverException, or maybe call it in your Dispose method.

Yes, it could cause the clean block of the proxied command to run twice, but that should be fine in general (like Dispose may be called twice when a type destructor is defined).

---

Only the clean{} block is called at a different point. (I can imagine some scenarios where that might have consequences, but those scenarios all seem avoidable.)

The clean block is handled at the pipeline level. For the steppable pipeline, it's done when .End() finishes, so the clean block of the script block hosted in the steppable pipeline will run at that point. For the outer pipeline 1..2 | ... | ... | ..., the clean block of each command in the pipeline will be called in order when the pipeline finishes.

[alx9r]

Thank you very much for the informative reply @daxian-dbw. I understand this much better now.

protected override void EndProcessing() { throw new WhateverException(); steppable.End(); }

The cmdlet will fail when throwing out the WhateverException, and thus the steppable.End() is not called... for a cmdlet, you may want to call steppable.clean() in the finally block that captures the WhateverException....

Calling .Clean() only from EndProcessing() will still result in clean{} not running for all the cases where EndProcessing() is never called. Here is one such example:

(I'm using this updated implementation of Invoke-Steppable which traces method calls for the following example.)

using System;
using System.Management.Automation;
using System.Collections.Generic;

[Cmdlet(VerbsLifecycle.Invoke, "Steppable")]
public sealed class InvokeSteppableCommand : PSCmdlet, IDisposable
{
    private SteppablePipeline steppable;

    [Parameter(ValueFromPipeline = true)]
    public object InputObject { get; set; }

    [Parameter(Mandatory = true, Position = 0)]
    public ScriptBlock ScriptBlock { get; set; }

    [Parameter()]
    public List<string> Log {get;set;}
    protected override void BeginProcessing() {
        Log.Add("BeginProcessing()");
        steppable = ScriptBlock
            .Create(string.Format("param($__sb) . $__sb"))
            .GetSteppablePipeline(
                CommandOrigin.Internal,
                new object[] { ScriptBlock });

        steppable.Begin(this);
    }

    protected override void ProcessRecord(){
        Log.Add("ProcessRecord()");
        steppable.Process(InputObject);
    }
    protected override void EndProcessing() {
        Log.Add("EndProcessing()");
        steppable.End();
    }
    public void Dispose() {
        Log.Add("Dispose()");
        steppable.Dispose();
    }
}

$log = [System.Collections.Generic.List[string]]::new()

try {
    1..2 |
        . {
            process {
                if ($_ -eq 2) { throw 'something'}
                $_
            }
        } |
        Invoke-Steppable -Log $log {
            begin   { $log.Add('command begin{}'      )     }
            process { $log.Add("command process{} $_" ); $_ }
            end     { $log.Add('command end{}'        )     }
            clean   { $log.Add('command clean{}'      )     }
        }
}
catch {}
$log

That outputs

1
BeginProcessing()
command begin{}
ProcessRecord(1)
command process{} 1
Dispose()

which shows that EndProcessing() does not run nor do the user script block's end{} or clean{} blocks. That happens despite that SteppablePipeline.Dispose() is called.

...or maybe call [steppable.Clean()] in your Dispose method.

I think that is the better choice here. Adding steppable.Clean() to InvokeSteppableCommand.Dispose() changes the output from the above scenario to

1
BeginProcessing()
command begin{}
ProcessRecord(1)
command process{} 1
Dispose()
command clean{}

Removing the throw from the example results in the following:

1
2
BeginProcessing()
command begin{}
ProcessRecord(1)
command process{} 1
ProcessRecord(2)
command process{} 2
EndProcessing()
command end{}
command clean{}
Dispose()

For what it's worth I ended up tracing a wide range of pipeline scenarios looking for equivalence class partitions. The test fixture is a bit involved, but I've put the test report here for reference.

test report

The Traces table below shows traces of the blocks and methods invoked in various pipeline scenarios. The following code is representative of the test setup for the prototypical "Normal Upstream" pipeline shown in the table:

1..2 |
    # upstream
    . {
        begin   { $log.Add("upstream begin{}")         }
        process { $log.Add("upstream process{$_}"); $_ }
        end     { $log.Add("upstream end{}")           }
        clean   { $log.Add("upstream clean{}")         }
    } |
    Invoke-Steppable -Log $log {
        begin   { $log.Add('command begin{}'     )    }
        process { $log.Add("command process{$_}"); $_ }
        end     { $log.Add('command end{}'       )    }
        clean   { $log.Add('command clean{}'     )    }
    } |
    . {
        begin   { $log.Add("downstream begin{}")          }
        process { $log.Add("downstream process{$_}"); $_ }
        end     { $log.Add("downstream end{}")            }
        clean   { $log.Add("downstream clean{}")          }
    }
    # downstream

Each column in the Traces table is a trace of a distinct pipeline scenario. Upstream and downstream in the column name refers to a command added to the prototypical pipeline at the # upstream and # downstream positions, respectively. "Empty Upstream" replaces the prototypical 1..2 with @(). throw indicates an exception thrown and begin{}, process{} and end{} refer to the blocks from which the exception is thrown. "None" omits all script blocks except Invoke-Steppable and leaves only that command. Wherever an exception is thrown from a process{} block with input, it is thrown such that 1 but not 2 makes it through the pipeline. "Halted" refers to the command Select-Object -First 1 included in the pipeline.

Traces table

None	Empty Upstream	Normal Upstream	throw downstream begin{}	throw downstream process{}	throw downstream end{}	throw upstream begin{}	throw upstream process{}	throw upstream end{}	Halted Upstream	Halted Downstream
	upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}		upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}
BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()		BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()
command begin{}	command begin{}	command begin{}	command begin{}	command begin{}	command begin{}		command begin{}	command begin{}	command begin{}	command begin{}
downstream begin{}	downstream begin{}	downstream begin{}	downstream begin{}	downstream begin{}	downstream begin{}		downstream begin{}	downstream begin{}	downstream begin{}	downstream begin{}
		upstream process{1}		upstream process{1}	upstream process{1}		upstream process{1}	upstream process{1}	upstream process{1}	upstream process{1}
ProcessRecord()		ProcessRecord(1)		ProcessRecord(1)	ProcessRecord(1)		ProcessRecord(1)	ProcessRecord(1)	ProcessRecord(1)	ProcessRecord(1)
command process{}		command process{1}		command process{1}	command process{1}		command process{1}	command process{1}	command process{1}	command process{1}
downstream process{}		downstream process{1}		downstream process{1}	downstream process{1}		downstream process{1}	downstream process{1}	downstream process{1}	downstream process{1}
		upstream process{2}			upstream process{2}			upstream process{2}
		ProcessRecord(2)			ProcessRecord(2)			ProcessRecord(2)
		command process{2}			command process{2}			command process{2}
		downstream process{2}			downstream process{2}			downstream process{2}
	upstream end{}	upstream end{}			upstream end{}				upstream end{}
EndProcessing()	EndProcessing()	EndProcessing()			EndProcessing()				EndProcessing()
command end{}	command end{}	command end{}			command end{}				command end{}
command clean{}	command clean{}	command clean{}		command clean{}	command clean{}				command clean{}	command clean{}
downstream end{}	downstream end{}	downstream end{}			downstream end{}				downstream end{}
	upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}		upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}
downstream clean{}	downstream clean{}	downstream clean{}	downstream clean{}	downstream clean{}	downstream clean{}		downstream clean{}	downstream clean{}	downstream clean{}	downstream clean{}
Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()
			command clean{}				command clean{}	command clean{}

With that report it's easy to see, for example, that EndProcessing() does not run for the following pipeline scenarios:

• exceptions thrown downstream in begin{}, process{}, but not end{}
• any exception thrown upstream
• Select-Object -First downstream but not upstream

Many of the results in that test report are counterintuitve to me. But it all looks manageable for my goals.

[daxian-dbw]

That is very thorough testing! I think there is one more case that EndProcessing doesn't run, which is when ctrl+c is pressed by user. In such a case, the StopProcessing method of a binary cmdlet will be called, and then the pipeline will be teared down. But I guess having steppable.Clear() in the Dispose method should cover that too.

which shows that EndProcessing() does not run nor do the user script block's end{} or clean{} blocks. That happens despite that SteppablePipeline.Dispose() is called.

This is an interesting point. Maybe we should explore and see if SteppablePipeline.Clean() can be called within SteppablePipeline.Dispose(), if it's not called already.

[alx9r]

That is very thorough testing!

Thank you @daxian-dbw. FWIW, the fixture isn't nearly as complicated as I thought it would be. In hindsight I should have done that long ago.

I think there is one more case that EndProcessing doesn't run, which is when ctrl+c is pressed by user.

Right. I added those scenarios to the test fixture. I'm using .StopJobAsync() rather than Ctrl+C because PowerShell/PowerShell#23786 interferes with cleanup behavior which makes test results very confusing.

Stop test report

None	stop upstream begin{}	stop upstream process{}	stop upstream end{}	stop downstream begin{}	stop downstream process{}	stop downstream end{}
		upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}
BeginProcessing()		BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()
command begin{}		command begin{}	command begin{}	command begin{}	command begin{}	command begin{}
downstream begin{}		downstream begin{}	downstream begin{}	downstream begin{}	downstream begin{}	downstream begin{}
		upstream process{1}	upstream process{1}		upstream process{1}	upstream process{1}
ProcessRecord()		ProcessRecord(1)	ProcessRecord(1)		ProcessRecord(1)	ProcessRecord(1)
command process{}		command process{1}	command process{1}		command process{1}	command process{1}
downstream process{}		downstream process{1}	downstream process{1}		downstream process{1}	downstream process{1}
			upstream process{2}			upstream process{2}
			ProcessRecord(2)			ProcessRecord(2)
			command process{2}			command process{2}
			downstream process{2}			downstream process{2}
						upstream end{}
EndProcessing()						EndProcessing()
command end{}						command end{}
command clean{}					command clean{}	command clean{}
downstream end{}						downstream end{}
		upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}
downstream clean{}		downstream clean{}	downstream clean{}	downstream clean{}	downstream clean{}	downstream clean{}
Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()

In such a case, the StopProcessing method of a binary cmdlet will be called, and then the pipeline will be teared down. But I guess having steppable.Clear() in the Dispose method should cover that too.

I was expecting this to work this way too. But calling steppable.Clean() after StopProcessing() throws PipelineStoppedException when StopProcess() is received at the following points:

• upstream process{}
• upstream end{}
• downstream begin{}

The clean{} block of the user script block doesn't run either in those scenarios which, I think, isn't the expected behavior. That unexpected behavior can be inferred from the Stop test report by looking for the missing "command clean{}" traces. The upstream and downstream clean{} run in all cases which suggests this is something specific to SteppablePipeline or compiled Cmdlets.

[daxian-dbw]

The clean{} block of the user script block should run in those scenarios, just like it runs when Stop happens in "downstream process{}". It looks like a bug to me.

Since you have the fixture handy, can you please also add the following scenarios and see if it works as expected in those cases:

1. stop at "command begin"
2. stop at "command process"
3. stop at "command end"

We should open a PowerShell issue about the unexpected behavior in those "ctrl+c/Stop" scenarios.

[alx9r]

Since you have the fixture handy, can you please also add the following scenarios and see if it works as expected in those cases:

• stop at "command begin"
• stop at "command process"
• stop at "command end"

@daxian-dbw Those results are indeed interesting.

updated report

None	stop upstream begin{}	stop upstream process{}	stop upstream end{}	stop command begin{}	stop command process{}	stop command end{}	stop downstream begin{}	stop downstream process{}	stop downstream end{}
		upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}	upstream begin{}
BeginProcessing()		BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()	BeginProcessing()
command begin{}		command begin{}	command begin{}	command begin{}	command begin{}	command begin{}	command begin{}	command begin{}	command begin{}
downstream begin{}		downstream begin{}	downstream begin{}		downstream begin{}	downstream begin{}	downstream begin{}	downstream begin{}	downstream begin{}
		upstream process{1}	upstream process{1}		upstream process{1}	upstream process{1}		upstream process{1}	upstream process{1}
ProcessRecord()		ProcessRecord(1)	ProcessRecord(1)		ProcessRecord(1)	ProcessRecord(1)		ProcessRecord(1)	ProcessRecord(1)
command process{}		command process{1}	command process{1}		command process{1}	command process{1}		command process{1}	command process{1}
downstream process{}		downstream process{1}	downstream process{1}			downstream process{1}		downstream process{1}	downstream process{1}
			upstream process{2}			upstream process{2}		upstream process{2}	upstream process{2}
			ProcessRecord(2)			ProcessRecord(2)		ProcessRecord(2)	ProcessRecord(2)
			command process{2}			command process{2}		command process{2}	command process{2}
			downstream process{2}			downstream process{2}		downstream process{2}	downstream process{2}
						upstream end{}		upstream end{}	upstream end{}
EndProcessing()						EndProcessing()		EndProcessing()	EndProcessing()
command end{}						command end{}		command end{}	command end{}
command clean{}				command clean{}	command clean{}	command clean{}		command clean{}	command clean{}
downstream end{}								downstream end{}	downstream end{}
		upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}	upstream clean{}
downstream clean{}		downstream clean{}	downstream clean{}		downstream clean{}	downstream clean{}	downstream clean{}	downstream clean{}	downstream clean{}
Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()	Dispose()

clean{} runs when the stop signal is received in the other blocks of the user script block.

If you think it would help, I could try to extract this test fixture and publish it somewhere.

[daxian-dbw]

Thank you! So, the unexpected behavior -- command clean{} should run but didn't -- happens only in the following scenarios as shown in your test results. And it should be a bug and should be further investigated.

• upstream process{}
• upstream end{}
• downstream begin{}

If you think it would help, I could try to extract this test fixture and publish it somewhere.

Yes please! That'd be very helpful. Also, I think we should create a new issue to track this bug in PowerShell repo.

[alx9r]

Thank you!

My pleasure @daxian-dbw.

So, the unexpected behavior -- command clean{} should run but didn't -- happens only in the following scenarios as shown in your test results. And it should be a bug and should be further investigated.

• upstream process{}
• upstream end{}
• downstream begin{}

I agree.

...I could try to extract this test fixture and publish it somewhere.

Yes please! That'd be very helpful.

I will try to do so.

Also, I think we should create a new issue to track this bug in PowerShell repo.

I agree. I guess we need a portable repro for that though. We'll either get that by me publishing this test fixture, or I'll cobble together another repro.

[sdwheeler]

Related to PowerShell/PowerShell#24679

[alx9r]

@daxian-dbw

Also, I think we should create a new issue to track this bug in PowerShell repo.

I'll do that next.

Test Fixture

I have attached a PowerShell module in Assuage.PowerShell.Diagnostics.ControlFlow-0.1.0.zip that reproduces the issues discussed here. That module contains the latest incarnation of the test fixture I've been using to generate the traces above.

You should just be able to expand that archive and invoke Import-Module .\Assuage.PowerShell.Diagnostics.ControlFlow to start using it.

I think the examples I put in Get-Help about_Flow_of_Control_Diagnostics should get you started. Of particular relevance to the discussion here, the command

Get-ControlFlowTestParameters |
    Test-ControlFlow |
    ? {-not $_.CleanupOk}

will test a number of scenarios and output the culprits that we discussed above (and a few others that turned up). That will output objects like

Name       : stop upstream process{}
Parameters : {[LogParameterName, Log],
              [WaitPoint, stop upstream process{}],
              [CommandName, Invoke-Steppable],
              [DryRun, False]}

whose Parameters property you can use to craft focused repros like

Trace-ControlFlow -CommandName Invoke-Steppable -WaitPoint 'stop upstream process{}' -DryRun $false | % Log

which produces the trace log

upstream begin{}
command begin{}
downstream begin{}
upstream process{1}
command process{1}
downstream process{1}
upstream clean{}
downstream clean{}

which, I think, you're already familiar with from above.

I documented the public parts of the module in the usual PowerShell help places. There's not much explanation of how the internals work, though. Hopefully it'll be clear enough for anyone stepping through it.

The .cs source files for the included .dll file are in the module folder. The .dll file was compiled from the included source using Update-Assembly.ps1. Hopefully that will be enough for you to get symbols for that part if you need them.

In any case, I think this will streamline the repros for future testing of clean{}.

Thoughts on documenting SteppablePipeline

The module that implements the test fixture includes the latest incarnation of Invoke-Steppable.

Consider the following premise: The flow of control when invoking a script block using Invoke-Steppable should resemble as closely as possible the flow of control when invoking the same script block using . dot-source operator.

The current implementation of Invoke-Steppable currently achieves that goal; only the execution order of clean{} sometimes differs. The implementation's ability to achieve the premise is, however, remarkably sensitive to various implementation details including the following:

• interfering with PipelineStoppedException passing through the overridden Cmdlet methods
• how Clean() and Dispose() are called

That leads to a rather complex task when implementing a Cmdlet using SteppablePipeline. I see three natural ways of managing that complexity:

1. Implementers run their Cmdlet implementation through a a suite of tests using a fixture like the one in the module I published here.
2. Create a specialization of PSCmdlet with a ScriptBlock member that correctly handles the SteppablePipeline invocation.
3. Document the detailed considerations that implementers should consider when using SteppablePipeline.

Given how intimately linked SteppablePipeline is with the PowerShell implementation, I think (2) is the most practical. That way the SteppablePipeline documentation could just read something like "It is recommended that implementers derive from PSSteppableScriptBlockCmdlet". And if there are future changes to the PowerShell engine that impact the correct invocation of SteppablePipeline, an updated implementation of PSSteppableScriptBlockCmdlet can be published that averts degrading existing implementations.

If you think there is merit to PSSteppableScriptBlockCmdlet (or whatever it would be called), I can write a proposal for that addition. All this test fixture work has left me with most of an implementation candidate which feels like it should be part of PowerShell itself.

[alx9r]

@daxian-dbw

Also, I think we should create a new issue to track this bug in PowerShell repo.

That is now PowerShell/PowerShell#24782.

[sdwheeler]

@alx9r Please add this discussion to PowerShell/PowerShell#24782.

[alx9r]

@sdwheeler Ah...good point. I have quoted it here.