livecodable dsp

[ahihi]

this is an implementation of livecodable DSP by interpreting sclang code embedded in the OSC messages sent from Tidal. the code is extensively commented, so please see files changed for the details!

this is currently implemented as a "hack", but imho it is kind of a game changer and i propose that it might be worth considering as a core feature (but disabled by default, given the security implications). thoughts?

[jwaldmann]

amazing. next (on the Haskell side): make an SC backend for https://hackage.haskell.org/package/csound-expression to produce programmatically the strings that describe the DSP ...

[ahihi]

there are certainly interesting possibilities for DSLs, but lets keep discussion here focused on the SuperDirt implementation :)

i am currently working on extending this idea to per-orbit global effects, so we can write our own delays, reverbs, etc!

tidal.gdsp.mp4

[telephon]

I really like it. Simple and appropriately dirty. Let's take some time to think about how to best integrate it into core – of course it would need to be switched on explicitly.

Moreover, we should test the address the OSC message was sent from and only accept if it is 127.0.0.1 (just as a minimum protection).

We used to have a message called avoidTheWorst in our Republic system, that filtered all kinds of dangerous stuff, but of course it was exactly only a protection against unserious attempts.

[telephon]

It could also be an OSC responder with a separate OSC message that builds the synth def. Then tidalcycles could do this separation into another message. Or at least a separate method.

Also, we need to rebuild only when something in the code changes. The code could be kept ins a set and you check for changes?

[ahihi]

here are my motivations for some of the decisions in the current implementation, and some thoughts on integration.

the module:

• synthdef is rebuilt on every event
  • this allows using sclang-side randomness and other control structures to determine the synth graph, a very inspiring form of dynamism that was natural in SC2
• receives the input signal as variable in and the output (and input) bus as out
  • enables both standalone sound source and effect unit use cases with minimal typing
• output is implicit
  • less typing
  • explicit output can still be done with Out.ar and the out variable
  • maybe needs some more sophisticated logic, like GraphBuilder.wrapOut used by Function.play?
• goes first in the module chain (after dirt_sound)
  • can further process the sound with existing SuperDirt modules
• reads event data via environment variables
  • ~freq (etc) is quick to type, and automatically available
  • this is also only possible if we rebuild on each event
  • a downside is that these values get baked into the synthdef and cannot be modulated via control buses
    • i am considering also adding support for using NamedControls (\freq.kr), similarly to what i did for the global effect

the global effect:

• synthdef (and synth) is rebuilt when the code (~gdsp) changes
  • this is really the only way that makes sense for long-lived effects like delay or reverb
• receives dry signal as variable in and the two buses dryBus and effectBus
  • consistent with the module
• output is implicit and replaces the content of dryBus
  • the idea was to easily support both effects that replace the input signal (RLPF.ar(in, …)) and ones that add to it (in + CombN.ar(in, …))
  • but explicit output still possible with dryBus and effectBus
• goes first in the effect chain
  • again, enables usage as both effect and sound source for further processing by existing effects
• reads event data via NamedControls
  • environment variables obviously will not work, given the persistent nature of the effect

with the above in mind, i am not sure using separate OSC messages would be an improvement, while it would certainly increase complexity. the current design requires no additional support from frontends, which i consider a big advantage.

as for integration into core, i think something like the structure i ended up with in the latest commit could work: we would introduce a new type of event diversion which runs in a Routine and can signal "i need to run some code after a server sync" by returning a post-sync callback. for hackability, it would also be nice if this was a list of functions rather than a single function (like the current diversions are), so that new diversions could easily be added while keeping existing ones in place.

with the above mechanism, enabling the livecodable dsp functionality would be just a matter of adding functions to the new "syncable diversions" list. it seems like there could be other use cases for this, though i cant think of any right now. it does raise the question of whether such a mechanism should be even more general, perhaps having the ability to wait for arbitrary things such as Conditions. but then the api design becomes a lot more complicated…

finally, there is some room for optimization:

• the silence synth should probably just free itself immediately
  • or we could have special handling for this and not spawn a synth at all
• rather than spawning the "default" global effect synths when there is no ~gdsp, it would be better to not spawn any synth.

[telephon]

Sorry, for some reason I haven't seen your reply, it wasn't in my notifications. I think we can make a version that does not require hacking the diversion.

But it'll have to wait till later this week.

[ahihi]

no worries, interested to hear your ideas whenever you have time!

[ahihi]

@telephon have you had a chance to look at the latest version? :)

[telephon]

Yes, I have, and thought about some things, just a few points to consider

• I think it would be possible to avoid the "syncable diversions": every event could sync on its own in parallel.
• currently, you are adding effects to all orbits with different names – I'd say that it would be better to either have it only from where you add it, or the same (identical) effect on all orbits. Maybe I overlook something.
• It would be good for testing and generality if one could call the functionality from within sclang. The things that happen in DirtLiveDsp::start could be separated into methods, which could be called when needed. (e.g. everybody will benefit from having methods like DirtOrbit.addGlobalEffect and DirtOrbit.removeGlobalEffect).

As I've already said, this will be a great addition. So just some refactoring, if you agree. After that, I may just have a few minor improvements on the code, mostly "cosmetic".

[ahihi]

I think it would be possible to avoid the "syncable diversions": every event could sync on its own in parallel.

events do already sync in parallel, the idea behind the syncable diversions was just to provide a general hackable mechanism for diversions that follow the pattern "run code A, wait for server sync, run code B" which could possibly be useful for other purposes. do you mean it would be better to not provide this, and instead have the live dsp logic more tightly coupled with DirtEvent.play?

currently, you are adding effects to all orbits with different names – I'd say that it would be better to either have it only from where you add it, or the same (identical) effect on all orbits. Maybe I overlook something.

this is to make it possible to write a different global effect for each orbit, which surely is much more useful musically. since GlobalDirtEffect.play spawns the synth called name.asString ++ numChannels, this leads to the requirement that the orbit number must be part of the effect name. but we could instead put a property on GlobalDirtEffect that, when non-nil, overrides the auto-derived synthdef name. then the effect name could be kept the same on each orbit. (in fact i see there is already a defName property which is never used, maybe there was already some intent to do this?)

It would be good for testing and generality if one could call the functionality from within sclang. The things that happen in DirtLiveDsp::start could be separated into methods, which could be called when needed. (e.g. everybody will benefit from having methods like DirtOrbit.addGlobalEffect and DirtOrbit.removeGlobalEffect).

good idea!

[telephon]

I think it would be possible to avoid the "syncable diversions": every event could sync on its own in parallel.

events do already sync in parallel, the idea behind the syncable diversions was just to provide a general hackable mechanism for diversions that follow the pattern "run code A, wait for server sync, run code B" which could possibly be useful for other purposes. do you mean it would be better to not provide this, and instead have the live dsp logic more tightly coupled with DirtEvent.play?

Thinking again, I tend towards keeping it general, it is good to have. I may be wrong, but I wouldn't make it a diversion, it is just syncFunctions. Couldn't it be like this?:

if(~sustain >= orbit.minSustain.value or: { ~play.notNil }) {
	this.finaliseParameters;
	// unless event diversion returns something, we proceed
	~play.(this) ?? { 
		
		if(~syncFunctions.notNil) {
			Routine { 
				this.callSyncFunctions;
				this.playSynths;
			}.()
			
		} {
			this.playSynths 
		}
		
		
	}
}

All the rest – preparing the right instrument name etc. would be done in the ~syncFunction.

One more point, but that can be done in a next step, is that there is no need to add the diversions to every orbit even if you are not using it for patterned dsp. All that is needed is adding one syncFunction?

currently, you are adding effects to all orbits with different names – I'd say that it would be better to either have it only from where you add it, or the same (identical) effect on all orbits. Maybe I overlook something.

this is to make it possible to write a different global effect for each orbit, which surely is much more useful musically. since GlobalDirtEffect.play spawns the synth called name.asString ++ numChannels, this leads to the requirement that the orbit number must be part of the effect name. but we could instead put a property on GlobalDirtEffect that, when non-nil, overrides the auto-derived synthdef name. then the effect name could be kept the same on each orbit. (in fact i see there is already a defName property which is never used, maybe there was already some intent to do this?)

My thinking is this: The GlobalDirtEffect is an object and there can be several different GlobalDirtEffect objects with the same name. All you need to do is to add that effect only to the current orbit. The dirt event will only use the effects in the orbit given:

orbit.globalEffects.do { |x| x.set(currentEnvironment) };

So the calls dirt.orbits.do … are not needed, I think. You can keep everything local.

It would be good for testing and generality if one could call the functionality from within sclang. The things that happen in DirtLiveDsp::start could be separated into methods, which could be called when needed. (e.g. everybody will benefit from having methods like DirtOrbit.addGlobalEffect and DirtOrbit.removeGlobalEffect).

good idea!

[telephon]

Another idea, what about this?

DirtModule {
	var <name, <func, <test, <preparationFunc, <preparationTest;

	*new { |name, func, test, preparationFunc, preparationTest|
		^super.newCopyArgs(name, func, test ? true, preparationFunc, preparationTest)
	}

	value { |orbit|
		if(test.value, { func.value(orbit) })
	}

	prepareValue { |orbit|
		if(preparationTest.value, { preparationFunc.value(orbit) })
	}

	mayNeedPreparation {
		^preparationFunc.notNil
	}

	== { arg that;
		^this.compareObject(that, #[\name])
	}

	hash {
		^this.instVarHash(#[\name])
	}

	printOn { |stream|
		stream  << this.class.name << "(" <<< name << ")"
	}

	storeArgs {
		^[name, func, test, preparationFunc, preparationTest]
	}
}

Then we could run:

prepareModules {
		if(modules.any(_.mayNeedPreparation)) {
			Routine {
				modules.do(_.prepareValue(this));
				server.sync;
				~latency = ~timeStamp - thisThread.seconds;
			}.value
		};
	}