Sverchok redux

[ly29]

Some ideas for Sverchok redux, grown out of the need to clean up the various messes.
For more starting discussion see nortikin/sverchok#688

• change the execution model
• optimize memory consumption.
• implement proper groups as code reusable code blocks
• redo data model
• interaction model
• make nodes serialize

Execution model
Instead of each node parsing the complete list view the nodes as source code and compile them to a program and execute it for each element of the list, simplifying most nodes a lot. Some, a few might need to manipulate the data structure itself but for many cases it isn't needed.
Benefit, we can lazily eval the data which can also, potentially, allow for parallelization of the execution.

Optimize memory consumption.
This ties together with the execution model, now every state of the every program is stored which leads to a lot of memory being wasted.

Implement groups as reusable blocks
This ties together with some other ideas, of how we view the node layouts. Each layout is essentially a source code file.

Redo data model.
Create sockets that makes sense, like mesh, color. Match those sockets to efficient representations that makes sense in blender context, like numpy arrays, bmesh etc using the most efficient means for accessing and setting data, foreach_get etc.

Interaction model.
First of all, take a look at where in the blender geometry nodes makes sense. Edit mode and object mode. (Perhaps also sculpting, weight paint and vertex paint.)
Example, in edit mode you could have number of Sverchok programs available as operators manipulating, say the the selection state of a mesh, perhaps being called from an auto generated/custom call menu.
Of course creation of parametric objects like we can do now is also possible.

Make nodes abstract from the blender node code.
This makes is possible to create json representations in a more sensible way, to compile and use custom classes also becomes easier. The idea being, on startup dynamiclly create the blender node classes and load the code.

---

Some concrete steps.
Bootstrap

sockets
node tree
nodes
compiler
execution engine
Development documentation
Here the number of nodes should be kept very small to keep changes very simple to implement.

Development

Node types - as needed, mostly to support the compiler
Execution modes
Dynamic node creation
Data structure - data types - finalize and develop
Make the structures serialize
And then, create full node set and documentation.

[ly29]

Also got more busy than expected today, nothing concrete to show.

[nortikin]

nothing reinforced concrete? )) joke

[nortikin]

Data model for me is main thing to user. IFC implementation (BIM) in output of bmesh viewer to write to RNA blender parameters of object and than export to IFC (there was importer of IFC). We need global IFc specifications realized in viewers.

Also threading needed to be implementing, i see one core loaded always from sverchok, even python has threading librery.

[ly29]

IFC is a complex file format/problem, I honestly don't see it tying directly into Sverchok, atleast in the short term. For IFC I would focus on import export from blender and trying to tie into that.
IFC http://www.buildingsmart-tech.org

Threading is also complex but decreasing the functional units makes it possible, how big advantage it will give is un clear but it should be in mind from the beginning. Some things like manipulating blender objects cannot be done safely from a threaded environment I suspect.

[zeffii]

I think from my discussions with campbell, the only real form of multi threaded code we could execute would be that code which doesn't use bpy. This because blender isn't thread safe with relation to bpy calls. But that may mean that bmesh could potentially work in multiprocessing but only as concurrent computation , not as 'delegating more cpu's to one computation' . argh. I don't know for certain until experimentation time :)

[ly29]

@zeffii
That is my understanding as well, we might get a bigger speedup with using numpy than threading and with less complexity.
Basically the only time when multiprocessing might help is when we have many objects without interdependencies.

[nortikin]

numpy is prime need offcourse. hanks for info, @zeffii

[nortikin]

as i understood, threading allows to make own timings, we can not bind to timeline to play animations, users asked it.

threading.Thread(target=``some definition``, args=(context,)).start()

and definition handle sleep option to delay animation. what you think? This part of code ws in music player, that i made nice in ui.

[nortikin]

how to translate redux?
returning or simplifying?

[zeffii]

"residue" or "reduction". bring it back to a core first, so yes "returning".

It's not a refactor because that asserts that the mechanisms are OK but the code itself was problematic.

I don't think there's a formal definition of Redux.. but it has become sort of synonymous with recode / reconsideration.

[nortikin]

thank you much @zeffii

[zeffii]

https://www.gliffy.com/ (for flowchart.. easy!)

[zeffii]

I would like to start working on this more serious for the next two weeks while I have time. Anyone up for it?!

[ccamara]

If there are basic and clear goals (I'm not familiar with sverchock's architecture), I'd be glad to help!

[nortikin]

@ccamara , for sure we need masterplan, or development diagram.
There are some issues we can delegate as numpy definitions to work with lists of vertices and polygons etc, as i understood, Dealga have some views on data that i not exactly understand. i.e. how speed up if we make the same approach to vertices as to polygons in numpy data.
We need draw mindmaps now

[ly29]

@zeffii
I will have some more time (I hope), I will try to bootstrap my ideas and then go from there.

[ccamara]

@nortikin glad to help with those minor tasks (whenever they are defined) and start progressing with you!

[nortikin]

http://www.gliffy.com/go/publish/9073213
just try tool

[zeffii]

visualizing the AST : https://www.quantifiedcode.com/blog/visualize-abstract-syntax-trees-asts-to-understand-python-better

[ly29]

Nice, tempted to redo that but use blender for viz. Custum node tree :)
And reuse for debugging testing out svrx.

[zeffii]

yeah, could use a custom node tree with bgl to draw it :)

[ly29]

Don't even need bgl, blender has nodes :)

[zeffii]

but I like writing GL code :)

[ly29]

Go for it :)