Skip to content

TODOs

Jump to bottom
Kostis Anagnostopoulos edited this page May 3, 2023 · 10 revisions

TODOs

  • [+] ENH: planning reports unsatisfied node modus, for plotting
  • [+] ENH: Plan accepts previous solutions
    • [+] fix: FnOp.compute() should (or should not?) accept Pipeline.compute() args.
    • [+] FEAT: compute(solution, recompute_from=["input_1"])
  • [ ] use setuptools-scm, replace __init__.py:__version__ with own template.
  • [ ] ++FEAT/BREAK: Python3.6-->3.8 for walrus operator, positional-only parameters f-string=, .
  • [ ] +++REFACT: Merge Config+Operation+Authographed to simplify them
  • [ ] !!BUG: since modify("/a/b") == "/a/b" ==> jsonps MUST CEASE TO BE strings!! (or search-n-replace all comparisons outside of the graph with get_jsonp(dep1) == get_jsonp(dep2)). It's not such a big issue bc few comparisons of dependencies happen before entering the graph (where in there they are properly splitted), and any important comparisons outside, such as aliases vs existing outputs, they work ok bc usually ALL /such/deps are indeed jsonps.
  • [ ] BUG(JSONPOINTER): json_pointer([]) == json_pointer([""]) == json_pointer("") == "" # root while jsonpath("") == [] # cwd and jsonpath("/") == [""]; PROBLEMATIC is that json_pointer("") accepts strings (must turn it into vararg), while it should return json_pointer([]) == "", json_pointer([""]) == "/".

Configs

  • [ ] FEAT: drop MANY contextvars with 1 cfg-dict in contextvars?
  • [ ] FEAT: drop contextvars, use merging-dicts on (cfg, pipe, op) instead? BUT then cannot change configs amidst run, unless next todo ...
  • [ ] FEAT: configs & op attributes accept 2-tuples: (flag, priority) default priorities: cfg, fnop (+cat: Compose)
  • [ ] FEAT: Config DEBUG flags:
    • [ ] skip - evictions(drop config)
    • [ ] keep SFX in outputs

Compose

  • [+] ++refact: change :term:`accessor` diacritic `` from $ to @``.

  • [+] +++feat: add a real implicit modifier.

  • [+] ++feat: ^ diacritic for implicits

  • [-] ++refact: change keyword diacritic to **. - NO, dep>kword is better.

  • [+] refact: rename modifier sfx --> token

  • [+] refact: introduce diacritic for sfx $ (vs sfx()).

  • [ ] refact: change (or separate) jsonp diacritic (from accessor) to / (@)? - [ ] NO, simply disallow non-jsonp deps with /.

  • [+] ++refact: simplify tokens simply as dolar-words

  • [ ] fix(modif): add ? diacritic to varargs; varargs are NOT obligatory optionals (its app's decision).

  • [ ] REFACT: construct deps also with DIACRITICS (vs modify()) and update ``repr()` (keep str as is: stripped dep)):

    arg or kw:    [?] dep [> keyword]  [$ token [...]]
vararg:       (*|+) dep  [$ token [...]]
implicit:     [?]^ dep
token:        [?]$ token

  • [?] REFACT: separate op-decorator from factory (to facilitate defining conveyor operations).

  • [?] cwd() modifier: Isn't already working with relative jsonps?

    • implicit with only the 2nd arg should work on cwd, no?

  • [ ] feat: :func:`.get_accessor()` should return a partial on the dependency.

  • [-] feat: :term:`alias` should accept dict[str, list] and list[tuple[str, list]].

    • REJECTED: prefer diacritics on provides

  • [ ] ++FEAT: Teach pipelines how to accept positional arguments with a tuple with their names

  • [ ] ++FEAT: +1 merge method for pipelines: nest=False: treat Pipelines as Operations (need pipeline feat for positional-args, above).

  • [ ] enh(modifier): augment :func:`keyword` to hold an "informative" value read from the function-default.

  • [ ] break/enh(modifier): augment modifier table with implicits for REPR.

  • [ ] BREAK(modif): "stuffy" names for is_xxx(modifier) funcs returning stuff.

  • [ ] modifier || jsonp --> pandalone.pstep (flexible syntax for a modifier on each step)

Autograph

  • [ ] +++FEAT: Autograph functions by type-annotations (PY39 PEP 593 e.g. Annotated[int, CWD])
  • [ ] ++feat: populate :func:`keyword` "informative" value from inspect-signature. (see also keyword-"informative" value, above)

Planning

  • [ ] FEAT: break cycles with dijkstra; +weights?
  • [ ] ++ENH: link graph inputs/outputs to virtual root-nodes, to replace custom visiting algos (eg unsatisfied-ops, prune-by-outs)
  • [ ] FEAT: per-operation EVICTs (+cat: Compose)
  • [ ] ++DROP PARALLEL: always produce a list of "parallelizable batches", to hook with other executors, and keep here just the single-process implementation. (+cat: Execution)
  • [ ] +++FEAT: Identify any jsonp-needs already in the given-inputs as sub-docs.

Execute

  • [+] FEAT: +2 callbacks before marshalling: (pre_batch, post_batch) (olds: pre_op, post_op).
    • PARTIAL: moved existing x2 callbacks to be called before marshalling.
  • [?] refact: named_inputs --> sol
  • [ ] ++ENH/REFACT: use Signature.Param from inspect module to match needs & zip provides and support already there ....
  • [ ] Solution-retriever modifier opcb();
    • WONTFIX: easier and more generic to access solution from Op-context.
    • REINSTATED: bc it's simpler and does not have any threading issues....
  • [ ] ++FEAT(fnop): vararg(s) for Outs collect all outs to the very end
  • [ ] BREAK/FEAT: allow marking SFX-LIST items:
    1. Implicits (default), to be checked/resolved downstream (see "FEAT: +2 callbacks"), or
    2. Sfx (pure), as they are now).
  • [ ] +++BREAK/DROP accessors for jsonp (move all functionality in solution)
    • REJECTED already used for hcat()/vcat()
    • REINSTATED as "ACCESSOR per jsonp-PART"! (h/vcat should have been like that) by default applied to the last part (or use [acc, ...] syntax for specific parts)
    • [ ] RENAME Solution --> DataTree
    • [ ] move dep_prefixed() as a top-level modifier utility (to prefix also Accessors)
    • [ ] revive "stable sort paths" branch on solution updates
    • [ ] hdf/vdf accessor example-utilities for typical level-0 resolving multi-indexed slices.
    • [ ] Drop int/attribute indexers from resolve-jsonp, accessors-per-part can do that.
    • [ ] Interoperate with Dask, Vaex, Modin etc (see https://12ft.io/proxy?&q=https%3A%2F%2Ftowardsdatascience.com%2F4-libraries-that-can-parallelize-the-existing-pandas-ecosystem-f46e5a257809)
  • [ ] replace in-house jetsam with PEP 678 for exception notes on PY3.11+.

Solution

  • [+] DROP/ENH: Solution updates GivenInputs only, yet still layers jsonp-refer to its values.
  • [+] FEAT: pd.concat() --> modifier+accessor, to avoid sfxed for multi-column updates.
  • [+] FEAT: + post_callback with pre_callback --> callbacks tuple (+cat: Compose)
    • [+] ENH(jsonp): mass-concat (not one-by-one).
    • [+] ENH: validate Implicits indeed added (in post_op_callback)?
    • [ ] FEAT: VALIDATE (by user) items on Solution-Insert (with a post_callback?): - auto-assertions {jsonp--> validators} - compare overwrites while recomputing - check implicits exist (??not done above??)

Plotting

  • [+] Badges on Data
  • [ ] generate legend dynamically
  • [ ] Draw nested graphs as such (hiding internal nodes)
  • [ ] SPHINXEXT: autodoc Pipelines & Ops
  • [ ] sphinxext: extend standard doctest or doctest module (instead of sphinx-builder)
  • [ ] FEAT/REFACT: dogfood when constructing graphviz graphs.
  • [ ] FEAT: add another dagviz plot-backend.

Docs

  • [ ] REFACT: move GitHub organization pygraphkit --> pygraphtik (+Travis, +RTD)
  • [ ] Merge tutorial (operations + composition)
  • [ ] +++DOC: explain Implicits in tutorial; mention ovunque.
    • [ ] TCs: Test DEBUG

DROPPED

  • [X] solution.executed pre-populated with all operations
  • [X] parallel batches restart from last position in steps
  • [X] covert custom op classes & modifiers directly into mergeable networkx graphs;
    • WONTFIX bc foreign function would not work with merged deps.
  • [X] conditionals
    • WONTFIX bc it permits pipelines with too complex execution flow to debug.
  • [X] DROP accessors:
    • REJECTED already used for hcat()/vcat() (but to be dropped for jsnop).

v9.0.0

  • [X] Accept jsonp inputs & outputs,
    • WONTFIX user's business to expand into given Inputs, Outputs already working.
  • [X] REVERT rename subdocs;
    • WONTFIX bc eventually made it work correctly and added TC.
  • [X] REFACT: separate op-decorator from factory (to facilitate defining conveyor operations):
    • NO, simplify passing fn=None.
  • [X] Nest-rename subdocs: not by default, possible by renamer/nester.
  • [X] accessors accept default (not to search x2 contain+get_path)
    • WONTFIX bc not worth it.
  • [X] Simplify Task-context by injecting it in a parametric argument of fn.
    • NO, opcb modifier works without inspect module.
  • See :gg:`1`.
Clone this wiki locally