Doc tweaks for ev/deadline and ev/with-deadline

[sogaiu]

This is an attempt to address #1349.

Background can be seen in that issue, and a summary is available at this comment.

Thanks to @amano-kenji and @llmII for discussions and investigations.

[pepe]

LGTM

[amano-kenji]

It is okay, but it doesn't explain the relationship between tocheck and tocancel. It doesn't document the art of choosing the right fiber for tocancel.

If I didn't read the background discussion, I would say what is this?

[pepe]

@amano-kenji, I would like to ask you to edit your last comment to remove vulgarism, please.

[llmII]

The only relationship between tocheck and tocancel that is definitive is within ev/with-deadline. With ev/deadline it is an optional but likely a typical association that the task tocancel started the fiber tocheck and is interested in receiving an error rather than waiting on tocheck to complete, and possible could respond by canceling tocheck. I'm not sure how or if the typical use case should be in the doc-string because it is possible to have something less typical. It might be that site documentation could benefit from outlining the typical usage, but I believe that if one is using the typical usage, they could likely just use ev/with-deadline. Which would of sorts, roundabouts, lead towards that one probably is using ev/deadline specifically for a special usage and thus documenting the typical usage as with ev/with-deadline would fail to make sense to do in the docstring?

[sogaiu]

it doesn't explain the relationship between tocheck and tocancel. It doesn't document the art of choosing the right fiber for tocancel.

AFAIU, there isn't necessarily a relationship between tochecck and tocancel.

IIUC, establishing one is entirely up to the programmer. There doesn't need to be one for this function operate.

I've started to take some notes for potential additions / changes to the website docs here.

The kind of thing you are describing might be a good addition there.

[amano-kenji]

If I was the author of this pull request, I would add more background information to the docstring for ev/deadline.

I would at least document how cancelling tocancel usually leads to cancellation of tocheck as well, but tocheck doesn't have to be cancelled along with tocancel in some cases. If I don't understand the art of choosing fibers for ev/deadline, I wouldn't feel comfortable about using it. People who aren't as OCD as I am would probably just avoid using ev/deadline because they don't know what tocancel is supposed to be.

They will likely think

I just passed nil to tocancel, and it just works. I don't know how it works or why it works, but it works.

What the hell is root fiber? How is it related to current fiber in any way? It doesn't make sense at all.

If cancelling tocancel doesn't cancel tocheck, is ev/deadline going to freeze for ever?

That's not good enough. This will lead to confusion or reverse-engineering. Perhaps, people can actually freeze ev/deadline for ever with a wrong fiber for tocancel.

If the background information is too verbose, then ev/deadline can refer people to an offline document.

[llmII]

@amano-kenji wrote:

I would at least document how cancelling tocancel usually leads to cancellation of tocheck as well, but tocheck doesn't have to be cancelled along with tocancel in some cases. If I don't understand the art of choosing fibers for ev/deadline, I wouldn't feel comfortable about using it. People who aren't as OCD as I am would probably just avoid using ev/deadline because they don't know what tocancel is supposed to be.

The problem would be that that isn't necessarily true. One can contrive ways in which this would not be the case. Say you have a task (A) and a task (B). B knows about A and knows A has an interest in the cancellations of fibers it runs under a deadline. ev/deadline allows for more uses than the typical, though ev/with-deadline outlines the typical usage and though I find it hard to have a valid use otherwise, I'm sure it will be imagined, and documentation stating the typical should probably say something more along the lines of "Typically used via ev/with-deadline, and not called manually".

What the hell is root fiber? How is it related to current fiber in any way? It doesn't make sense at all.

That's not good enough. This will lead to confusion or reverse-engineering. Perhaps, people can actually freeze ev/deadline for ever with a wrong fiber for tocancel.

If the background information is too verbose, then ev/deadline can refer people to an offline document.

Documenting what is a Fiber, and a Task, and a Root Fiber in every single function using them is not a good idea. That's way too verbose. It shouldn't be surprising or unexpected for one to glean those terms from referencing the language documents.

Edit:
I did notice that a Fiber calling into the event loop after it's root fiber got canceled due to a deadline did receive the deadline error.

[amano-kenji]

I would likely mention something like

Cancelling tocancel must lead to cancellation of tocheck. Otherwise, ev/deadline may freeze for ever. The current fiber is cancelled when the root fiber is cancelled.

This is not too verbose. You can probably condense it down to

Cancelling tocancel must lead to cancellation of tocheck. Otherwise, ev/deadline may freeze for ever.

After reading this, people will realize cancelling the root fiber leads to cancellation of the current fiber.

[llmII]

I would likely mention something like

Cancelling tocancel must lead to cancellation of tocheck. Otherwise, ev/deadline may freeze for ever. The current fiber is cancelled when the root fiber is cancelled.

This is not too verbose. You can probably condense it down to

Cancelling tocancel must lead to cancellation of tocheck. Otherwise, ev/deadline may freeze for ever.

After reading this, people will realize cancelling the root fiber leads to cancellation of the current fiber.

If anything other than the root fiber of tocheck is given as tocancel then tocheck might not be canceled. Cancelling the root fiber of tocheck is no guarantee that tocheck will be canceled. If tocheck protects all it's calls into the event loop it will remain un-errored (the fiber created by protect receives the error thrown by calling into the event loop via ev/* functions).

[amano-kenji]

You can freeze ev/deadline unknowingly?

[llmII]

It should be possible. The likelihood that someone will without purposefully mucking around in ev/deadline is minimal. It's also why one should limit their error handlers to the minimum body of code needing to run within an error handler (so that the error propagates upwards). I think some of the code I displayed in my review actually illustrated that the body of ev/with-deadline can continue to run even after the root fiber of that body which is put into a fiber is cancelled. tocheck is just never directly cancelled by ev/deadline.

Basically, it counts on the fact that tocheck will either yield, or yield to the event loop at some point, entering a state from one of the following [:dead :suspended :pending].

Any fiber that doesn't do one of those things - yield, or yield to the event loop - is in danger of freezing Janet in general, not just ev/deadline, and I'm sure this is documented on Janet's site somewhere (and is general knowledge for concurrency within a single thread).

[amano-kenji]

I read the definition of ev/with-deadline.

(defmacro ev/with-deadline
    `Run a body of code with a deadline, such that if the code does not complete before
    the deadline is up, it will be canceled.`
    [deadline & body]
    (with-syms [f]
      ~(let [,f (coro ,;body)]
         (,ev/deadline ,deadline nil ,f)
         (,resume ,f))))

I wrote test.janet

(def fiber (coro
             (try
               (forever (ev/sleep 1))
               ([err f]
                (eprint "fiber cancelled")
                (propagate err f)))))

(def task (ev/spawn
            (try
              (forever
                (ev/sleep 1))
              # ignore errors
              ([err f]))))

(pp (ev/deadline 5 task fiber))
(print "ev/deadline returned...")
(resume fiber)

Running this script results in

$ janet test.janet
<fiber 0x561235B2B600>
ev/deadline returned...

Thus, ev/deadline returns tocancel immediately. It doesn't block. ev/with-deadline cancels coro by sending an error to the root fiber which is running tocheck right now. Because the root fiber is resuming tocheck, error is sent to tocheck through the root fiber. tocheck is not directly cancelled.

Actually, tocheck continues sleeping in the main task in test.janet.

[llmII]

Borrowing from my review:

This also shows the body is still alive...

(pp
  (protect
    (ev/with-deadline 0.01
      (protect (ev/sleep 2)) # the call to `ev/sleep` receives the error...
      (pp (fiber/status (fiber/current)))
      (pp [:fib (fiber/current)])
      (pp [:lineage (debug/lineage (fiber/root))])
      (flush)
      (yield :hi))))

With results:

:alive
(:fib <fiber 0x06EB603B8050>)
(:lineage @[<fiber 0x06EB6026CED0> <fiber 0x06EB603B7BF0> <fiber 0x06EB603B7FE0> <fiber 0x06EB603B8050>])
(true :hi)
nil

And

(let [fib (coro
            (pp [:fib (fiber/current)])
            (pp [:lineage (debug/lineage (fiber/root))])
            (ev/sleep 0.2)
            (yield :hi))]
  (pp [:tocancel (ev/deadline 0.1 nil fib)])
  (pp (protect (ev/sleep 5)))
  (print "Cancellation has occurred.")
  (resume fib))

Has results:

:tocancel <fiber 0x06EB6026CED0>)
(false "deadline expired")
Cancellation has occurred.
(:fib <fiber 0x06EB6026DD40>)
(:lineage @[<fiber 0x06EB6026CED0> <fiber 0x06EB6026DA30> <fiber 0x06EB6026DD40>])
:hi

[llmII]

@amano-kenji wrote:

I think ev/deadline should mention that it doesn't cancel tocheck and returns tocancel immediately without waiting for anything.

It already mentions in both the new documentation and the old that it is tocancel that gets canceled. It takes a lot of assumption to arrive at the conclusion that tocheck is actually canceled by ev/deadline.

The current documentation for ev/with-deadline is wrong though, and the tweaked documentation doesn't highlight this fact either. I only realized while reviewing that I was automatically interpreting the word "it" as "root fiber" and that the root fiber was (fiber/root).

[amano-kenji]

I think ev/deadline should mention that it returns tocancel immediately. deadline makes me assume that the function waits until the deadline before returning tocancel.

[amano-kenji]

The current docstring for ev/deadline says

Set a deadline for a fiber tocheck.

deadline is a terrible choice of word for tocheck. This word makes me think tocheck is cancelled beyond the deadline.

Deadline for tocheck means tocheck is dead...!! If you don't finish within deadline, you are dead. You understand me?

tocheck dead is not what happens.

[llmII]

That's going to argue what words mean or imply. I'm not sure if there is a better selection of a name.

In life, one often continues and completes work after a deadline has failed to be met, but it means they did the work for nothing (it's unaccepted as past due). This is exactly what happens to tocheck. It continues it's work unless told otherwise even after it failed to meet the deadline.

[amano-kenji]

If you don't finish fiber A within a deadline, I will cancel your mom's lunch, but I won't do anything to fiber A.

That would be easier to understand.

(ev/deadline 5-seconds your-mom's-lunch fiber-A)

[amano-kenji]

A better docstring would be

Wait fiber tocheck for sec seconds. If it doesn't finish in sec seconds, tocancel will be cancelled as with ev/cancel, but nothing happens to tocheck.

tocancel is the hostage. tocheck is the fiber with a deadline.

[llmII]

@amano-kenji

The proposed docstring (from C side) is as below:

              "Set a deadline for a fiber `tocheck`. If `tocheck` is not finished after `sec` seconds, "
              "`tocancel` will be canceled as with `ev/cancel`. Note that if `tocancel` is given, "
              "it must be a task (aka root fiber). `sec` is a number that can have a fractional part. "
              "If `tocancel` and `tocheck` are not given, they default to `(fiber/root)` and "
              "`(fiber/current)` respectively. Returns `tocancel`.") {

Your proposal for a better one fails to outline what happens to tocancel and what tocancel must be. The current proposal doesn't lack in that information. Adding an explicit note that about tocheck is unnecessary as the only thing ev/deadline does is check it's liveliness. Nowhere in the docstring is there a hint that any action is taken against tocheck. Your proposal also fudges the truth of the matter a tibt with "but nothing happens to tocheck" since that can be untrue depending on if tocancel is the ancestor of tocheck and tocheck makes event loop calls. Nothing is done to tocheck directly, true. Claiming nothing happens to tocheck due to the cancellation of tocancel can be untrue in some cases though.

[amano-kenji]

Set a deadline for a fiber tocheck

This wording is still confusing. This made me think tocheck dies....

I'd rather write

Set a deadline for tocancel. If tocheck doesn't finish in sec seconds, tocancel is cancelled as with ev/cancel.

I may even add

Cancelling tocancel can lead to cancellation of tocheck if tocancel is the root fiber for tocheck.

Deadline for something means death for the thing in plain english. The documentation should make it clear that death directly comes to tocancel only. If documentation isn't as obvious as llama, it becomes an IQ test.

[llmII]

Set a deadline for a fiber tocheck

This wording is still confusing. This made me think tocheck dies....

I'd rather write

Set a deadline for tocancel. If tocheck doesn't finish in sec seconds, tocancel is cancelled as with ev/cancel.

That would be wrong. The thing checked for liveliness beyond a deadline expiration is the thing which has a deadline set upon it. tocancel does not get checked for liveliness.

I may even add

Cancelling tocancel can lead to cancellation of tocheck if tocancel is the root fiber for tocheck.

Deadline for something means death for the thing in plain english. The documentation should make it clear that death directly comes to tocancel only.

Adding an ambiguous truth is likely less useful than adding nothing and letting it remain true that "there is nothing documented about anything being done to tocheck so we cannot assume anything is done against it".

[amano-kenji]

Then, this is better.

Wait fiber tocheck for sec seconds. If it doesn't finish in sec seconds, tocancel will be cancelled as with ev/cancel, but ev/deadline doesn't do anything directly to tocheck.

[llmII]

Then, this is better.

Wait fiber tocheck for sec seconds. If it doesn't finish in sec seconds, tocancel will be cancelled as with ev/cancel, but ev/deadline doesn't do anything directly to tocheck.

Wait is untrue, there is no waiting, according to what you said earlier, ev/deadline returns immediately.

[amano-kenji]

How about this?

If tocheck doesn't finish in sec seconds, tocancel will be cancelled as with ev/cancel. ev/deadline doesn't do anything directly to tocheck.

Deadline for tocheck was the thing that confused me. It can mean two things.

• If tocheck doesn't finish in deadline, tocancel dies.
• If tocheck doesn't finish in deadline, tocheck and tocancel die, but the docstring assumes users will arrive at a conclusion that tocheck dies, too. That assumption is a secret knowledge among inner circle. Until you are in the inner circle, you have to suffer. Only people in the inner circle have the full knowledge. Janet is an esoteric langauge. The real knowledge is shared verbally outside documentation.

How do I know whether the docstring is or is not incomplete or esoteric?

I want to eliminate the possibility of connotations. Expecting users to take one interpretation over another is bad.

[llmII]

How about this?

If tocheck doesn't finish in sec seconds, tocancel will be cancelled as with ev/cancel. ev/deadline doesn't do anything directly to tocheck.

Deadline for tocheck was the thing that confused me. It can mean two things.

* If tocheck doesn't finish in deadline, tocancel dies.

* If tocheck doesn't finish in deadline, tocheck and tocancel die, but the docstring assumes users will arrive at a conclusion that tocheck dies, too. That assumption is a secret knowledge among inner circle. Until you are in the inner circle, you have to suffer. Only people in the inner circle have the full knowledge. Janet is an esoteric langauge. The real knowledge is shared verbally outside documentation.

I want to eliminate the possibility of connotations. Expecting users to take one interpretation over another is bad.

You're confusion probably stems from a misunderstanding of the term deadline. A deadline means the work done is no longer timely, not that it isn't done, or isn't continuing to be worked upon.

Your proposed doc string leaves out key details such as the default values, the necessary type for tocancel and probably other things (those are just the two I recognize as lacking offhand).

Finally, the second thing in your bullet list goes with the idea that people read documentation and make assumptions where the documentation outlines nothing occurring. If a behavior is not documented, it either doesn't happen, or should be documented. In this case, the documentation would align with "it doesn't happen, nothing to document".

[amano-kenji]

a misunderstanding of the term deadline

It is not a misunderstanding. Deadline by itself doesn't explicitly mention which things die after deadline.

In matters of death, we should explicitly mention which things are killed directly.

Oops, I forgot to mention tocheck dies, too. <--- That's what I was thinking...

[llmII]

A deadline implies the death of validity of work, not the death of that doing the work.

[amano-kenji]

A deadline has many meanings. It's not just one thing. It can mean a different thing to a different person.

One of definitions on dictionary.com says

a boundary around a military prison beyond which a prisoner could not venture without risk of being shot by the guards.

This is the definition I thought of. Deadline for prisoner A means death for prisoner A.

Unless you want to define the meaning of deadline in ev/deadline, you need to let people know that tocheck is not cancelled directly.

[amano-kenji]

That's confusing as hell.......

[sogaiu]

I'm still working on ev/with-deadline.

These are very much in draft state, but FWIW.

I'm very much against inventing fictions for the most part.

[sogaiu]

@amano-kenji Please stop with the use of profanity and consider rephrasing the instances made already.

It is completely unnecessary. Alternative expressions would do fine.

[amano-kenji]

Fine... But, that docstring doesn't explain the intention behind ev/with-deadline.

[sogaiu]

Which docstring are you referring to? The current one?

I don't necessarily disagree.

[llmII]

@amano-kenji
It is known that obviating what exactly is getting canceled with ev/with-deadline is difficult, to the degree I nop'd and went with just say exactly what happens. The only other way I could figure out to outline what's getting canceled would mean the docstring would contain documentation on the event loop.

@sogaiu
Looking at those
I so far like the first one the best.

[amano-kenji]

the value of (fiber/root) will be canceled

This sounds esoteric and confusing. It would throw me off balance if I haven't been in the discussion for the last few days.

I would write

The task(the root fiber) running the body of ev/with-deadline will be cancelled with an error.

If the body of ev/with-deadline blocks the whole task with CPU intensive operations, then what happens? Perhaps, the task will receive an error after the body finishes CPU intensive operations?

[sogaiu]

If the body of ev/with-deadline blocks the whole task with CPU intensive operations, then what happens?

I didn't test, but IIUC, the website docs say this about the event loop:

When using the event loop, it is important to be aware that CPU bound loops will block all other tasks on the event loop.

Have you read that page recently?

[amano-kenji]

So, reception of the cancellation error will be delayed after deadline if the body of ev/with-deadline stalls the task.

[sogaiu]

As I mentioned, I have not tested.

[llmII]

I would write

The task(the root fiber) running the body of ev/with-deadline will be cancelled.

That's closer to the wording @sogaiu has in what he linked which I'm fine with.

If the body of ev/with-deadline blocks the whole task with CPU intensive operations, then what happens?

It does exactly that, it blocks the whole task... It's why we can't say that ev/with-deadline kills the body because the body can be ill-behaved. It could even perhaps interact with some event loop from another project where it ends up not actually being cpu intensive but since it never calls into Janet's event loop it actually successfully completes.

[amano-kenji]

The value of (fiber/root)

This has a disorienting effect on my psyche. This is a riddle unless the relationship between the root fiber and the body of ev/with-deadliine is made clear.

The word, root fiber, still throws me off balance. The value of (fiber/root) throws me off balance even more.

I kind of understand the root fiber, but I don't really.

[llmII]

That relationship I believe is beyond the scope of a docstring (wherein it'd likely pertain to documenting the event loop in more detail than belongs in a docstring). That said, something similar to your wording of

The task (the root fiber) running

is fine with me.

@sogaiu has a link that if you look has similar wording for one of the ideas of what the docstring should be.

[sogaiu]

Re: (fiber/root)

This sounds esoteric and confusing.

I think this is not a reasonable characterization for a few reasons:

• It specifically mentions something you can look up the docstring for
• fiber/root is explicitly mentioned in the website docs event loop page in this section

ev/* callables have to do with the event loop and the docstrings for ev/deadline and ev/with-deadline should be able to assume a certain amount of familiarity with the event loop.

---

If you are thrown off-balance, perhaps you should consider whether there are other factors that are contributing to that as well. Are you calm right now? Or are you caught up in emotion?

[amano-kenji]

I read the event loop documentation three times from top to bottom while working on j3blocks. Perhaps, the documentation doesn't inculcate the concept of root fiber enough.

Root fiber just slips off my memory...

[sogaiu]

Glad you read them.

I am thinking to suggest the addition of a glossary to the website docs. You can see the beginning of a draft here.

[amano-kenji]

I think https://janet-lang.org/docs/event_loop.html doesn't account for the fact that humans don't remember concepts they pass by just once.

Only after they encounter example making you think about root fiber several times, root fiber sinks in your psyche.

[sogaiu]

Your observations are not lost on me. It's part of why I work on docstrings and examples and have been doing so over the past few years.

[amano-kenji]

Because humans just forget words they pass by briefly, docstring should reinforce the concept of root fiber... here and there....

The task (the root fiber) running the body of ev/with-deadline will be cancelled..

This reinforces the concept of root fiber and task, and makes people read the documentation again...

[sogaiu]

I don't agree with that necessarily though.

Update: the text that the above was a response to changed after I wrote the text above. So I am adding a fuller response below.

---

I find the reasoning in:

Because humans just forget words they pass by briefly, docstring should reinforce the concept of root fiber... here and there....

to be flawed.

However, I don't disagree that occasional hints / reinforcement in docstrings is necessarily a bad idea and I have actually been doing this.

Regarding "humans just forget words they pass by briefly" -- knowing this, I think the authors of documents can make an effort, but at the same time, readers can also take steps to help themselves in ways customized to their own needs.

One concrete step one can take is to take one's own notes along with writing and keeping sample code. This can also include the projects one has worked on. This method has worked well for me -- I am often able to recreate enough context / knowledge by looking over what I produced earlier. It's also much safer from other parties modifying it (^^;

---

I'll stop here though because this is not about this issue directly.

[sogaiu]

I'll remind you that bakpakin has stated that this is an issue tracker and not a chat room.

Please consider our discussion from yesterday regarding drafting your responses and going over them / revising them before posting.

[llmII]

LGTM!

[amano-kenji]