Merge pull request #71 from visualDust/dev

Dev

README.md

@@ -2,9 +2,34 @@
 
 [![wakatime](https://wakatime.com/badge/user/b93a26b6-8ea1-44ef-99ed-bcb6e2c732f1/project/8f99904d-dbb1-49e4-814d-8d18bf1e6d1c.svg)](https://wakatime.com/badge/user/b93a26b6-8ea1-44ef-99ed-bcb6e2c732f1/project/8f99904d-dbb1-49e4-814d-8d18bf1e6d1c)
 
-~~一个自产自销的仓库~~ Logging/Debugging/Tracing/Managing/Facilitating your deep learning projects
+![](./doc/static/img/readme.png)
 
-A small part of the documentation at [neetbox.550w.host](https://neetbox.550w.host). (We are not ready for the doc yet)
+## docs & quick start
+
+Logging/Debugging/Tracing/Managing/Facilitating your deep learning projects. A small part of the documentation at [neetbox.550w.host](https://neetbox.550w.host). (We are not ready for the doc yet)
+
+## installation
+
+```bash
+pip install neetbox
+```
+
+## use neetbox in your project
+
+in your project folder:
+```
+neet init
+```
+neetbox cli generates a config file for your project named `neetbox.toml`
+
+in your code:
+```python
+import neetbox
+```
+
+## usage examples
+
+[how to guides](todo) provides easy examples of basic neetbox funcionalities.
 
 ## Star History
 

doc/docs/develop/daemon/index.md

@@ -0,0 +1,140 @@
+# Testing DAEMON
+
+NEETBOX daemon consists of client side and server side. While client side syncs status of running project and platform information including hardware, server side provides apis for status monitoring and websocket forcasting between client and frontends.
+
+Basically neetbox will also launch a backend on localhost when a project launching configured with daemon server address at localhost. The server will run in background without any output, and you may want to run a server with output for debug purposes.  
+
+## How to test neetbox server
+
+at neetbox project root:
+
+```bash
+python neetbox/daemon/server/_server.py
+```
+
+script above should launch a server in debug mode on `localhost:5000`, it wont read the port in `neetbox.toml`. a swegger UI is provided at [localhost:5000/docs](http://127.0.0.1:5000/docs) in debug mode. websocket server should run on port `5001`.
+
+If you want to simulate a basic neetbox client sending message to server, at neetbox project root:
+```bash
+cd tests/client
+python test.py
+```
+script above should launch a simple case of neetbox project with some logs and status sending to server.
+
+## Websocket message standard
+
+websocke messages are described in json. There is a dataclass representing websocket message:
+
+```python
+@dataclass
+class WsMsg:
+    event_type: str
+    payload: Any
+    event_id: int = -1
+
+    def json(self):
+        return {
+            EVENT_TYPE_NAME_KEY: self.event_type,
+            EVENT_ID_NAME_KEY: self.event_id,
+            PAYLOAD_NAME_KEY: self.payload,
+        }
+```
+
+```json
+{
+    "event-type" : ...,
+    "payload" : ...,
+    "event-id" : ...
+}
+```
+
+|    key     | value type |                      description                       |
+| :--------: | :--------: | :----------------------------------------------------: |
+| event-type |   string   |            indicate type of data in payload            |
+|  payload   |   string   |                      actual data                       |
+|  event-id  |    int     | for events who need ack. default -1 means no event id. |
+
+## Event types
+
+the table is increasing. a frequent check would keep you up to date.
+
+| event-type |      accepting direction      |                            means                             |
+| :--------: | :---------------------------: | :----------------------------------------------------------: |
+| handshake  | cli <--> server <--> frontend |  string in `payload` indicate connection type ('cli'/'web')  |
+|    log     |   cli -> server -> frontend   |                 `payload` contains log data                  |
+|   action   |   cli <- server <- frontend   |              `payload` contains action trigger               |
+|    ack     | cli <--> server <--> frontend | `payload` contains ack, and `event-id` should be a valid key |
+
+## Examples of websocket data
+
+### handshake
+
+for instance, frontend connected to server. frontend should report connection type immediately by sending:
+
+```json
+{
+  "event-type": "handshake",
+  "name": "project name",
+  "payload": {
+    "who": "web"
+  },
+  "event-id": X
+}
+```
+
+where `event-id` is used to send ack to the starter of the connection, it should be a random int value.
+
+### cli sending log to frontend
+
+cli sents log(s) via websocket, server will receives and broadcast this message to related frontends. cli should send:
+
+```json
+{
+  "event-type": "log",
+  "name": "project name",
+  "payload": {
+    "log" : {...json representing log data...}
+  },
+  "event-id": -1
+}
+```
+
+where `event-id` is a useless segment, leave it default. it's okay if nobody receives log.
+
+### frontend(s) querys action to cli
+
+frontend send action request to server, and server will forwards the message to cli. frontend should send:
+
+```json
+{
+    "event-type" : "action",
+    "name": "project name",
+    "payload" : {
+      "action" : {...json representing action trigger...}
+    },
+    "event-id" : x
+}
+```
+
+front may want to know the result of action. for example, whether the action was invoked successfully. therefore, `event-id` is necessary for cli to shape a ack response.
+
+### cli acks frontend action query
+
+cli execute action query(s) from frontend, and gives response by sending ack:
+
+```json
+{
+    "event-type" : "ack",
+    "name": "project name",
+    "payload" : {
+      "action" : {...json representing action result...}
+    },
+    "event-id" : x
+}
+```
+
+where `event-id` is same as received action query.
+
+---
+
+Those are only examples. use them wisely.

doc/docs/guide/index.md

@@ -10,12 +10,6 @@ sidebar_position: 1
 pip install neetbox
 ```
 
-If you want to enable torch-related feature, please try below:
-
-```bash
-pip install neetbox[torch]
-```
-
 Since NEETBOX is under ~~heavy~~ development, it's better to forcely reinstall the newest version:
 
 ```bash

neetbox/__init__.py

@@ -12,18 +12,6 @@
 config_file_name = f"{module}.toml"
 
 
-def post_init():
-    import setproctitle
-
-    project_name = get_module_level_config()["name"]
-    setproctitle.setproctitle(project_name)
-
-    from neetbox.daemon.client._connection import connection
-
-    # post init ws
-    connection._init_ws()
-
-
 def init(path=None, load=False, **kwargs) -> bool:
     if path:
         os.chdir(path=path)
@@ -77,11 +65,24 @@ def init(path=None, load=False, **kwargs) -> bool:
         raise e
 
 
+def post_init():
+    import setproctitle
+
+    project_name = get_module_level_config()["name"]
+    setproctitle.setproctitle(project_name)
+
+    from neetbox.daemon.client._client import connection
+
+    # post init ws
+    connection._init_ws()
+
+
 is_in_daemon_process = (
     "NEETBOX_DAEMON_PROCESS" in os.environ and os.environ["NEETBOX_DAEMON_PROCESS"] == "1"
 )
 # print('prevent_daemon_loading =', is_in_daemon_process)
 if os.path.isfile(config_file_name) and not is_in_daemon_process:  # if in a workspace
+    # todo check if running in cli mode
     success = init(load=True)  # init from config file
     if not success:
         os._exit(255)

neetbox/daemon/__init__.py

@@ -9,8 +9,8 @@
 import time
 
 from neetbox.daemon.client._action_agent import _NeetActionManager as NeetActionManager
-from neetbox.daemon.client._connection import connection
-from neetbox.daemon.client._daemon_client import connect_daemon
+from neetbox.daemon.client._client import connection
+from neetbox.daemon.client._update_thread import connect_daemon
 from neetbox.daemon.server.daemonable_process import DaemonableProcess
 from neetbox.logging import logger
 from neetbox.pipeline import listen, watch

neetbox/daemon/_protocol.py

@@ -0,0 +1,27 @@
+from dataclasses import dataclass
+from typing import Any
+
+FRONTEND_API_ROOT = "/web"
+CLIENT_API_ROOT = "/cli"
+
+
+EVENT_TYPE_NAME_KEY = "event-type"
+EVENT_ID_NAME_KEY = "event-id"
+NAME_NAME_KEY = "name"
+PAYLOAD_NAME_KEY = "payload"
+
+
+@dataclass
+class WsMsg:
+    name: str
+    event_type: str
+    payload: Any
+    event_id: int = -1
+
+    def json(self):
+        return {
+            NAME_NAME_KEY: self.name,
+            EVENT_TYPE_NAME_KEY: self.event_type,
+            EVENT_ID_NAME_KEY: self.event_id,
+            PAYLOAD_NAME_KEY: self.payload,
+        }

neetbox/daemon/client/_action_agent.py

@@ -5,6 +5,8 @@
 from typing import Callable, Optional
 
 from neetbox.core import Registry
+from neetbox.daemon._protocol import *
+from neetbox.daemon.client._client import connection
 from neetbox.logging import logger
 from neetbox.pipeline import watch
 from neetbox.utils.mvc import Singleton
@@ -41,7 +43,7 @@ def get_action_dict():
         action_names = _NeetActionManager.__ACTION_POOL.keys()
         for name in action_names:
             action = _NeetActionManager.__ACTION_POOL[name]
-            action_dict[name] = action.argspec.args
+            action_dict[name] = {"args": action.argspec.args, "blocking": action.blocking}
         return action_dict
 
     def eval_call(name: str, params: dict, callback: None):
@@ -83,6 +85,21 @@ def _register(function: Callable, name: str = None, blocking: bool = False):
         return function
 
 
+@connection.ws_subscribe(event_type_name="action")
+def __listen_to_actions(msg):
+    _payload = msg[PAYLOAD_NAME_KEY]
+    _event_id = msg[EVENT_ID_NAME_KEY]
+    _action_name = _payload["name"]
+    _action_args = _payload["args"]
+    _NeetActionManager.eval_call(
+        name=_action_name,
+        params=_action_args,
+        callback=lambda x: connection.ws_send(
+            event_type="action", payload={"name": _action_name, "result": x}, _event_id=_event_id
+        ),
+    )
+
+
 # example
 if __name__ == "__main__":
     import time