How to shared memory between main process and plugin ?

[LoickZoty]

Hello !

I have my main file :

    def start_proxy(self, id: int, port: int, traffic_max: int):

        self.proxy_service.create(id)
        with proxy.Proxy(input_args=self.create_input_args(id, port, traffic_max)) as p:
            while self.proxy_service.proxies[id]:
                time.sleep(1)
            p.shutdown()

    def create_input_args(self, id: int, port: int, traffic_max: int) -> []:
        return [
            "--plugin", "plugin.dataSizePlugin.DataSizePlugin",
            "--proxy-id", str(id),
            "--port", str(port),
            "--max-upstream-size", str(traffic_max)
        ]

and my plugin

class DataSizePlugin(HttpProxyBasePlugin):
    def __init__(self, uid, flags, client, event_queue, upstream_conn_pool) -> None:
        super().__init__(uid, flags, client, event_queue, upstream_conn_pool)
        self.proxy_service = ProxyService()
        self.total_data_size = 0

    def before_upstream_connection(self, request: HttpParser) -> Optional[HttpParser]:
        self.proxy_service.delete(self.flags.proxy_id)
        return request

I want my plugin edit a variable which will allow to the main process to shutdown proxy.

I already read this https://github.com/abhinavsingh/proxy.py/blob/develop/tutorial/eventing.ipynb but i don't understand how i can make this.

Please help by directing me to achieve this.

[abhinavsingh]

I want my plugin edit a variable which will allow to the main process to shutdown proxy.

I think for this purpose you can just use multiprocessing.Event. Give it a try and lemme know.

Purpose of eventing is mostly to communicate information between processes rather than sharing a global variable.

[LoickZoty]

Initially, I opted for this solution: https://docs.python.org/3/library/multiprocessing.shared_memory.html.
This allowed me to share my variables directly from the buffer.

However, I became interested in your solution, which gave me the following result.

app.py

def start_proxy(self, p: ProxyModel):
    p.started = True
    with proxy.Proxy(input_args=self.create_input_args(p)) as p_process:
        while proxy_service.proxies[p.id] and proxy_service.proxies[p.id].started:
            self.update_proxy(p, p_process.event_manager.queue.queue)
            self.stop_proxy(p)

def update_proxy(self, p: ProxyModel, queue: Queue):
    try:
        update = queue.get(timeout=1)
        if update["event_name"] == eventNames.TRAFFIC_IN:
            p.traffic_in_current += update["chunk"]

    except Empty:
        pass

plugin.py

def handle_upstream_chunk(self, chunk):
    self.event_queue.queue.put({"event_name": eventNames.TRAFFIC_IN, "chunk": len(chunk)})
    return chunk

This solution is more concise and does not require managing variable synchronization; it is simply better!

[abhinavsingh]

Totally agreed. Myself I would have just used primitives provided by proxy.py. Since, your described use case was to signal restart, a multiprocessing event or shared memory would have also worked.

Overall your implementation looks good to me. Ideally, developers should see sleep_loop as a method they can "override" and "customise" to do several things.

[abhinavsingh]

One problem I see with build in eventing mechanism is, in-ability to provide/declare custom event names, which currently is a hardcoded enum IIRC. We'll try to (somehow) address this limitation in future releases. Otherwise, currently, you must have been forced to change a piece of code in the core to add eventNames.TRAFFIC_IN.