Intermittent Failure to read all fans on Braiins OS+

[cryptographicturk]

Describe the bug
Approximately 0.5% of the time, when using pyasic to asynchronously read any number of miners, the fan data is truncated.

e.g. Here are several reads in a row where we saw the issue occur.
PyASIC: [{'speed': 4800}, {'speed': 4740}, {'speed': 4860}, {'speed': 4800}]
PyASIC: [{'speed': 4800}, {'speed': 4740}, {'speed': 4800}, {'speed': 4800}]
PyASIC: [{'speed': 4800}, {'speed': 4740}]
PyASIC: [{'speed': 4800}, {'speed': 4800}, {'speed': 4860}, {'speed': 4800}]
PyASIC: [{'speed': 4740}, {'speed': 4740}, {'speed': 4860}, {'speed': 4800}]
PyASIC: [{'speed': 4800}, {'speed': 4800}, {'speed': 4860}, {'speed': 4800}]

To Reproduce

import asyncio
import statistics
from pyasic import MinerData, get_miner

from datetime import datetime

pyasic_fans = []
pyasic_ets = []

async def get_fans_pyasic(semaphore):
    async with semaphore:
        fan_count = 0
        start_time = datetime.now()
        pyasic_miner = await get_miner("x.x.x.x")
        if pyasic_miner is None:
            return
        miner_data: MinerData = await pyasic_miner.get_data()
        miner = miner_data.as_dict()
        if miner is None:
            return
        if 'fans' in miner and miner['fans'] is not None:
            print(f"PyASIC: {miner['fans']}")
            for fan in miner['fans']:
                if 'speed' in fan:
                    if fan['speed'] is not None:
                        fan_count += 1
        pyasic_fans.append(fan_count)
        end_time = datetime.now()
        pyasic_ets.append((end_time - start_time).total_seconds())


async def test_pyasic():
    semaphore = asyncio.Semaphore(5)
    start_time = datetime.now()
    tasks = []
    for index in range(1500):
        tasks.append(get_fans_pyasic(semaphore))

    await asyncio.gather(*tasks)

    end_time = datetime.now()

    print(f"Total time: {(end_time - start_time).total_seconds()} seconds")
    print(f"PyASIC: {len(pyasic_fans)}, {statistics.mean(pyasic_fans)}, {statistics.mean(pyasic_ets)}")

Expected behavior
These should all come back every time.

Desktop (please complete the following information):

• OS: Ubuntu 22.04

Miner Information (If applicable):

• Multiple Braiins OS+ firmware version on multiple hardware.

Additional context
We used the same code to validate it against querying the CGMiner API and the GRPC API to eliminate a problem with the miner. Those APIs succeeded to get the correct data every time.

[b-rowan]

This is similar to something I see with BOS+ on S9s, where occasionally when repeatedly querying data, data for one of the hashboards will be completely omitted from the API response. This makes me wonder if the API result is getting truncated for some reason (i.e. the response is too long, as the get_data call does use + delimited multicommands). In theory this shouldn't be a problem for gRPC, as those commands get sent separately, but it seems like the only way to troubleshoot this may be to log the response the API is getting, here -

pyasic/pyasic/miners/backends/braiins_os.py

Line 882 in 0b69fe5

[b-rowan]

Just got around to testing this with an S9, I can't get it to reproduce... You can try checking if len(data["fans"]) == data["expected_fans"] and just not wait until the next piece of data to update possibly?

[b-rowan]

The only other thing I could imagine here is that the miner isn't being identified properly, you are getting the miner each time instead of storing the instance (I slightly modified your script to store the instance).

Here is an updated script, can you see if this problem occurs with this version?

import asyncio
import statistics
from pyasic import MinerData, get_miner

from datetime import datetime

pyasic_fans = []
pyasic_ets = []


async def get_fans_pyasic(pyasic_miner, semaphore):
    async with semaphore:
        fan_count = 0
        start_time = datetime.now()
        miner_data: MinerData = await pyasic_miner.get_data()
        miner = miner_data.as_dict()
        if miner is None:
            return
        if not len(miner_data.fans) == pyasic_miner.expected_fans:
            print("ERR")
        if "fans" in miner and miner["fans"] is not None:
            print(f"PyASIC: {miner['fans']}")
            for fan in miner["fans"]:
                if "speed" in fan:
                    if fan["speed"] is not None:
                        fan_count += 1
        pyasic_fans.append(fan_count)
        end_time = datetime.now()
        pyasic_ets.append((end_time - start_time).total_seconds())


async def test_pyasic():
    semaphore = asyncio.Semaphore(5)
    start_time = datetime.now()
    tasks = []
    pyasic_miner = await get_miner("10.0.1.62")

    for index in range(1500):
        tasks.append(get_fans_pyasic(pyasic_miner, semaphore))

    await asyncio.gather(*tasks)

    end_time = datetime.now()

    print(f"Total time: {(end_time - start_time).total_seconds()} seconds")
    print(
        f"PyASIC: {len(pyasic_fans)}, {statistics.mean(pyasic_fans)}, {statistics.mean(pyasic_ets)}"
    )


if __name__ == "__main__":
    asyncio.run(test_pyasic())