Can't directly run cpp files in lux-ai-2021 (ETXTBSY error) #71

Desperationis · 2021-07-04T16:37:07Z

So I've heard, you can run cpp files directly into lux-ai-2021. However, when I try to run it with my own bot, I get this error. https://pastebin.com/CQDev0Kw error in its entirety.

diego@adhoc:~/Desktop/LuxBot$ sudo lux-ai-2021 src/main.cpp src/main.cpp 

-=-=-=-=-=-=-=-=-=-=-=-| [INFO] match_QiUDBKVEPU7A |-=-=-=-=-=-=-=-=-=-=-=-
[INFO] (match_QiUDBKVEPU7A) - Design: lux_ai_2021 | Initializing match - ID: QiUDBKVEPU7A, Name: match_QiUDBKVEPU7A
Error: spawn ETXTBSY
    at ChildProcess.spawn (internal/child_process.js:403:11)
    at spawn (child_process.js:580:9)
    at Object.spawnWithSignal [as spawn] (child_process.js:717:17)
    at Object.spawn [as default] (/usr/local/lib/node_modules/@lux-ai/2021-challenge/node_modules/cross-spawn/index.js:12:24)
    at /usr/local/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Agent/index.js:653:46
    at new Promise (<anonymous>)
    at Agent._spawnProcess (/usr/local/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Agent/index.js:646:16)
    at Agent.<anonymous> (/usr/local/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Agent/index.js:628:56)
    at step (/usr/local/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Agent/index.js:46:23)
    at Object.next (/usr/local/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Agent/index.js:27:53) {
  errno: -26,
  code: 'ETXTBSY',
  syscall: 'spawn'
}

I get this error when trying to run the v1.1.x branch of the cpp kit as well; Not sure what is causing it. It works fine when transpiled into JS. I've tried running the lux-ai-2021 command as sudo and not sudo, though it doesn't make a difference. Here's the exact main.cpp code I tried to run in simple/:

#include "lux/kit.hpp"
#include "lux/define.cpp"
#include <string.h>
#include <vector>
#include <set>
#include <stdio.h>

using namespace std;
using namespace lux;
int main()
{
  kit::Agent gameState = kit::Agent();
  gameState.initialize();

  while (true)
  {
    gameState.update();

    vector<string> actions = vector<string>();
    Player player = gameState.players[gameState.id];
    Player opponent = gameState.players[(gameState.id + 1) % 2];

    GameMap gameMap = gameState.map;

    vector<Cell *> resourceTiles = vector<Cell *>();
    for (int y = 0; y < gameMap.height; y++)
    {
      for (int x = 0; x < gameMap.width; x++)
      {
        Cell *cell = gameMap.getCell(x, y);
        if (cell->hasResource())
        {
          resourceTiles.push_back(cell);
        }
      }
    }

    int citiesToBuild = 0;
    for (auto it : player.cities)
    {
      City *city = it.second;
      if (city->fuel > city->getLightUpkeep() * (int) GAME_CONSTANTS["PARAMETERS"]["NIGHT_LENGTH"] + 1000)
      {
        citiesToBuild += 1;
      }
      for (auto citytile : city->citytiles)
      {
        if (citytile->canAct()) {
          // you can use the following to get the citytile to research or build a worker
          // actions.push_back(citytile.research());
          // actions.push_back(citytile.buildWorker());
        }
      }
    }

    for (int i = 0; i < player.units.size(); i++)
    {
      Unit unit = player.units[i];
      if (unit.isWorker() && unit.canAct())
      {
        if (unit.getCargoSpaceLeft() > 0)
        {
          // if the unit is a worker and we have space in cargo, lets find the nearest resource tile and try to mine it
          Cell *closestResourceTile;
          float closestDist = 9999999;
          for (auto it = resourceTiles.begin(); it != resourceTiles.end(); it++)
          {
            auto cell = *it;
            if (cell->resource.type == ResourceType::coal && !player.researchedCoal()) continue;
            if (cell->resource.type == ResourceType::uranium && !player.researchedUranium()) continue;
            float dist = cell->pos.distanceTo(unit.pos);
            if (dist < closestDist)
            {
              closestDist = dist;
              closestResourceTile = cell;
            }
          }
          if (closestResourceTile != nullptr)
          {
            auto dir = unit.pos.directionTo(closestResourceTile->pos);
            actions.push_back(unit.move(dir));
          }
        }
        else
        {
          if (player.cities.size() > 0)
          {
            auto city_iter = player.cities.begin();
            auto city = city_iter->second;

            float closestDist = 999999;
            CityTile *closestCityTile;
            for (auto citytile : city->citytiles)
            {
              float dist = citytile->pos.distanceTo(unit.pos);
              if (dist < closestDist)
              {
                closestCityTile = citytile;
                closestDist = dist;
              }
            }
            if (closestCityTile != nullptr)
            {
              auto dir = unit.pos.directionTo(closestCityTile->pos);

              if (citiesToBuild > 0 && unit.pos.isAdjacent(closestCityTile->pos) && unit.canBuild(gameMap))
              {
                actions.push_back(unit.buildCity());
              }
              else
              {
                actions.push_back(unit.move(dir));
              }
            }
          }
        }
      }
    }

    for (int i = 0; i < actions.size(); i++)
    {
      if (i != 0)
        cout << ",";
      cout << actions[i];
    }
    cout << endl;
    gameState.end_turn();
  }
  return 0;
}

The text was updated successfully, but these errors were encountered:

Desperationis · 2021-07-04T16:53:38Z

Replays show nothing special. When run in the same seed as their JS counterparts, they result in the exact same behavior.

Desperationis · 2021-07-04T16:58:41Z

I've attached the simple bot I've been trying to compile below. The exact command I used was sudo lux-ai-2021 main.cpp main.cpp
simple.zip

I should mention that I'm on Linux Mint btw, so this error might not appear on Windows

Desperationis · 2021-07-04T17:34:19Z

After rigorous testing, I am more confused than I was before. I spammed that command on a specific seed for a single bot and it still failed every so often, meaning the chance of this issue occurring is completely random and not dependent on seed or the bot run. Wth

StoneT2000 · 2021-07-05T00:27:10Z

The ETXTBSY error is unrelated to the game engine and is something else. Works fine on my machine. Please stay tuned to this thread, may ask you to test some things.

StoneT2000 · 2021-07-16T00:23:20Z

Just making sure this isn't stale, is this still and issue @Desperationis ?

Desperationis · 2021-07-16T01:32:00Z

It's still a issue, not sure what's causing it. What I know for certain though is that running the lux-ai command multiple times on the same seed will eventually run the match

StoneT2000 · 2021-07-16T11:34:41Z

Oh ok so this error means you are trying to modify an executable while it is already running which is a weird error.

StoneT2000 · 2021-07-16T11:35:55Z

Do you by any chance have the executable file opened for some reason.

Additionally can you compile your C++ as you would and then execute from the command line yourself and not via lux ai (it should just hang)

Desperationis · 2021-07-16T16:15:07Z

No, I don't. In fact, even if I run the executable compiled with g++ wiht lux-ai, it has no effect on the behaviour

StoneT2000 · 2021-07-21T17:23:49Z

@Desperationis Ok so this issue is not even consistently happening either, hard to reproduce, which leads me to think there's something up with your setup.

Can you tell me what node and npm version you are on and what linux mint version you are on?

I think worst case, use docker to run matches (we can provide a simple script for this) but hopefully that won't be necessary.

Desperationis · 2021-07-21T17:45:24Z

I'm running node v14.17.2 and npm v7.19.1 on Linux Mint Cinnamon Uma, though this happened on Ulyssa as well.

StoneT2000 · 2021-07-24T04:42:39Z

still not sure sorry.

I found this thread where someone else was using a nodejs application for something else and got the same error:
alixaxel/chrome-aws-lambda#69

Still not exactly sure what's going on. Maybe permissions? In which case can you do ls -la in the folder with the main.cpp file?

Desperationis · 2021-07-24T19:58:56Z

This is before running lux with main.cpp:

total 44
drwx------  4 diego diego 4096 Jul 23 20:30 .
drwxr-xr-x 16 diego diego 4096 Jul 24 12:46 ..
-rwxr-xr-x  1 diego diego  105 Jul 23 20:30 compile.bat
-rwxr-xr-x  1 diego diego  118 Jul 23 20:30 compile.sh
drwx------  3 diego diego 4096 Jul 23 20:30 internals
drwx------  3 diego diego 4096 Jul 23 20:30 lux
-rw-rw-r--  1 diego diego 4441 Jul 23 20:30 main.cpp
-rw-rw-r--  1 diego diego 1928 Jul 23 20:30 main.py
-rw-rw-r--  1 diego diego  265 Jul 23 20:30 package.json
-rw-rw-r--  1 diego diego  383 Jul 23 20:30 package-lock.json

After a successful run right after:

total 212
drwx------  6 diego diego   4096 Jul 24 12:48 .
drwxr-xr-x 16 diego diego   4096 Jul 24 12:46 ..
-rwxr-xr-x  1 diego diego    105 Jul 23 20:30 compile.bat
-rwxr-xr-x  1 diego diego    118 Jul 23 20:30 compile.sh
drwxrwxr-x  2 diego diego   4096 Jul 24 12:48 errorlogs
drwx------  3 diego diego   4096 Jul 23 20:30 internals
drwx------  3 diego diego   4096 Jul 23 20:30 lux
-rw-rw-r--  1 diego diego   4441 Jul 23 20:30 main.cpp
-rwxrwxr-x  1 diego diego 162032 Jul 24 12:48 main.out
-rw-rw-r--  1 diego diego   1928 Jul 23 20:30 main.py
-rw-rw-r--  1 diego diego    265 Jul 23 20:30 package.json
-rw-rw-r--  1 diego diego    383 Jul 23 20:30 package-lock.json
drwxrwxr-x  2 diego diego   4096 Jul 24 12:48 replays

After a bad run with the EXTBSY error:

total 212
drwx------  6 diego diego   4096 Jul 24 12:49 .
drwxr-xr-x 16 diego diego   4096 Jul 24 12:46 ..
-rwxr-xr-x  1 diego diego    105 Jul 23 20:30 compile.bat
-rwxr-xr-x  1 diego diego    118 Jul 23 20:30 compile.sh
drwxrwxr-x  2 diego diego   4096 Jul 24 12:49 errorlogs
drwx------  3 diego diego   4096 Jul 23 20:30 internals
drwx------  3 diego diego   4096 Jul 23 20:30 lux
-rw-rw-r--  1 diego diego   4441 Jul 23 20:30 main.cpp
-rwxrwxr-x  1 diego diego 162032 Jul 24 12:49 main.out
-rw-rw-r--  1 diego diego   1928 Jul 23 20:30 main.py
-rw-rw-r--  1 diego diego    265 Jul 23 20:30 package.json
-rw-rw-r--  1 diego diego    383 Jul 23 20:30 package-lock.json
drwxrwxr-x  2 diego diego   4096 Jul 24 12:48 replays

As a test, I ran sudo chmod +x on main.py, main.cpp, and main.out and still got the EXTBSY error. I did the same thing to only main.py and main.cpp on a fresh run of simple and got the same result.

StoneT2000 · 2021-07-27T14:15:26Z

Hi @Desperationis can you try this script as a test. Put this script into the same directory you call lux-ai-2021 from. Replace line 2 with the correct CWD. (so remove 'path/to/dir/of/main.out').

const { spawn } = require('child_process');
const p = spawn('./main.out', {cwd: 'path/to/dir/of/main.out');

p.stdout.on('data', (data) => {
  console.log(`stdout: ${data}`);
});

p.stderr.on('data', (data) => {
  console.error(`stderr: ${data}`);
});

p.on('close', (code) => {
  console.log(`child process exited with code ${code}`);
});

let me know if the EXTBSY error pops up.

Desperationis · 2021-07-27T19:06:07Z

I tried the JS script with node and the program just hung; It didn't produce any output on stdout or stderr, and it was like if you just ran main.out raw.

Btw you missed a } on line 2 of the code.

Desperationis · 2021-07-27T19:06:28Z

But yeah, no EXTBSY error. Idk if this piece of information is useful or anything but running main.out raw from the command line never produces the error, though it doesn't produce any output either.

StoneT2000 · 2021-07-28T20:33:30Z

ok this is helpful. It's supposed to just hang as the agent is waiting for input (match information and state) and didn't quit with a EXTBSY error. Small chance the bug may be caused by the cross-spawn package. Basically normally this is how we open your bot as shown in the stack trace

Error: spawn ETXTBSY
    at ChildProcess.spawn (internal/child_process.js:403:11)
    at spawn (child_process.js:580:9)

that just called the same command.

StoneT2000 · 2021-07-28T20:34:57Z

A safe solution is to just use the kaggle-environments match running tool, although less recommended but still usable and probably doesn't break. Information on using that will be released later.

Desperationis · 2021-07-28T20:55:02Z

Alright, got it

StoneT2000 · 2021-08-08T00:57:31Z

@Desperationis Can you test our latest dockerized version of the lux-ai-2021 tool? there's instructions here: https://github.com/Lux-AI-Challenge/Lux-Design-2021#cli-docker

It matches the lux-ai-2021 tool 1-1 (also please use the compile.sh tool in the new C++ starter kit and pass in main.out as the bot files)

So copy over the cli.sh file and simply run

sh cli.sh src/main.out src/main.out and you should be good to go.

Desperationis · 2021-08-10T03:28:33Z

@StoneT2000 I tried your docker image and I wasn't able to have the container read neither the executable or source file. Keep in mind I have very little docker experience. The script worked fine, though it only worked with bash cli.sh and not sh cli.sh on my system; Not sure why. You might also want to add the sudo prefix on the command tbh. Lmk what I should I do about this; The error is here:

diego@adhoc:~/Desktop/Lux-Design-2021-master/kits/cpp/simple$ sudo bash cli.sh src/main.out src/main.out 

-=-=-=-=-=-=-=-=-=-=-=-| [INFO] match_VKyPzeUzVHhO |-=-=-=-=-=-=-=-=-=-=-=-

[INFO] (match_VKyPzeUzVHhO) - Design: lux_ai_2021 | Initializing match - ID: VKyPzeUzVHhO, Name: match_VKyPzeUzVHhO
AgentFileError: src/main.out does not exist, check if file path provided is correct
    at AgentFileError.AgentError [as constructor] (/usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/DimensionError/AgentError/index.js:31:28)
    at new AgentFileError (/usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/DimensionError/AgentError/index.js:124:28)
    at new Agent (/usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Agent/index.js:176:23)
    at /usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Agent/index.js:1000:29
    at Array.forEach (<anonymous>)
    at Function.Agent.generateAgents (/usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Agent/index.js:995:19)
    at Match.<anonymous> (/usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Match/index.js:221:53)
    at step (/usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Match/index.js:33:23)
    at Object.next (/usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Match/index.js:14:53)
    at fulfilled (/usr/local/nvm/versions/node/v14.16.0/lib/node_modules/@lux-ai/2021-challenge/node_modules/dimensions-ai/lib/main/Match/index.js:5:58) {
  agentID: 0
}

StoneT2000 · 2021-08-10T03:30:10Z

Yes you need bash, I'll update the readme later.

So you are saying you still can't run a game when using bash cli.sh?

Desperationis · 2021-08-10T04:34:02Z

Yeah, but hey, at least it's not the same error. All I know is that the docker pseudo-vm cannot find the executable or cpp file in my local machine.compile.sh works completely fine with no errors, so I think it's cli.sh parameters that are causing me issue. No matter what I throw at it, I get the same "no such file or directory" error.

Probably due to the bind mount not working correctly, or just me not using the script correctly. Here, main.out is a legitimate file created by compile.sh.

djkeyes · 2021-08-14T20:50:10Z

Chiming in here that I am able to reproduce this issue. Here are some results that I found:

bash cli.sh ./kits/cpp/simple/main.cpp ./kits/cpp/simple/main.cpp
This command reproduces the issue about 50% of the time. pastebin of the output. The other 50% of the time, it runs successfully.

bash cli.sh ./kits/cpp/simple/main.cpp ./kits/python/simple/main.py
This command always works for me (cpp starter + python starter).

bash cli.sh ./kits/cpp/simple/main.out ./kits/cpp/simple/main.out
If main.out exists, this command always works for me (precompiled cpp starter + precompiled cpp starter).

This is a stab in the dark, but could there be an issue with two compilation processes trying to write to the same file? Maybe agent 0 is trying to execute main.out, but agent 1 is trying to write to main.out. I'm not exactly sure how the game engine runs under the hood.

Desperationis · 2021-08-14T22:24:30Z

@djkeyes That is very, very peculiar. Personally, I never encountered the ETXTBSY error again once I used the docker container, only the ENOENT error mentioned above but me and @StoneT2000 were able to resolve that. Maybe check sudo docker ps to see if you're running two instances of the docker image? Keep in mind I only run with two executable files, not cpp files.

I think you're right though on the write-conflict. The command sees both files as individual files and tries to compile a main.out for each, then load it into memory. Because of this, if you try to compile two .cpp files at the same time, the program tries to make two main.out's at the same time (possibly) through multithreading, One might finish earlier before the other even begins, and have time to load it into memory before the other starts a new main.out file. What also might be happening is that the file library used to read each file may not close the file in time, also leading to different writing times. This explains the 50/50 random odds of you being able to compile.

This also explain the py-cpp and out-out combinations. py-cpp only needs to transpile a single cpp file, while the out-out combination can read it directly.

If this is indeed is what is happening, a possible solution for this is to simply name each transpiled executable either 1 or 2 depending on where it is in the parameters. This way, no confict occurs.

StoneT2000 · 2021-08-15T00:58:58Z

@djkeyes That is very, very peculiar. Personally, I never encountered the ETXTBSY error again once I used the docker container, only the ENOENT error mentioned above but me and @StoneT2000 were able to resolve that. Maybe check sudo docker ps to see if you're running two instances of the docker image? Keep in mind I only run with two executable files, not cpp files.

I think you're right though on the write-conflict. The command sees both files as individual files and tries to compile a main.out for each, then load it into memory. Because of this, if you try to compile two .cpp files at the same time, the program tries to make two main.out's at the same time (possibly) through multithreading, One might finish earlier before the other even begins, and have time to load it into memory before the other starts a new main.out file. What also might be happening is that the file library used to read each file may not close the file in time, also leading to different writing times. This explains the 50/50 random odds of you being able to compile.

This also explain the py-cpp and out-out combinations. py-cpp only needs to transpile a single cpp file, while the out-out combination can read it directly.

If this is indeed is what is happening, a possible solution for this is to simply name each transpiled executable either 1 or 2 depending on where it is in the parameters. This way, no confict occurs.

OH you two may be right! I never knew this could be an issue.

StoneT2000 · 2021-08-15T01:03:39Z

Chiming in here that I am able to reproduce this issue. Here are some results that I found:

bash cli.sh ./kits/cpp/simple/main.cpp ./kits/cpp/simple/main.cpp
This command reproduces the issue about 50% of the time. pastebin of the output. The other 50% of the time, it runs successfully.

bash cli.sh ./kits/cpp/simple/main.cpp ./kits/python/simple/main.py
This command always works for me (cpp starter + python starter).

bash cli.sh ./kits/cpp/simple/main.out ./kits/cpp/simple/main.out
If main.out exists, this command always works for me (precompiled cpp starter + precompiled cpp starter).

This is a stab in the dark, but could there be an issue with two compilation processes trying to write to the same file? Maybe agent 0 is trying to execute main.out, but agent 1 is trying to write to main.out. I'm not exactly sure how the game engine runs under the hood.

I believe you are exactly right! I think we will change the documentation to instead tell users to pass in main.out as the file instead of main.cpp.

Our CLI tool's backing engine has the problem here: https://github.com/StoneT2000/Dimensions/blob/master/src/MatchEngine/index.ts#L107

For each agent we initialize, if the file given is .cpp, we run a default C++ compilation command (in hindsight, not very smart, should let the user decide on how to compile it better, maybe add a option to submit a compile.sh file (barring security problems with that)). However, in the line above, each agent is asynchronously compiled lmao, so two processes are probably trying to write at the same time or something.

StoneT2000 · 2021-08-15T01:04:25Z

I'll leave this issue open in case others encounter this and i've added (ETXTBSY error) to the title so its more discoverable, thanks!

Desperationis mentioned this issue Jul 4, 2021

Fatal memory overflow in specific seeds in C++ #73

Closed

StoneT2000 added bug Something isn't working starter-kit anything related to kit updates labels Jul 6, 2021

StoneT2000 self-assigned this Jul 11, 2021

StoneT2000 changed the title ~~Can't directly run cpp files in lux-ai-2021~~ Can't directly run cpp files in lux-ai-2021 (ETXTBSY error) Aug 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't directly run cpp files in lux-ai-2021 (ETXTBSY error) #71

Can't directly run cpp files in lux-ai-2021 (ETXTBSY error) #71

Desperationis commented Jul 4, 2021 •

edited

Loading

Desperationis commented Jul 4, 2021

Desperationis commented Jul 4, 2021

Desperationis commented Jul 4, 2021

StoneT2000 commented Jul 5, 2021

StoneT2000 commented Jul 16, 2021

Desperationis commented Jul 16, 2021

StoneT2000 commented Jul 16, 2021

StoneT2000 commented Jul 16, 2021 •

edited

Loading

Desperationis commented Jul 16, 2021

StoneT2000 commented Jul 21, 2021

Desperationis commented Jul 21, 2021 •

edited

Loading

StoneT2000 commented Jul 24, 2021

Desperationis commented Jul 24, 2021

StoneT2000 commented Jul 27, 2021 •

edited

Loading

Desperationis commented Jul 27, 2021

Desperationis commented Jul 27, 2021 •

edited

Loading

StoneT2000 commented Jul 28, 2021

StoneT2000 commented Jul 28, 2021

Desperationis commented Jul 28, 2021

StoneT2000 commented Aug 8, 2021 •

edited

Loading

Desperationis commented Aug 10, 2021

StoneT2000 commented Aug 10, 2021

Desperationis commented Aug 10, 2021 •

edited

Loading

djkeyes commented Aug 14, 2021

Desperationis commented Aug 14, 2021 •

edited

Loading

StoneT2000 commented Aug 15, 2021 •

edited

Loading

StoneT2000 commented Aug 15, 2021

StoneT2000 commented Aug 15, 2021

Can't directly run cpp files in lux-ai-2021 (ETXTBSY error) #71

Can't directly run cpp files in lux-ai-2021 (ETXTBSY error) #71

Comments

Desperationis commented Jul 4, 2021 • edited Loading

Desperationis commented Jul 4, 2021

Desperationis commented Jul 4, 2021

Desperationis commented Jul 4, 2021

StoneT2000 commented Jul 5, 2021

StoneT2000 commented Jul 16, 2021

Desperationis commented Jul 16, 2021

StoneT2000 commented Jul 16, 2021

StoneT2000 commented Jul 16, 2021 • edited Loading

Desperationis commented Jul 16, 2021

StoneT2000 commented Jul 21, 2021

Desperationis commented Jul 21, 2021 • edited Loading

StoneT2000 commented Jul 24, 2021

Desperationis commented Jul 24, 2021

StoneT2000 commented Jul 27, 2021 • edited Loading

Desperationis commented Jul 27, 2021

Desperationis commented Jul 27, 2021 • edited Loading

StoneT2000 commented Jul 28, 2021

StoneT2000 commented Jul 28, 2021

Desperationis commented Jul 28, 2021

StoneT2000 commented Aug 8, 2021 • edited Loading

Desperationis commented Aug 10, 2021

StoneT2000 commented Aug 10, 2021

Desperationis commented Aug 10, 2021 • edited Loading

djkeyes commented Aug 14, 2021

Desperationis commented Aug 14, 2021 • edited Loading

StoneT2000 commented Aug 15, 2021 • edited Loading

StoneT2000 commented Aug 15, 2021

StoneT2000 commented Aug 15, 2021

Desperationis commented Jul 4, 2021 •

edited

Loading

StoneT2000 commented Jul 16, 2021 •

edited

Loading

Desperationis commented Jul 21, 2021 •

edited

Loading

StoneT2000 commented Jul 27, 2021 •

edited

Loading

Desperationis commented Jul 27, 2021 •

edited

Loading

StoneT2000 commented Aug 8, 2021 •

edited

Loading

Desperationis commented Aug 10, 2021 •

edited

Loading

Desperationis commented Aug 14, 2021 •

edited

Loading

StoneT2000 commented Aug 15, 2021 •

edited

Loading