{2023.06}[system] CUDA v12.1.1 #349

trz42 · 2024-05-04T20:03:05Z

Adding CUDA to NESSI. Follows EESSI PR EESSI#434

License: NVIDIA

Missing packages:

1 out of 1 required modules missing:

* CUDA/12.1.1 (CUDA-12.1.1.eb)

nessi-bot · 2024-05-04T20:03:09Z

Instance AWS-MC-NESSI is configured to build:

arch x86_64/generic for repo nessi-2023.06-cl
arch x86_64/generic for repo nessi-2023.06-swl-deb10
arch x86_64/generic for repo nessi-2023.06-swl-deb11
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-cl
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-swl-deb10
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-swl-deb11
arch x86_64/amd/zen2 for repo nessi-2023.06-cl
arch x86_64/amd/zen2 for repo nessi-2023.06-swl-deb10
arch x86_64/amd/zen2 for repo nessi-2023.06-swl-deb11
arch aarch64/generic for repo nessi-2023.06-cl
arch aarch64/generic for repo nessi-2023.06-swl-deb10
arch aarch64/generic for repo nessi-2023.06-swl-deb11

nessi-bot · 2024-05-04T20:03:09Z

Instance eX3-NESSI is configured to build:

arch x86_64/amd/zen2 for repo nessi-2023.06-cl
arch x86_64/amd/zen2 for repo nessi-2023.06-swl-deb10
arch x86_64/amd/zen2 for repo nessi-2023.06-swl-deb11
arch aarch64/generic for repo nessi-2023.06-cl
arch aarch64/generic for repo nessi-2023.06-swl-deb10
arch aarch64/generic for repo nessi-2023.06-swl-deb11

nessi-bot · 2024-05-04T20:03:09Z

Instance Fram-NESSI is configured to build:

arch x86_64/generic for repo nessi-2023.06-cl
arch x86_64/generic for repo nessi-2023.06-swl-deb10
arch x86_64/generic for repo nessi-2023.06-swl-deb11
arch x86_64/intel/broadwell for repo nessi-2023.06-cl
arch x86_64/intel/broadwell for repo nessi-2023.06-swl-deb10
arch x86_64/intel/broadwell for repo nessi-2023.06-swl-deb11

nessi-bot · 2024-05-04T20:03:09Z

Instance Saga-NESSI is configured to build:

arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-cl
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-swl-deb10
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-swl-deb11
arch x86_64/intel/broadwell for repo nessi-2023.06-cl
arch x86_64/intel/broadwell for repo nessi-2023.06-swl-deb10
arch x86_64/intel/broadwell for repo nessi-2023.06-swl-deb11
arch x86_64/generic for repo nessi-2023.06-cl
arch x86_64/generic for repo nessi-2023.06-swl-deb10
arch x86_64/generic for repo nessi-2023.06-swl-deb11

trz42 · 2024-05-04T20:03:21Z

bot: build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell
bot: build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake
bot: build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic
bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2
bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic

nessi-bot · 2024-05-04T20:03:24Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic resulted in:
- submitted job 186140, for details & status see {2023.06}[system] CUDA v12.1.1 #349 (comment)
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-05-04T20:03:24Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- submitted job 10167, for details & status see {2023.06}[system] CUDA v12.1.1 #349 (comment)
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- submitted job 10168, for details & status see {2023.06}[system] CUDA v12.1.1 #349 (comment)

nessi-bot · 2024-05-04T20:03:24Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- submitted job 5790573, for details & status see {2023.06}[system] CUDA v12.1.1 #349 (comment)
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-05-04T20:03:24Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb10 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- submitted job 11397842, for details & status see {2023.06}[system] CUDA v12.1.1 #349 (comment)
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb10 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-05-04T20:03:28Z

New job on instance Fram-NESSI for architecture x86_64-intel-broadwell for repository nessi-2023.06-swl-deb11 in job dir /cluster/projects/nn9992k/pilot.nessi.no/jobs/2024.05/pr_349/5790573

date	job status	comment
May 04 20:03:27 UTC 2024	submitted	job id `5790573` awaits release by job manager
May 04 20:04:17 UTC 2024	released	job awaits launch by Slurm scheduler
May 04 20:46:22 UTC 2024	finished	🤷 UNKNOWN (click triangle for details) Job results file `_bot_job5790573.result` does not exist in job directory, or parsing it failed. No artefacts were found/reported.
May 04 20:46:22 UTC 2024	test result	🤷 UNKNOWN (click triangle for details) Job test file `_bot_job5790573.test` does not exist in job directory, or parsing it failed.

nessi-bot · 2024-05-04T20:03:29Z

New job on instance AWS-MC-NESSI for architecture x86_64-amd-zen2 for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.05/pr_349/10167

date	job status	comment
May 04 20:03:29 UTC 2024	submitted	job id `10167` awaits release by job manager
May 04 20:03:40 UTC 2024	released	job awaits launch by Slurm scheduler
May 04 20:09:00 UTC 2024	running	job `10167` is running
May 04 20:36:21 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-10167.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1714853709.tar.gz` size: 1004 MiB (1053710530 bytes) entries: 5518 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all `CUDA/12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/software `CUDA/12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2 no other files in tarball
May 04 20:36:21 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-10167.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 04 20:53:43 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-1714853709.tar.gz` to S3 bucket succeeded
May 04 08:57:59 PM UTC 2024	staged	tarball `eessi-2023.06-software-linux-x86_64-amd-zen2-1714853709.tar.gz` downloaded to Stratum-0
May 04 08:59:14 PM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/266 to approve ingest
May 04 09:06:08 PM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-x86_64-amd-zen2-1714853709.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/266
May 04 09:13:09 PM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-x86_64-amd-zen2-1714853709.tar.gz` successfully ingested at `2023.06/software/linux/x86_64/amd/zen2/`

nessi-bot · 2024-05-04T20:03:30Z

New job on instance eX3-NESSI for architecture aarch64-generic for repository nessi-2023.06-swl-deb10 in job dir /home/thomarob/pilot.nessi.no/jobs/2024.05/pr_349/186140

date	job status	comment
May 04 08:03:29 PM UTC 2024	submitted	job id `186140` awaits release by job manager
May 04 08:03:58 PM UTC 2024	released	job awaits launch by Slurm scheduler
May 04 08:05:03 PM UTC 2024	running	job `186140` is running
May 04 08:40:53 PM UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-186140.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-generic-1714853413.tar.gz` size: 996 MiB (1045233583 bytes) entries: 3777 modules under 2023.06/software/linux/aarch64/generic/modules/all `CUDA/12.1.1.lua` software under 2023.06/software/linux/aarch64/generic/software `CUDA/12.1.1` other under 2023.06/software/linux/aarch64/generic no other files in tarball
May 04 08:40:53 PM UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-186140.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 04 08:53:55 PM UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-aarch64-generic-1714853413.tar.gz` to S3 bucket succeeded
May 04 08:57:52 PM UTC 2024	staged	tarball `eessi-2023.06-software-linux-aarch64-generic-1714853413.tar.gz` downloaded to Stratum-0
May 04 08:58:48 PM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/265 to approve ingest
May 04 09:06:05 PM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-aarch64-generic-1714853413.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/265
May 04 09:09:48 PM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-aarch64-generic-1714853413.tar.gz` successfully ingested at `2023.06/software/linux/aarch64/generic/`

nessi-bot · 2024-05-04T20:03:33Z

New job on instance AWS-MC-NESSI for architecture x86_64-generic for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.05/pr_349/10168

date	job status	comment
May 04 20:03:32 UTC 2024	submitted	job id `10168` awaits release by job manager
May 04 20:03:42 UTC 2024	released	job awaits launch by Slurm scheduler
May 04 20:07:57 UTC 2024	running	job `10168` is running
May 04 20:42:52 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-10168.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1714853699.tar.gz` size: 1004 MiB (1053697110 bytes) entries: 5518 modules under 2023.06/software/linux/x86_64/generic/modules/all `CUDA/12.1.1.lua` software under 2023.06/software/linux/x86_64/generic/software `CUDA/12.1.1` other under 2023.06/software/linux/x86_64/generic no other files in tarball
May 04 20:42:52 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-10168.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 04 20:54:21 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1714853699.tar.gz` to S3 bucket succeeded
May 04 08:58:06 PM UTC 2024	staged	tarball `eessi-2023.06-software-linux-x86_64-generic-1714853699.tar.gz` downloaded to Stratum-0
May 04 08:59:40 PM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/267 to approve ingest
May 04 09:06:10 PM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-x86_64-generic-1714853699.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/267
May 04 09:16:20 PM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-x86_64-generic-1714853699.tar.gz` successfully ingested at `2023.06/software/linux/x86_64/generic/`

nessi-bot · 2024-05-04T20:04:02Z

New job on instance Saga-NESSI for architecture x86_64-intel-skylake_avx512 for repository nessi-2023.06-swl-deb11 in job dir /cluster/projects/nn9992k/pilot.nessi.no/jobs/2024.05/pr_349/11397842

date	job status	comment
May 04 08:04:02 PM UTC 2024	submitted	job id `11397842` awaits release by job manager
May 04 08:05:05 PM UTC 2024	released	job awaits launch by Slurm scheduler
May 04 08:06:13 PM UTC 2024	running	job `11397842` is running
May 04 08:46:04 PM UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-11397842.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1714853564.tar.gz` size: 1004 MiB (1053713583 bytes) entries: 5518 modules under 2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all `CUDA/12.1.1.lua` software under 2023.06/software/linux/x86_64/intel/skylake_avx512/software `CUDA/12.1.1` other under 2023.06/software/linux/x86_64/intel/skylake_avx512 no other files in tarball
May 04 08:46:04 PM UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11397842.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 04 08:54:41 PM UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1714853564.tar.gz` to S3 bucket succeeded
May 04 08:58:20 PM UTC 2024	staged	tarball `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1714853564.tar.gz` downloaded to Stratum-0
May 04 09:00:34 PM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/269 to approve ingest
May 04 09:06:16 PM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1714853564.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/269
May 04 09:22:45 PM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1714853564.tar.gz` successfully ingested at `2023.06/software/linux/x86_64/intel/skylake_avx512/`

trz42 · 2024-05-04T20:04:14Z

bot: build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell

nessi-bot · 2024-05-04T20:04:16Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted

nessi-bot · 2024-05-04T20:04:16Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted

nessi-bot · 2024-05-04T20:04:16Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted

nessi-bot · 2024-05-04T20:04:17Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- submitted job 11397844, for details & status see {2023.06}[system] CUDA v12.1.1 #349 (comment)

nessi-bot · 2024-05-04T20:04:49Z

New job on instance Saga-NESSI for architecture x86_64-intel-broadwell for repository nessi-2023.06-swl-deb11 in job dir /cluster/projects/nn9992k/pilot.nessi.no/jobs/2024.05/pr_349/11397844

date	job status	comment
May 04 08:04:48 PM UTC 2024	submitted	job id `11397844` awaits release by job manager
May 04 08:05:00 PM UTC 2024	released	job awaits launch by Slurm scheduler
May 04 08:06:11 PM UTC 2024	running	job `11397844` is running
May 04 08:37:16 PM UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-11397844.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-broadwell-1714853508.tar.gz` size: 1004 MiB (1053699813 bytes) entries: 5518 modules under 2023.06/software/linux/x86_64/intel/broadwell/modules/all `CUDA/12.1.1.lua` software under 2023.06/software/linux/x86_64/intel/broadwell/software `CUDA/12.1.1` other under 2023.06/software/linux/x86_64/intel/broadwell no other files in tarball
May 04 08:37:16 PM UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 7/7 test case(s) from 7 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-11397844.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
May 04 08:53:56 PM UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-intel-broadwell-1714853508.tar.gz` to S3 bucket succeeded
May 04 08:58:13 PM UTC 2024	staged	tarball `eessi-2023.06-software-linux-x86_64-intel-broadwell-1714853508.tar.gz` downloaded to Stratum-0
May 04 09:00:06 PM UTC 2024	pr_opened	merge PR https://github.com/NorESSI/2024-staging/pull/268 to approve ingest
May 04 09:06:13 PM UTC 2024	approved	👍 tarball `eessi-2023.06-software-linux-x86_64-intel-broadwell-1714853508.tar.gz` approved, see PR https://github.com/NorESSI/2024-staging/pull/268
May 04 09:19:36 PM UTC 2024	ingested	🎉 tarball `eessi-2023.06-software-linux-x86_64-intel-broadwell-1714853508.tar.gz` successfully ingested at `2023.06/software/linux/x86_64/intel/broadwell/`

trz42 · 2024-05-04T20:50:04Z

Checklist before starting deployment (setting bot:deploy label):

Check if the SPDX license identifier is provided
Check whether builds for all required architectures succeed (SUCCESS message + reasonably sized tarball)
Check if the PR is up-to-date with the target branch nessi.no-2023.06 in the repository (if not what are the differences)
Assess if all requested changes are sound (checking files changed on GitHub.com)
Verify that all easyconfig/s being built are included with the EB version used (if not why not)
Review changes (if any) needed to get the build(s) succeed (common changes for all architectures, changes for a single architecture, changes because of build environment specifics, etc.)

trz42 · 2024-05-04T20:56:23Z

Target architectures

x86_64-intel-broadwell
x86_64-intel-skylake_avx512
x86_64-amd-zen2
x86_64-generic
aarch64-generic

Checklist for deployment/ingestion

at least one tarball for each architecture has been uploaded
at least one tarball for each architecture has been staged
for each staged tarball was a PR for approval/rejection opened
for each architecture one and only one tarball has been approved
for each architecture one and only one tarball was ingested
all tests in the CI have succeeded (rerun failed tests ~ 10 mins after last ingest)
has the package become available via CernVM-FS for all architectures ingested (for lmod cache updates pay attention to timestamp of files)

command & log

command

cd /cvmfs/pilot.nessi.no/versions/2023.06/software/linux ; \
ls -l \
      x86_64/{generic,amd/zen2,intel/[bsc]*}/{software,modules/all}/CUDA/12.1.1* \
      aarch64/generic/{software,modules/all}/CUDA/12.1.1*

log

-rw-r--r--.  1 cvmfs cvmfs 2120 May  4 20:09 aarch64/generic/modules/all/CUDA/12.1.1.lua
-rw-rw-r--.  1 cvmfs cvmfs 2120 May  4 20:14 x86_64/amd/zen2/modules/all/CUDA/12.1.1.lua
-rw-rw-r--.  1 cvmfs cvmfs 2116 May  4 20:14 x86_64/generic/modules/all/CUDA/12.1.1.lua
-rw-rw-r--.  1 cvmfs cvmfs 2148 May  4 20:11 x86_64/intel/broadwell/modules/all/CUDA/12.1.1.lua
-rw-rw-r--.  1 cvmfs cvmfs 2168 May  4 20:10 x86_64/intel/skylake_avx512/modules/all/CUDA/12.1.1.lua

aarch64/generic/software/CUDA/12.1.1:
total 122
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:09 bin
dr-xr-xr-x. 4 cvmfs cvmfs  4096 May  4 20:09 compute-sanitizer
lrwxrwxrwx. 1 cvmfs cvmfs   102 May  4 20:09 DOCS -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/aarch64/generic/software/CUDA/12.1.1/DOCS
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:09 easybuild
-r--r--r--. 1 cvmfs cvmfs 61498 May  4 20:09 EULA.txt
dr-xr-xr-x. 4 cvmfs cvmfs  4096 May  4 20:08 extras
lrwxrwxrwx. 1 cvmfs cvmfs    26 May  4 20:09 include -> targets/sbsa-linux/include
lrwxrwxrwx. 1 cvmfs cvmfs     5 May  4 20:09 lib -> lib64
lrwxrwxrwx. 1 cvmfs cvmfs    22 May  4 20:09 lib64 -> targets/sbsa-linux/lib
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:09 nsight-compute-2023.1.1
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:09 nsight-systems-2023.1.2
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:07 nvml
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:09 nvvm
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:09 pkgconfig
-r--r--r--. 1 cvmfs cvmfs   524 May  4 20:09 README
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:08 share
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:09 stubs
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:07 targets
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:09 tools

x86_64/amd/zen2/software/CUDA/12.1.1:
total 145
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:14 bin
dr-xr-xr-x. 5 cvmfs cvmfs  4096 May  4 20:14 compute-sanitizer
lrwxrwxrwx. 1 cvmfs cvmfs   102 May  4 20:14 DOCS -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/amd/zen2/software/CUDA/12.1.1/DOCS
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:14 easybuild
-r--r--r--. 1 cvmfs cvmfs 61498 May  4 20:14 EULA.txt
dr-xr-xr-x. 5 cvmfs cvmfs  4096 May  4 20:14 extras
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:13 gds
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 gds-12.1
lrwxrwxrwx. 1 cvmfs cvmfs    28 May  4 20:14 include -> targets/x86_64-linux/include
lrwxrwxrwx. 1 cvmfs cvmfs     5 May  4 20:14 lib -> lib64
lrwxrwxrwx. 1 cvmfs cvmfs    24 May  4 20:14 lib64 -> targets/x86_64-linux/lib
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:14 libnvvp
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:14 nsight-compute-2023.1.1
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 nsightee_plugins
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:13 nsight-systems-2023.1.2
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:13 nvml
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:14 nvvm
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 pkgconfig
-r--r--r--. 1 cvmfs cvmfs   524 May  4 20:14 README
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:13 share
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 src
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:14 stubs
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:13 targets
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 tools
lrwxrwxrwx. 1 cvmfs cvmfs   110 May  4 20:14 version.json -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/amd/zen2/software/CUDA/12.1.1/version.json

x86_64/generic/software/CUDA/12.1.1:
total 145
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:14 bin
dr-xr-xr-x. 5 cvmfs cvmfs  4096 May  4 20:14 compute-sanitizer
lrwxrwxrwx. 1 cvmfs cvmfs   101 May  4 20:14 DOCS -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/generic/software/CUDA/12.1.1/DOCS
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:14 easybuild
-r--r--r--. 1 cvmfs cvmfs 61498 May  4 20:14 EULA.txt
dr-xr-xr-x. 5 cvmfs cvmfs  4096 May  4 20:14 extras
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:13 gds
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 gds-12.1
lrwxrwxrwx. 1 cvmfs cvmfs    28 May  4 20:14 include -> targets/x86_64-linux/include
lrwxrwxrwx. 1 cvmfs cvmfs     5 May  4 20:14 lib -> lib64
lrwxrwxrwx. 1 cvmfs cvmfs    24 May  4 20:14 lib64 -> targets/x86_64-linux/lib
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:14 libnvvp
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:14 nsight-compute-2023.1.1
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 nsightee_plugins
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:13 nsight-systems-2023.1.2
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:12 nvml
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:14 nvvm
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 pkgconfig
-r--r--r--. 1 cvmfs cvmfs   524 May  4 20:14 README
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:13 share
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 src
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:14 stubs
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:12 targets
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:14 tools
lrwxrwxrwx. 1 cvmfs cvmfs   109 May  4 20:14 version.json -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/generic/software/CUDA/12.1.1/version.json

x86_64/intel/broadwell/software/CUDA/12.1.1:
total 145
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:11 bin
dr-xr-xr-x. 5 cvmfs cvmfs  4096 May  4 20:10 compute-sanitizer
lrwxrwxrwx. 1 cvmfs cvmfs   109 May  4 20:10 DOCS -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/broadwell/software/CUDA/12.1.1/DOCS
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:11 easybuild
-r--r--r--. 1 cvmfs cvmfs 61498 May  4 20:10 EULA.txt
dr-xr-xr-x. 5 cvmfs cvmfs  4096 May  4 20:10 extras
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:09 gds
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:11 gds-12.1
lrwxrwxrwx. 1 cvmfs cvmfs    28 May  4 20:10 include -> targets/x86_64-linux/include
lrwxrwxrwx. 1 cvmfs cvmfs     5 May  4 20:10 lib -> lib64
lrwxrwxrwx. 1 cvmfs cvmfs    24 May  4 20:10 lib64 -> targets/x86_64-linux/lib
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:11 libnvvp
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:10 nsight-compute-2023.1.1
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:11 nsightee_plugins
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:10 nsight-systems-2023.1.2
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:08 nvml
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:10 nvvm
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:11 pkgconfig
-r--r--r--. 1 cvmfs cvmfs   524 May  4 20:10 README
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:10 share
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:11 src
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:10 stubs
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:08 targets
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:10 tools
lrwxrwxrwx. 1 cvmfs cvmfs   117 May  4 20:10 version.json -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/broadwell/software/CUDA/12.1.1/version.json

x86_64/intel/skylake_avx512/software/CUDA/12.1.1:
total 145
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:10 bin
dr-xr-xr-x. 5 cvmfs cvmfs  4096 May  4 20:10 compute-sanitizer
lrwxrwxrwx. 1 cvmfs cvmfs   114 May  4 20:10 DOCS -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/software/CUDA/12.1.1/DOCS
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:11 easybuild
-r--r--r--. 1 cvmfs cvmfs 61498 May  4 20:10 EULA.txt
dr-xr-xr-x. 5 cvmfs cvmfs  4096 May  4 20:10 extras
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:09 gds
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:10 gds-12.1
lrwxrwxrwx. 1 cvmfs cvmfs    28 May  4 20:10 include -> targets/x86_64-linux/include
lrwxrwxrwx. 1 cvmfs cvmfs     5 May  4 20:10 lib -> lib64
lrwxrwxrwx. 1 cvmfs cvmfs    24 May  4 20:10 lib64 -> targets/x86_64-linux/lib
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:10 libnvvp
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:10 nsight-compute-2023.1.1
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:10 nsightee_plugins
dr-xr-xr-x. 6 cvmfs cvmfs  4096 May  4 20:10 nsight-systems-2023.1.2
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:09 nvml
dr-xr-xr-x. 7 cvmfs cvmfs  4096 May  4 20:10 nvvm
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:10 pkgconfig
-r--r--r--. 1 cvmfs cvmfs   524 May  4 20:10 README
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:09 share
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:10 src
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:10 stubs
dr-xr-xr-x. 3 cvmfs cvmfs  4096 May  4 20:09 targets
dr-xr-xr-x. 2 cvmfs cvmfs  4096 May  4 20:10 tools
lrwxrwxrwx. 1 cvmfs cvmfs   122 May  4 20:10 version.json -> /cvmfs/pilot.nessi.no/host_injections/2023.06/software/linux/x86_64/intel/skylake_avx512/software/CUDA/12.1.1/version.json

poksumdo

it's a start

{2023.06}[system] CUDA v12.1.1

d0cc7bd

trz42 added NESSI/2023.06 sync-with-EESSI GPU labels May 4, 2024

update README.md for easystack files

45f8c20

trz42 added the bot:deploy label May 4, 2024

poksumdo added the ingested label May 4, 2024

poksumdo approved these changes May 4, 2024

View reviewed changes

poksumdo merged commit 73ada9c into NorESSI:nessi.no-2023.06 May 4, 2024
25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{2023.06}[system] CUDA v12.1.1 #349

{2023.06}[system] CUDA v12.1.1 #349

trz42 commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024

nessi-bot bot commented May 4, 2024

nessi-bot bot commented May 4, 2024

nessi-bot bot commented May 4, 2024

trz42 commented May 4, 2024

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

trz42 commented May 4, 2024

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

trz42 commented May 4, 2024 •

edited

Loading

trz42 commented May 4, 2024 •

edited

Loading

poksumdo left a comment

{2023.06}[system] CUDA v12.1.1 #349

{2023.06}[system] CUDA v12.1.1 #349

Conversation

trz42 commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024

nessi-bot bot commented May 4, 2024

nessi-bot bot commented May 4, 2024

nessi-bot bot commented May 4, 2024

trz42 commented May 4, 2024

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited by trz42 Loading

nessi-bot bot commented May 4, 2024 • edited by trz42 Loading

nessi-bot bot commented May 4, 2024 • edited by trz42 Loading

nessi-bot bot commented May 4, 2024 • edited by trz42 Loading

trz42 commented May 4, 2024

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited Loading

nessi-bot bot commented May 4, 2024 • edited by trz42 Loading

trz42 commented May 4, 2024 • edited Loading

trz42 commented May 4, 2024 • edited Loading

poksumdo left a comment

Choose a reason for hiding this comment

trz42 commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited

Loading

nessi-bot bot commented May 4, 2024 •

edited by trz42

Loading

trz42 commented May 4, 2024 •

edited

Loading

trz42 commented May 4, 2024 •

edited

Loading