ggml-cuda was compiled without support for the current GPU architecture. WSL error

[KraiiFox]

Hi I am having trouble trying to run koboldcpp on WSL on windows, every time that I try to run it spams the error ERROR: ggml-cuda was compiled without support for the current GPU architecture. and aborts.

The VRAM spikes up for just a second and then it aborts

ollama and oobabooga text-generation works fine with gpu offloading, koboldcpp also works fine on Windows natively, but not in WSL.

Specs:
CPU: Ryzen 5 3600
RAM: 16GB
GPU: GTX 1070 8GB

[Spacellary]

Hopefully you don't mind I ask a bit of a tangent question in this discussion, is there's a benefit in running Koboldcpp in WSL (WSL2?) instead of natively on windows? Also a GTX 1070 (Ti) user.

Or are you just experimenting?

[KraiiFox]

Nop still getting that error on 1.55.1

[Spacellary]

@KraiiFox Jus tested here too. I guess same as yours in WSL2 then.

❑ Ubuntu in ❑ WSL2

./koboldcpp-linux-x64 silicon-maid-7b.Q4_K_M.gguf --multiuser --gpulayers 33 --contextsize 4096 --port 6900 --usecublas --quiet --remotetunnel
***
Welcome to KoboldCpp - Version 1.55.1
Attempting to use CuBLAS library for faster prompt ingestion. A compatible CuBLAS will be required.
Initializing dynamic library: koboldcpp_cublas.so
==========
Namespace(bantokens=None, blasbatchsize=512, blasthreads=5, config=None, contextsize=4096, debugmode=0, forceversion=0, foreground=False, gpulayers=33, highpriority=False, hordeconfig=None, host='', launch=False, lora=None, model=None, model_param='silicon-maid-7b.Q4_K_M.gguf', multiuser=1, noavx2=False, noblas=False, nommap=False, noshift=False, onready='', port=6900, port_param=5001, preloadstory='', quiet=True, remotetunnel=True, ropeconfig=[0.0, 10000.0], skiplauncher=False, smartcontext=False, ssl=None, tensor_split=None, threads=5, useclblast=None, usecublas=[], usemlock=False)
==========
Loading model: /home/spacellary/downloads/silicon-maid-7b.Q4_K_M.gguf
[Threads: 5, BlasThreads: 5, SmartContext: False, ContextShift: True]

The reported GGUF Arch is: llama

---
Identified as LLAMA model: (ver 6)
Attempting to Load...
---
Using automatic RoPE scaling. If the model has customized RoPE settings, they will be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1070 Ti, compute capability 6.1, VMM: yes
llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from /home/spacellary/downloads/silicon-maid-7b.Q4_K_M.gguf (version GGUF V3 (latest))
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = unknown, may not work
llm_load_print_meta: model params     = 7.24 B
llm_load_print_meta: model size       = 4.07 GiB (4.83 BPW)
llm_load_print_meta: general.name     = sanjiwatsuki_silicon-maid-7b
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size       =    0.11 MiB
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: system memory used  =   70.42 MiB
llm_load_tensors: VRAM used           = 4095.05 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
...................................................................................................
Automatic RoPE Scaling: Using (scale:1.000, base:10000.0).
llama_new_context_with_model: n_ctx      = 4176
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: VRAM kv self = 522.00 MB
llama_new_context_with_model: KV self size  =  522.00 MiB, K (f16):  261.00 MiB, V (f16):  261.00 MiB
llama_build_graph: non-view tensors processed: 676/676
llama_new_context_with_model: compute buffer total size = 296.35 MiB
llama_new_context_with_model: VRAM scratch buffer: 293.16 MiB
llama_new_context_with_model: total VRAM used: 4910.21 MiB (model: 4095.05 MiB, context: 815.16 MiB)
CUDA error: CUBLAS_STATUS_EXECUTION_FAILED
  current device: 0, in function ggml_cuda_op_mul_mat_cublas at ggml-cuda.cu:8118
  cublasSgemm_v2(g_cublas_handles[id], CUBLAS_OP_T, CUBLAS_OP_N, row_diff, src1_ncols, ne10, &alpha, src0_ddf_i, ne00, src1_ddf1_i, ne10, &beta, dst_dd_i, ldc)
GGML_ASSERT: ggml-cuda.cu:229: !"CUDA error"
Aborted

[Spacellary]

@LostRuins Does this fall under #610? WSL2, same hardware. Although I understand this isn't an ideal use case, as it already is working on Windows directly.

[LostRuins]

1.55.1 linux prebuilt is still using all-major, which does seem to trigger this issue on Pascal.

[Spacellary]

I see. Alright, thanks again. Just saw.

[Spacellary]

1.56 solved the WSL2/Linux issues with Pascal cards for me.