I tried to run llama.cpp on the Jolla Phone 2026, and well, it works! I took the latest Ubuntu arm64 (Vulkan) build from Releases · ggml-org/llama.cpp · GitHub. Turns out OpenMP is needed, but this is readily available from the official repository, so as root, a simple pkcon install libgomp fixes that.
I extracted the llama.cpp binaries on phone, and run as $ ./llama-server -hf unsloth/gemma-4-E4B-it-qat-GGUF:UD-Q4_K_XL --host 0.0.0.0 -c 16384. This will download the Gemma 4 E4B model (~5GiB, stored in ~/.cache/huggingface) and run the http server on the phone. For convenience, I use my desktop and navigate to http://192.168.2.4:8080/ to interact with the model.
It’s very slow, but hey, AI on you mobile is fun.
But, while I’ve downloaded the Vulkan build for GPU acceleration, and to my knowledge the phone should support Vulkan, just the CPU is used. Since the phone is in relative pristine state, I guess Vulkan libraries are missing. So, I as root I did pkcon install libhybris-libvulkan, and tried running llama-server again. (I’ve added --cache-ram 0 as to limit the memory use):
[defaultuser@JollaPhone2026 llama-b10586]$ ./llama-server -hf unsloth/gemma-4-E4B-it-qat-GGUF:UD-Q4_K_XL --host 0.0.0.0 -c 16384 -lv 4 --cache-ram 0
library "libselinux.so" not found
Could not open module param file '/sys/module/mali_kbase/parameters/large_page_conf'
0.00.890.542 I cmn common_param: common_params_print_info: build 10586 (b21e4de74) with GNU 14.2.0 for Linux aarch64
0.00.890.588 I cmn common_param: common_params_print_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
0.00.890.590 I cmn common_param: device_info:
0.00.891.026 I cmn common_param: - Vulkan0 : Mali-G610 MC2 (11593 MiB, 11593 MiB free)
0.00.891.107 I cmn common_param: - CPU : CPU (11593 MiB, 11593 MiB free)
0.00.891.342 I cmn common_param: system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.891.390 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.891.545 I srv init: running without SSL
0.00.891.818 I srv init: using 8 threads for HTTP server
...
0.04.359.907 I load_tensors: offloading output layer to GPU
0.04.359.933 I load_tensors: offloading 41 repeating layers to GPU
0.04.359.934 I load_tensors: offloaded 43/43 layers to GPU
0.04.359.942 I load_tensors: Vulkan0 model buffer size = 2493.31 MiB
0.04.359.943 I load_tensors: Vulkan_Host model buffer size = 1872.00 MiB
Vulkan device found! Offloading enabled! But. It’s slower than CPU Way slower. Prompt processing at 0.3t/s, an tg at 2.5t/s (>4t/s for CPU)
Personal Mistral AI API key (free tier available at console.mistral.ai)
Internet connection
I want offline, private, on-device, without subscription or token limits. Not somewhere on someone else’s computer in some cloud. Llama.cpp runs AI fully local, on your own phone.
(Yes I do have a cloud-based subscription too, which works fine in a browser)
Yes, I do understand. What I said is just for my phone. My PC (home network) is a totally different thing, where I do agree with your opinions.
Phone is somewhat restricted by storage options, so LLM’s need to be small. or online.
The best option for acceptable performance will probably be to run the model on your home hardware and access it via VPN from the phone,for some time.
Probably right. But that would require a certain amount of network skills, that I’m pretty sure that not everyone is gifted with. I might manage it, but I’m pretty sure my neighbor’s wife won’t.
Sure, I can run much bigger models at home, via VPN. In fact, I do have 30B-models running on my always-on machine (64GiB with iGPU isn’t faster with dense models 30B models like Qwen, Gemma 4, Glimmer, but it is way smarter)
And llama.cpp can use tensor split to offload much, or even all of the work to this machine via ggml-rpc-serve to this machine. Haven’t tested it with this, actually, because running locally is precisely the point, even if it’s just “because I can”.
But so far, I’m none the wiser on why using Vulkan is slower, and even so so much slower than CPU. It will never be fast, the DDR4 memory just isn’t the right match for this, but it might be practical for some uses.
I came across this bugreport for llama.cpp regarding MedaTek CPUs. They suggest lowering the number of threads to the number of big cores on the CPU. And indeed! With -t 4 (J2 has 4 big cores) it’s somewhat faster. Or at least not slower.
There is no gles3 backend in llama.cpp. And, this isn’t about graphics, more about compute. Our MediaTek processor has no NPU, so Vulkan is probably is native as it can get. But I don’t know how good/optimized Vulkan is on the JP2026.
Yeah, I’ve compiled that for ARM64, but silly me did it on a Ubuntu 26.04 host, so GLIBC is much too new for our Sailfish OS devices. Haven’t tried on Ubuntu 24.04 yet. Even then, no guarantee it would even run with OpenCL. And even then, it’s probably geared fully towards the Adreno GPU, not our Mali GPU.
Ah, on a volla (also mediathekismeky) with openCL I get issues of this sort:
Problem: nothing provides 'libc.so.6(GLIBC_2.38)(64bit)' needed by the to be installed libhybris-0.0.5.58-1.14.1.bso.aarch64
Solution 1: do not install libhybris-libOpenCL-0.0.5.58-1.14.1.bso.aarch64
Solution 2: break libhybris-0.0.5.58-1.14.1.bso.aarch64 by ignoring some of its dependencies
Ah, well, I think I’ll not go break libhybris just yet
Ok, cross-compiled it using Ubuntu 24.04, statically linked libstdc++. But, it seems to require OpenCL 3.0, or at least a symbol of that, which is not available on the phone. When I run llama-server I get:
./llama-server: symbol lookup error: ./llama-server: undefined symbol: clCreateBufferWithProperties, version OPENCL_3.0
Bummer, since the JP2026’s SoC does support OpenCL 3.0, and actually the vendor lib in /vendor/lib64/libOpenCL.so provides OpenCL 3.0. But that’s of course an Android build (damn you, google), and It appears that libhybris’ OpenCL wrapper doesn’t support it, or at least these symbols.
Well, I’m out of my depths here. I suppose libhybris could be patched/updated with the OpenCL 3.0 symbols. And I could have AI have a go at it, but somehow I doubt it’s worth the effort, given that the OpenCL backend is written quite specifically for another GPU.
So, I put Claude to work. TL;DR: it works on OpenCL, but that has no benefit over CPU - if it wouldn’t produce garbage anyway.
Claude mentioned that the OpenCL build will bail out with “unsupported GPU” if it’s not run on either Adreno or Intel GPU. It then proceed to patch it, to accept Mali as well. And to patch it further to make it work. And then to write a custom shim for wrapping the native, Android OpenCL vendor library. And then to debug things.
And.. it actually works! Kinda. Claude tested with ancient Qwen2.5 0.5B q4 model, and ran it using OpenCL offloading, which works fine. I then tested with Gemma 4. And indeed, it does load, it does offload to GPU via OpenCL, but… it’s still a bit slower than plain CPU (but much faster than Vulkan) and.. it only responded with garbage.
I’m glad I didn’t went down that rabbit hole myself; it’s just bizarre what AI can and will do to find and fix problems. And in the end, it was all for nothing, really. The GPU we have is of rather old design, and quite “budget” anyway. And in the end, it’s probably mostly down to memory bandwidth anyway, which is limited, and relative slow due to DDR4 being used.
There is an even smaller Gemma 4 variant, specific for mobile use, but I haven’t even tested it because I suspect the usefulness has degraded too much. I mean, I like Gemma 4 31B at higher quants, but I doubt a Q2 variant of a MOE-like model at one sixth the size makes much sense.
I guess I can answer my own question now: the Mali G610 GPU of our JP2026s is somewhat dated, doesn’t have matrix ops and other for AI inference useful bits. And only 2 shader cores. It’s never going to be fast for this.
With the help of AI I’ve compiled a Vulkan build of llama.cpp using the Sailfish SDK, and… it was still slow, about on par with the OpenCL build, but pure CPU has 6x faster pp, 2.5x faster tg. And it’s unlikely that can be improved much.
So, CPU it is.
FWIW, here’s the AI skill created/used, which your AI can use to build llama.cpp using sfdk:
sailfish-llama-server.targz (6.3 KB) (it’s a .tar.gz, you’ll probably have to rename it.. Weird forum quirk)
It only builds llama-server, CPU-only, optimized for JP2026’s SoC, where only the 4 big cores are used. Results:
Way slower. Prompt processing at 0.3t/s, an tg at 2.5t/s (>4t/s for CPU)
