Llama.cpp on Fedora 43 with cuda support

With every iteration and upgrade to a new major Fedora version, there goes this brand new adventure of making llama.cpp work again. What worked a year ago or even just a half a year ago may be water under the bridge by now.

Between Fedora 39 and 42, I was able to rebuild llama.cpp more or less the same way, resorting to an older gcc version. After the latest update to Fedora 43, it seemed that more things have changed.

For one, llama.cpp is now available as a package in the Fedora repositories. That makes it much easier to install, it’s just a matter of one dnf command. But to the best of my knowledge, this packaged version doesn’t work well with CUDA so I decided to continue building it myself for now so that I can leverage CUDA, otherwise it doesn’t make much sense to use it locally running just on CPU.

Another variable that has changed in between was the introduction of DNF version 5. Therefore, the structure of the commands is now as follows in case you want to enable CUDA repos from Nvidia. As of now, there’s no repo for Fedora 43 so I’m using 42 instead:

# dnf config-manager addrepo --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora42/x86_64/cuda-fedora42.repo
# dnf install cuda

Also the build command itself has changed to the following:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

Other than that, the procedure was the same as with the earlier Fedora and llama.cpp versions.