Generative AI using Linuxulator and eGPU on FreeBSD
3369 words, 16 minutes
Not so long ago, I experimented on so-called Generative AI using an external eGPU and Slackware Linux .
Because I’m a BSD fanboy, I started looking at doing the same on FreeBSD. But I faced a lot of missing dependencies issues and Python compilation errors. As a non fluent Python person, I couldn’t solve all the errors I encountered and decided to see if the FreeBSD Linux Binary Compatibility feature would be able to achieve the goal; after all, there are people using it to watch DRM stuff from the Clouds.
Spoiler alert: it does work given an organized small amount of command line spells.
AI opinion
Feel free to jump to the next section if you don’t care about AI opinions ;-)
These days, AI is sold (and forced) everywhere. I am not the last to yell at Mozilla for pushing some shit into Firefox. I’m also pretty sure vibe coding is a bad idea.
I have tested things like ChatGPT for a few things. Asking for help to debug error messages lead nowhere but to Stack Overflow and Unix & Linux Stack Exchange. Asking for pre-made code offered mostly non-working stuff but I must admit it gave me ideas by identifying features from software I didn’t know. Asking for product comparison was more or less an extract of Tom’s Hardware or Les Numériques. Asking for differences between technology A and B and better use cases felt approximate enough that I usually ended up sending words to SearXNG in order to get more informations.
I also tested the Bing Image Generator; although I’m aware of the global harvesting of copyrighted data this implied. My opinion was that it was not as good as it was sold. It was about the same quality as child-me collecting free cliparts, colouring in Paint and organising in Designer…
All in all, my take is that AI is not Artificial Intelligence. It’s not intelligence at all, yet. At best, it is Algorithm Induce. At worse, it is Aleatory Inference. And most of all, Machine Learning, Neural Networks and LLMs are not the same thing.
I hate the Big Tech’s AI because they are thieves and liars. But I am still interested in the local software that can provide features other software don’t yet. And this is why I keep an eye on running stuff like Python Torch. I have not yet looked at LLaMA. To paraphrase some Monday meme, my take is that You don’t hate AI. You hate LLMs in the context of capitalism and patriarchy.
Overall software overview
There are quite a few bricks to assemble here. There may be smarter way but I have not found any all-in-one documentation. So I settled on choices based on my (lack of) knowledge.
The OS is installed without particular requirements. I install software using binary packages as much as possible.
The FreeBSD NVIDIA drivers are installed and configured first. When everything seems to be working, the Linux Binary Compatibility part can begin.
There is a need to have NVIDIA Linux libraries available. Because it has
to be the same version as the FreeBSD ones, I installed the dedicated
package. From there, you get a /compat/linux directory with a fairly
basic Rocky Linux 9.7 installation. It is bare enough to not ship with a
package manager.
As I have to install more software in the Linux userland, I’m using a
temporary RL9 installation to grab dnf packages that will be installed
in the /compat/linux directory. This allows installing some more
software in an easy way - dealing with binary packages and
dependencies. There is a dnf package in FreeBSD ports but I never
understood how to use it with /compat/linux.
Using the dnf package manager, I installed various required tools and
libraries for later usage.
I grabbed, compiled and installed Python 3.10 into the /compat/linux
directory. Mostly because this version seem to be the one all the tools
I tested need.
The I installed a few additional NVIDIA / CUDA tools and librairies inside
/compat/linux. There is also a specific NVIDIA Unified Memory (UVM)
program to install in order to use the NVIDIA GPU from with Linuxulator.
Finally, a bunch of Python virtual environments can be populated to use PyTorch based-software.
Hardware, BIOS, OS
As described in details here , I am using an NVIDIA RTX 4060 Ti, connected via a Thunderbolt eGPU docking station to a Topton GM1 head-less machine.
The BIOS is configured with “No Security” so that the hardware is recognised automatically.
I have done all my trial & errors on a ThinkPad T480s running FreeBSD 14.3. It went the same way as my final configuration using FreeBSD 14.4 on the Topton. I didn’t go for 15.0 as I already had two bad experience with it on different projects and I see a lot of people on the Fediverse having issues with 15 too.
Install and update FreeBSD 14.4/amd64.
# freebsd-update fetch
# freebsd-update install
Everything else will be done remotely using SSH.
FreeBSD NVIDIA drivers
For some reasons, FreeBSD doesn’t support eGPU hotplug on the ThinkPad. The NVIDIA card has to be powered on and connected to the computer when it boots FreeBSD. This may change in the future.
The Handbook explains how to install the NVIDIA FreeBSD drivers .
# pkg install -y nvidia-drm-kmod
# pkg info | grep nvidia
nvidia-driver-580.119.02_1 NVidia graphics card binary drivers for hardware OpenGL rendering
nvidia-drm-61-kmod-580.119.02.1403000_1 NVIDIA DRM Kernel Module
nvidia-drm-kmod-580.119.02 NVIDIA DRM Kernel Module
nvidia-kmod-580.119.02.1404000_1 kmod part of NVidia graphics card binary drivers for hardware OpenGL rendering
# sysrc kld_list+=nvidia-drm
During my testing period, the meta package installed incompatible versions of the serveral NVIDIA stuff. So I had to force installation of the proper version. For the record, this went like this:
# pkg -N install nvidia-drm-kmod
(...)
nvidia-driver: 580.95.05 [FreeBSD]
nvidia-drm-61-kmod: 580.95.05.1403000 [FreeBSD]
nvidia-drm-kmod: 580.95.05_1 [FreeBSD]
nvidia-kmod: 580.105.08.1403000 [FreeBSD-kmods]
(...)
# pkg install nvidia-drm-kmod-580.95.05_1 nvidia-kmod-580.95.05.1403000
(...)
nvidia-driver: 580.95.05 [FreeBSD]
nvidia-drm-61-kmod: 580.95.05.1403000 [FreeBSD]
nvidia-drm-kmod: 580.95.05_1 [FreeBSD]
nvidia-kmod: 580.95.05.1403000 [FreeBSD]
(...)
A reboot was performed to ensure I configured everything properly. The NVIDIA card was luckily identified and connected properly.
# pciconf -lv | grep -B3 display
vgapci0@pci0:0:2:0: class=0x030000 rev=0x0c hdr=0x00 vendor=0x8086 device=0x4628 subvendor=0x8086 subdevice=0x2112
vendor = 'Intel Corporation'
device = 'Alder Lake-UP3 GT2 [UHD Graphics]'
class = display
--
vgapci1@pci0:5:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2805 subvendor=0x19da subdevice=0x7717
vendor = 'NVIDIA Corporation'
device = 'AD106 [GeForce RTX 4060 Ti 16GB]'
class = display
# dmesg | grep -C 5 -i nvidia
nvidia1: <NVIDIA GeForce RTX 4060 Ti> on vgapci1
vgapci1: child nvidia1 requested pci_enable_io
vgapci1: child nvidia1 requested pci_enable_io
nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 580.119.02 Mon Dec 8 07:29:16 UTC 2025
[drm] [nvidia-drm] [GPU ID 0x00000500] Loading driver
sysctl_add_oid: can't re-use a leaf (hw.dri.debug)!
sysctl_add_oid: can't re-use a leaf (hw.dri.vblank_offdelay)!
sysctl_add_oid: can't re-use a leaf (hw.dri.timestamp_precision)!
[drm] Initialized nvidia-drm 0.0.0 20160202 for nvidia1 on minor 1
# sysctl hw.nvidia
hw.nvidia.gpus.1.type: PCIe
hw.nvidia.gpus.1.firmware:
hw.nvidia.gpus.1.vbios: ??.??.??.??.??
hw.nvidia.gpus.1.model: NVIDIA GeForce RTX 4060 Ti
hw.nvidia.version: NVIDIA UNIX x86_64 Kernel Module 580.119.02 Mon Dec 8 08:42:31 UTC 2025
Side note for dual GPU configuration
While trying to have all the following working, I used a laptop with an embedded Intel GPU. But when I booted with the NVIDIA eGPU connected, Xorg would detect it and use it as the primary display device. And because nothing was connected to the NVIDIA card output, I went blind on the laptop. To force Xorg to not use the NVIDIA GPU at all, I had to create a dedicated configuration snippet. This way, I was still able to run Xfce on the Intel GPU using the laptop monitor and the NVIDIA GPU for the Torch things.
# cat /usr/local/share/X11/xorg.conf.d/20-gpu.conf
Section "Device"
Identifier "intel0"
Driver "modesetting"
BusID "pci0:0:2:0"
Option "PrimaryGPU" "true"
EndSection
Section "Device"
Identifier "nvidia0"
Driver "nvidia"
BusID "pci0:9:0:0"
EndSection
Section "ServerFlags"
Option "AutoAddGPU" "false"
EndSection
Linux Binary Compatibility
Reading the Handbook Chapter 12. Linux Binary Compatibility is probably a good idea.
For the moment, I just needed to enable linux(4) binary compatibility:
# service linux enable
# service linux start
This creates the /compat/linux directory and mount the required file systems.
# mount | grep linux
linprocfs on /compat/linux/proc (linprocfs, local)
linsysfs on /compat/linux/sys (linsysfs, local)
devfs on /compat/linux/dev (devfs)
fdescfs on /compat/linux/dev/fd (fdescfs)
tmpfs on /compat/linux/dev/shm (tmpfs, local)
Temporary Rocky Linux 9
A temporary RL9 instance is deployed on the server. I used the 9.7
version as this is the one available in the ports. It is based on an OCI
image and is only used to easily grab all the packages required to
install the dnf package manager in /compat/linux.
Grab the RL97 container image and deploy in a temporary location:
# fetch https://dl.rockylinux.org/pub/rocky/9.7/images/x86_64/Rocky-9-Container-Base.latest.x86_64.tar.xz
# mkdir Rocky-9 /tmp/rl97
# tar xf Rocky-9-Container-Base.latest.x86_64.tar.xz -C Rocky-9/
# find Rocky-9/blobs/sha256 -type f -exec tar xpzf {} -C /tmp/rl97 \; 2>/dev/null
# cp -p /etc/resolv.conf /etc/hosts /tmp/rl97/etc/
# chroot /tmp/rl97 /bin/bash -l
Update the Rocky Linux system and grab the required dnf packages:
# update-ca-trust
# dnf --releasever 9.7 -y update
# dnf --releasever 9.7 -y install python3-dnf-plugin-modulesync
# dnf --releasever 9.7 -y download --resolve --alldeps --downloaddir ~/DNF dnf
# exit
Back on the FreeBSD host, we have all the stuff available to manage
packages in /compat/linux.
Rocky Linux 9 Linuxulator
When using the ports packages, several Linux distributions can be used.
The NVIDIA Linux libraries will deploy a bare Rocky Linux 9.7 system
inside /compat/linux. The rpm package is used to deploy the dnf
packages we previously grabbed into /compat/linux.
# pkg install -y linux-nvidia-libs rpm4
# cd /compat/linux
# for f in /tmp/rl97/root/DNF/*rpm; do rpm2cpio < "$f" | cpio -id; done
# cp -p /etc/resolv.conf /etc/hosts /compat/linux/etc/
# chroot /compat/linux /bin/bash -l
# update-ca-trust
# dnf --releasever 9.7 -y update
# exit
From there, the /compat/linux directory contains the NVIDIA librairies
and a package manager suitable to add more things to the Linux instance.
Don’t forget to remove the temporary instance as we should not need it
anymore.
Python 3.10 for Linux
All the PyTorch software I tested recommand (if not require) Python 3.10. But because of RedHat policy, this version is not available as binary packages. AFAIK.
Luckily, it can still be build from sources and installed in the Linux directory.
# chroot /compat/linux /bin/bash -l
# dnf --releasever 9.7 -y install gcc make tar wget \
bzip2-devel libffi-devel openssl-devel xz-devel zlib-devel
# cd /root
# wget https://www.python.org/ftp/python/3.10.19/Python-3.10.19.tgz
# tar xzf Python-3.10.19.tgz
# cd Python-3.10.19
# ./configure --enable-optimizations
# make -j $(nproc)
# make altinstall
# python3.10 -V
Python 3.10.19
# pip3.10 -V
pip 23.0.1 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)
# exit
Python 3.10 is now available inside /compat/linux and can be used to
deploy virtual environments for PyTorch applications.
Using the NVIDIA eGPU in Linuxulator
A first quick test shows that the NVIDIA card is recognised inside the Linux emulation system.
From /compat/linux, I can access the NVIDIA device.
# chroot /compat/linux /bin/bash -l
# uname -a
Linux topton 5.15.0 FreeBSD 14.4-RELEASE releng/14.4-n273675-a456f852d145 GENERIC x86_64 x86_64 x86_64 GNU/Linux
# nvidia-smi
Thu Mar 12 14:42:37 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.119.02 Driver Version: 580.119.02 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4060 Ti Off | 00000000:09:00.0 Off | N/A |
| 33% 26C P0 21W / 165W | 0MiB / 16380MiB | 3% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
# dnf --releasever 9.7 -y install epel-release
# dnf --releasever 9.7 -y install nvtop
# nvtop
Let’s build a simple Python script that will be used to identify if the device is really accessible or not.
# cat > test_tensor.py
# test_tensor.py
import torch
device = torch.device("cuda")
a = torch.ones(3, 3).to(device)
print(f"Calculation Success! \n{a + a}")
^D
# python3.10 -m venv venv
# source ./venv/bin/activate
# pip install torch
# python test_tensor.py
/root/venv/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:283: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
cpu = _conversion_method_template(device=torch.device("cpu"))
Traceback (most recent call last):
File "/root/test_tensor.py", line 4, in <module>
a = torch.ones(3, 3).to(device)
File "/root/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 424, in _lazy_init
torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS
# deactivate
As detailed here , there is some need for UVM correction something. I have no idea what this is all about so I’m thankful to Isapon Soft for making their notes public. Lets build the magic thing and run the Torch test again
# wget https://gist.githubusercontent.com/shkhln/40ef290463e78fb2b0000c60f4ad797e/raw/0e1fd8e8ea52b7445c3d33f5e5975efd20388dcb/uvm_ioctl_override.c
# gcc -m64 -std=c99 -Wall -ldl -fPIC -shared -fno-lto -o dummy-uvm.so uvm_ioctl_override.c
# source ./venv/bin/activate
# LD_PRELOAD="/root/dummy-uvm.so" python test_tensor.py
/root/venv/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:283: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
cpu = _conversion_method_template(device=torch.device("cpu"))
Calculation Success!
tensor([[2., 2., 2.],
[2., 2., 2.],
[2., 2., 2.]], device='cuda:0')
# deactivate
# exit
From there, we are now ready to run a bunch of PyTorch stuff inside our
/compat/linux directory.
Run upscaler
I have a bunch of small family photos. And it is great to be able to get them a bit bigger. We are not in a Jason Bourne or NCIS movie so anything blurred will remain blurred. Upscaling and AI don’t guess/invent details. But visible stuff will be visible and a bit bigger.
upscaler is an image upscaler using Real-ESRGAN with a web API and UI . There are maybe better software elsewhere but this one is really easy to use and provides a Web GUI that is perfect for using with a headless server.
Once started, the interface is accessible using a Web browser targeting http://0.0.0.0:8000.
# chroot /compat/linux /bin/bash -l
# [ ! -x /bin/git ] && dnf --releasever 9.7 -y install git
# dnf --releasever 9.7 -y install libglvnd-glx
# git clone https://github.com/robertsLando/upscaler.git
# cd upscaler
# python3.10 -m venv pyenv
# source pyenv/bin/activate
# pip install uv
# pip install --upgrade pip
# uv sync
# LD_PRELOAD="/root/dummy-uvm.so" make run
auv run python -m upscaler
INFO: Started server process [17878]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
(...)
INFO:upscaler.upscaler:Real-ESRGAN model initialized successfully
INFO:upscaler.utils:Starting upscaling process...
INFO:upscaler.utils:Upscaled to: (4316, 3392)
INFO:upscaler.utils:Final image size: (3237, 2544)
INFO: 10.15.5.61:42284 - "POST /upscale HTTP/1.1" 200 OK
^C
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [17878]
# deactivate
# exit
Using nvtop, one can see that the GPU is used during the upscaling
process.
Run Fooocus
The installation is pretty well the same as the upscaler. The online documentation of the project works just well.
But because I had a lot of trial & errors during my journey on FreeBSD, I wrote a script that would deploy, update and run the software in a more automatic way.
# cat /compat/linux/root/fooocus
#!/bin/bash
[ ! -d ~/sources ] && mkdir ~/sources
[ ! -d ~/Applications ] && mkdir ~/Applications
if [ ! -d ~/sources/Fooocus ]; then
cd ~/sources
git clone https://github.com/lllyasviel/Fooocus.git
fi
if [ ! -d ~/Applications/Fooocus ]; then
python3.10 -m venv ~/Applications/Fooocus
source ~/Applications/Fooocus/bin/activate
cd ~/sources/Fooocus
pip install -r requirements_versions.txt
pip install pip -U
else
source ~/Applications/Fooocus/bin/activate
cd ~/sources/Fooocus
fi
if [ "$1" = "update" ]; then
git pull
pip install -r requirements_versions.txt
fi
LD_PRELOAD="/root/dummy-uvm.so" python entry_with_update.py --listen
deactivate
exit 0
#EOF
This script makes it simpler to run Fooocus chrooted straight from a connexion on the FreeBSD host:
# chroot /compat/linux /root/fooocus
(...)
Fooocus V2 Expansion: Vocab with 642 words.
Fooocus Expansion engine loaded for cuda:0, use_fp16 = True.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 1.08 seconds
Started worker with PID 18059
App started successful. Use the app with http://localhost:7865/ or 0.0.0.0:7865
Browsing to http://0.0.0.0:7865/ and looking at nvtop while generating
shit stuff proves everything works as expected.
Tips for other projects
I will not detail all the GitHub projects that I tested. But some
required extra stuff in the Linux environment. They were not always easy
to understand. And they also explain why I wanted to have a working
dnf environment.
Need g++
If the project requires g++ to build Python stuff, the missing
development tools can be installed using:
# dnf --releasever 9.7 -y groupinstall "Development Tools"
Need ffmpeg
If the project requires ffmpeg to be installed, this can be done this
way:
# dnf --releasever 9.7 -y config-manager --enable crb
# dnf --releasever 9.7 -y install https://mirrors.rpmfusion.org/free/el/rpmfusion-free-release-9.noarch.rpm
# dnf --releasever 9.7 -y install https://mirrors.rpmfusion.org/nonfree/el/rpmfusion-nonfree-release-9.noarch.rpm
# dnf repolist
repo id repo name
appstream Rocky Linux 9 - AppStream
baseos Rocky Linux 9 - BaseOS
crb Rocky Linux 9 - CRB
epel Extra Packages for Enterprise Linux 9 - x86_64
epel-cisco-openh264 Extra Packages for Enterprise Linux 9 openh264 (From Cisco) - x86_64
extras Rocky Linux 9 - Extras
rpmfusion-free-updates RPM Fusion for EL 9 - Free - Updates
rpmfusion-nonfree-updates RPM Fusion for EL 9 - Nonfree - Updates
# dnf --releasever 9.7 -y install ffmpeg
# ffmpeg -version
ffmpeg version 5.1.8 Copyright (c) 2000-2025 the FFmpeg developers
built with gcc 11 (GCC)
configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libcdio --enable-libdrm --enable-libjack --enable-libjxl --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libshaderc --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzvbi --enable-lv2 --enable-avfilter --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libmfx --enable-runtime-cpudetect
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100
Need NVIDIA CUDA
If the project yells something likeFailed to load library libonnxruntime_providers_cuda.so with error: libcudnn_adv.so.9: cannot open shared object file: No such file or directory, it means you probably miss some NVIDIA CUDA Deep Neural
Network
(cuDNN)
librairies.
# dnf --releasever 9.7 -y config-manager --add-repo \
https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
# dnf --releasever 9.7 -y install --allowerasing cudnn9-cuda-12
If the project yells something likeFailed to load library libonnxruntime_providers_cuda.so with error: libnvrtc.so.12: cannot open shared object file: No such file or directory, it means you probably miss some CUDA toolkit libraries.
# dnf --releasever 9.7 -y install cuda-toolkit-12
Final thoughts
It doesn’t look like FreeBSD did better than Slackware on the workloads I tested. Long processes still took hours. Short processes took from couple of tenth of seconds to minutes. CPU, memory and GPU resources were used about the same way and with the same amount.
Nothing special regarding power consumption either: Topton used about 14W at IDLE and 33W while processing, the RTX eGPU idles at 26W and requires up to 140W while processing. Temperature never went above 36°C but I added an USB fan on the case and the room is about 19-20°C.
All in all, it works great. Not better, not less.
The objective me can’t go without thinking: what’s the point of running FreeBSD to do this if you need the Linux emulation to do the exact same thing than with a native Linux system. Just because you can doesn’t mean you should :D And given that the NVIDIA drivers have to carefully match both the FreeBSD host and the Linux layer, one could fear what will happen during the next upgrade…
Still, this proves that BSDs are capable of running hype technology and that some more workload can be migrated from Linux. Because diversity matters!!!
And if you already have a FreeBSD desktop, and you 🎶 need some slop stuff baby tonight 🎶, you don’t need a dedicated Linux machine.