Notes for setting up Ollama in Docker on the Nvidia DGX Spark
The full write-up lives in the gist: ollama_docker_userns_remap.txt. This post is the short version — and the comments on the gist, where the thinking-out-loud landed, are included near the end.
Switching to Ollama on Docker, with userns-remap
I switched to running Ollama on Docker — with userns-remap, so it’s not running as root either:
docker run -e 'OLLAMA_HOST=localhost:11434' -d --gpus=all \
-v /usr/share/ollama/.ollama/:/root/.ollama --net host \
--restart unless-stopped \
--name ollama ollama/ollama:0.40.9-rc1
(there’s a newer command further down, copy-pasteable, that you’ll actually want)
ollama/ollama:0.40.9-rc1 — if you need newer tags, see the Ollama tags on Docker Hub. The version we were on before I think was .40.6, and the one before I upgraded the other day was 0.33.2; I’m gonna pull them so they’re on hand and the container can just be bounced with the ver tag if need be.
This way it’s easy to switch between versions / upgrade to newer / rc versions. I actually needed 40.9-rc1 just to get access to flint:40b this morning (the ollama installer would only give up like 40.8… although then again, I never tried running the installer with --help or anything like that — I just assumed it’s not distributed that way at all. In any case you can get it with docker.)
- disabled the ollama systemd unit (
systemctl disable ollama) - enabled the docker systemd unit (
systemctl enable docker) - verified that it is actually only listening on 127.0.0.1:11434:
root@spark-4d11:~# netstat -an | grep 11434
tcp 0 0 127.0.0.1:11434 0.0.0.0:* LISTEN
root@spark-4d11:~#
- nvidia-ctk was already installed, it’s enabled for docker, there’s an apt update for it I think. nvidia-ctk documentation is here, and the apt repositories were already set up — no surprise, this is nvidia hardware, so one would hope. You don’t have to do all that shit :)
docker logs -f ollama, and the one line that stands out
docker logs -f ollama
Full server config dump and GPU discovery, eliding the noisy parts and wrapping the long lines for readability (docker logs -f $(docker ps -q) works too):
Listening on 127.0.0.1:11434 (version 0.40.9-rc1)
discovering available GPUs...
level=INFO source=llama_server.go:295 msg="skipping CUDA device — compute capability not in compiled architectures"
device="NVIDIA GB10" cc=1210 archs="[500 520 600 610 700 750 800 860 890
900 1000 1200]"
libDirs="[/usr/lib/ollama /usr/lib/ollama/cuda_v12]"
level=INFO source=types.go:32 msg="inference compute" id=0 filter_id=0 library=CUDA compute=12.1 name=CUDA0
description="NVIDIA GB10" libdirs=ollama,cuda_v13 driver=13.0
pci_id=000f:01:00.0 type=iGPU total="121.7 GiB" available="117.9 GiB"
level=INFO source=routes.go:2562 msg="vram-based default context" total_vram="121.7 GiB" default_num_ctx=262144
One message stands out:
msg=“skipping CUDA device — compute capability not in compiled architectures” device=“NVIDIA GB10”
Found a bit of information on X; @GilleandKristi writes:
Updated #Ollama 0.23 → 0.30 on my DGX Spark today. Two things worth knowing if you’re on a #GB10. 1/ OLLAMA_VULKAN now defaults ON. On CUDA hardware you don’t want the Vulkan path — set OLLAMA_VULKAN=0 in your systemd override. 2/ You’ll see a scary “skipping CUDA device — compute capability not in compiled architectures” line at startup. It’s probing cuda_v12 (built up to sm_120), failing on the GB10’s sm_121, then falling through to cuda_v13 which works fine. 100% GPU, no action needed.
So I added -e 'OLLAMA_VULKAN=0' to the docker run but it seems to make no difference — I’ll keep researching. However reading their message a bit more:
and then falls through to v13
…checks out:
libdirs=ollama,cuda_v13 driver=13.0
So yeah that’s probably ok. Not a bad idea to make sure it’s not using Vulkan anyway, for future reference. I’m just curious why it was defaulting to that for him and not here (unless they realized that was a pain in the ass and fixed it since June 2027).
Switching back (docker → host install), if you want to
Two steps:
docker stop ollama(won’t restart; the restart policy is unless-stopped — but be sure:docker rm ollama)systemctl enable ollama && systemctl start ollama
The only thing that I can think of that might be an issue here is the permissions on /usr/share/ollama/.ollama/ vs /root/.ollama — maybe different now than what the host-installed version expects:
root@spark-4d11:~# grep -P "(User|Group)" /etc/systemd/system/ollama.service
User=ollama
Group=ollama
Related: ollama/ollama#5986. Idk, it’s running on 127.0.0.1 and we’re the only ppl using it so :)
Another solution to this problem that I would recommend over messing with making the container user non-root: User namespace remapping — an advanced feature to protect your Docker environments. And I’ve had pretty good luck with userns-remap — but I’m willing to bet it will cause problems for Nvidia’s container toolkit, or at the very least permissions problems, because:
# ls -lah /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Oct 22 08:11 /dev/nvidia-modeset
crw-rw-rw- 1 root root 497, 0 Oct 22 08:11 /dev/nvidia-uvm
crw-rw-rw- 1 root root 497, 1 Oct 22 08:11 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195, 0 Oct 22 08:11 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Oct 22 08:11 /dev/nvidiactl
/dev/nvidia-caps:
total 0
drwxr-xr-x 2 root root 80 Oct 22 08:11 .
drwxr-xr-x 7 root root 480 Oct 22 08:11 ..
cr-------- 1 root root 500, 1 Oct 22 08:11 nvidia-cap1
cr--r--r-- 1 root root 500, 2 Oct 22 08:11 nvidia-cap2
#
For docker (usually /etc/default/docker/ or /etc/sysconfig):
--userns-remap=default|uid:gid|user:group|user|uid
Enable user namespaces for containers on the daemon. Specifying "default"
will cause a new user and group to be created to handle UID and GID range
remapping for the user namespace mappings used for contained processes.
Specifying a user (or uid) and optionally a group (or gid) will cause the
daemon to lookup the user and group's subordinate ID ranges for use as the
user namespace mappings for contained processes.
And yeah — I remember this being kinda obtuse, where 'default' is not really the default; “nothing is remapped” is the default.
Related:
So I can revisit this if need be, but it’s a point of “gilding the lily” at this point — daemons only listening on 127.0.0.1 (with docker’s host networking driver), so it’s not even reachable except thru ssh.
Checking status:
root@spark-4d11:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fd2d24965132 ollama/ollama:0.40.9-rc1 "/bin/ollama serve" 22 minutes ago Up 22 minutes ollama
Checking logs:
docker logs -f ollama
# or
docker logs -f fd
# or
docker logs -f fd2d24965132
Ollama env vars
This doc covers new and historic — pretty much every env var that matters: ollama environment variables.
OLLAMA_KEEP_ALIVE=5mis default;-1== forever. I thought about this a bit, and-1probably is ok.OLLAMA_KV_CACHE_TYPE=f16is default andq8_0is common.OLLAMA_CONTEXT_LENGTH=0is default (means default); probably don’t need to change this.OLLAMA_DEBUGdefault isinfolevel; common override is1?OLLAMA_ORIGINS— also the other CORS settings; these will be necessary to make it work with yacy.OLLAMA_VULKANdefault is ‘on where the backend is installed’. Not really sure what models would benefit from this since most things run better/faster on nvidia directly, not sure if it being on by default is even really a problem — but disabling anyway.OLLAMA_HOSTdefault is127.0.0.1:11434(wait, no it’s not — I checked earlier, I had to override it. Maybe the docker container is overriding it because it’s probably not meant to be run with--net host. I hate docker networking most of the time —--net hostmakes more sense.)OLLAMA_NUM_PARALLEL— concurrent requests handled per model. Default is 1, setting to 2:
VRAM / Memory Constraints: Each parallel request requires its own KV-cache (key-value memory) scaled by OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. If the model is too large for your available VRAM or spills over to system RAM, Ollama automatically scales down or defaults back to 1 slot.
In my experience if it runs out of memory ollama will crash, and that will be reflected in the uptime of the ollama container (docker ps) and also the logs — but usually when I’m on here I only ever see the huihui_ai/mistral-small-abliterated:24b model in use, which I never use.
Command:
docker run \
-e 'OLLAMA_KEEP_ALIVE=-1' \
-e 'OLLAMA_VULKAN=0' \
-e 'OLLAMA_HOST=localhost:11434' \
-e 'OLLAMA_NUM_PARALLEL=2' \
-d --gpus=all \
-v /usr/share/ollama/.ollama/:/root/.ollama --net host \
--restart unless-stopped \
--name ollama ollama/ollama:0.40.9-rc1
The only thing that stands out still as possibly needing to change is OLLAMA_KV_CACHE_TYPE, because I remember people were talking in the signal chat about this at one point, but I can’t remember or didn’t get much detail.
UPDATE
Something I just noticed:
root@spark-4d11:/etc/systemd# grep -r OLLAMA
system/ollama.service.d/override.conf:Environment="OLLAMA_KEEP_ALIVE=-1"
system/ollama.service.d/override.conf:Environment="OLLAMA_HOST=0.0.0.0:11434"
system/ollama.service.d/override.conf.bak.20260717-000549:Environment="OLLAMA_KEEP_ALIVE=-1"
system/ollama.service.d/.#override.conffebf80d25938e8af:Environment="OLLAMA_KEEP_ALIVE=-1"
root@spark-4d11:/etc/systemd#
^__ confirms OLLAMA_KV_CACHE_TYPE was not tuned — but explains why I thought the default behavior was listen on all hosts.
Apparently the host install of ollama was set up to listen on 0.0.0.0:11434 — huh. Ok, I’ll set that for now and ask about it later:
docker run \
-e 'OLLAMA_KEEP_ALIVE=-1' \
-e 'OLLAMA_VULKAN=0' \
-e 'OLLAMA_HOST=0.0.0.0:11434' \
-e 'OLLAMA_NUM_PARALLEL=2' \
-d --gpus=all \
-v /usr/share/ollama/.ollama/:/root/.ollama --net host \
--restart unless-stopped \
--name ollama ollama/ollama:0.40.9-rc1
UPDATE 2: nvidia-ctk, userns-remap, and the subuid squat
nvidia-ctk totally seems to work with userns-remap, so the process no longer actually runs as root. I only had to:
- add
--userns-remap=defaultto the ExecStart override in/etc/systemd/system/docker.service.d/docker-override.conf systemctl daemon-reloadsystemctl restart docker
root@spark-4d11:~# ps auxw | grep dockerd
root 1819051 7.6 0.0 2945020 87360 ? Ssl 12:20 0:58 /usr/bin/dockerd -H fd://
--containerd=/run/containerd/containerd.sock --userns-remap=default
--net host won’t work with --userns-remap, so I had to substitute -p 11434:11434:
docker run \
-e 'OLLAMA_KEEP_ALIVE=-1' \
-e 'OLLAMA_VULKAN=0' \
-e 'OLLAMA_HOST=0.0.0.0:11434' \
-e 'OLLAMA_NUM_PARALLEL=2' \
-d --gpus=all \
-v /usr/share/ollama/.ollama/:/root/.ollama \
--restart unless-stopped \
-p 11434:11434 \
--name ollama ollama/ollama:0.40.9-rc1
root@spark-4d11:~# ps aux | grep ollama
100000 1819798 0.0 0.0 2329156 35580 ? Ssl 12:24 0:00 /bin/ollama serve
100000 1820101 13.7 3.1 174263824 3956544 ? Sl 12:25 1:11 /usr/lib/ollama/llama-server --model
/root/.ollama/models/blobs/sha256-729f... --port 39969 --host 127.0.0.1 --offline -c 262144 -np 1 ...
The process now runs with uid 100000 even though to the container (guest OS) it appears to be 0. And the mapped filesystem permissions have to be:
sudo chown -R 100000 /usr/share/ollama/.ollama/
To be sure, set perms then:
docker restart ollama && docker logs -f ollama
To change this back for the host-installed ollama you would just do:
chown -R ollama:ollama /usr/share/ollama/.ollama/
I’m actually surprised nvidia-ctk just works with this, because I remember back in the day — whatever it used to be that you needed to get nvidia gpu access (7 years ago) in docker — barely worked at all.
Something I did just notice though is that it created some overlap (I guess I didn’t even know this was a thing unless somebody set this up):
root@spark-4d11:~# cat /etc/subuid
fare:100000:65536
ober:165536:65536
vyzo:231072:65536
sq:296608:65536
dockremap:100000:65536
so:
sudo sed -i 's/^dockremap:100000:65536/dockremap:362144:65536/' /etc/subuid
sudo sed -i 's/^dockremap:100000:65536/dockremap:362144:65536/' /etc/subgid
root@spark-4d11:~# grep "." /etc/subuid /etc/subgid
/etc/subuid:fare:100000:65536
/etc/subuid:ober:165536:65536
/etc/subuid:vyzo:231072:65536
/etc/subuid:sq:296608:65536
/etc/subuid:dockremap:362144:65536
/etc/subgid:fare:100000:65536
/etc/subgid:ober:165536:65536
/etc/subgid:vyzo:231072:65536
/etc/subgid:sq:296608:65536
/etc/subgid:dockremap:362144:65536
- tear down the container
- delete it
^ this is important, because when you change the uid map:
root@spark-4d11:~# ls /var/lib/docker/
100000.100000/ containers/ network/ runtimes/ volumes/
362144.362144/ engine-id overlay2/ swarm/
buildkit/ image/ plugins/ tmp/
it creates a new subdirectory (/var/lib/docker gets used if you don’t specify --userns-remap=default, and if you do specify then it creates a subdir /var/lib/docker/<whatever range in /etc/subuid that you have>).
- restart docker (the subuid change won’t take effect until you do)
sudo chown -R 362144:362144 /usr/share/ollama/.ollama/- recreate the container (has to re-pull the image, because new working directory):
docker run \
-e 'OLLAMA_KEEP_ALIVE=-1' \
-e 'OLLAMA_VULKAN=0' \
-e 'OLLAMA_HOST=0.0.0.0:11434' \
-e 'OLLAMA_NUM_PARALLEL=2' \
-d --gpus=all \
-v /usr/share/ollama/.ollama/:/root/.ollama \
--restart unless-stopped \
-p 11434:11434 \
--name ollama ollama/ollama:0.40.9-rc1
Cool, that works:
root@spark-4d11:~# ps aux | grep ollama
362144 1821239 0.2 0.0 2320244 34780 ? Ssl 12:49 0:00 /bin/ollama serve
So the only hang up was I think docker whatever-it-does to find a continuous range isn’t really trying hard enough — that was easy to fix, but it probably wouldn’t even have mattered. In any case though, the processes that run in docker containers now can never truly run as root, and this breaks a lot of things:
No, you cannot run a standard
--privilegedcontainer while userns-remap is enabled in Docker, because privileged mode is incompatible with user namespaces.
No, you cannot use
--network=host(or--net=host) together with user namespace remapping (userns-remap) in Docker.
Yes, you can technically use Docker Swarm with userns-remap, but it is heavily discouraged due to major architectural limitations, long-standing bugs, and a lack of support in Swarm service definitions.
But it actually makes using docker kinda worthwhile, because to the guest that’s running in the container:
root@spark-4d11:~# docker exec -it ollama ps auxw
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2402436 34976 ? Ssl 12:55 0:00 /bin/ollama serve
root 161 14.7 2.6 173748448 3437664 ? Sl 12:55 1:47 /usr/lib/ollama/llama-server --model /root/.ollama/models/blobs/s...
root 207 0.0 0.0 7640 3636 pts/0 Rs+ 13:07 0:00 ps auxw
root@spark-4d11:~# docker exec -it ollama id
uid=0(root) gid=0(root) groups=0(root)
It still thinks it’s running as root even though it isn’t.
Oct-24 update
Added sysctl -w kernel.unprivileged_userns_clone=0 to /etc/sysctl.conf.
UPDATE: OLLAMA_NUM_PARALLEL=4, latest version
Switched OLLAMA_NUM_PARALLEL from 2 to 4 — running huihui_ai/mistral-small-abliterated:24b, 36 GB, 100% GPU, llama.cpp, Forever — and moved to the latest image:
docker run \
-e 'OLLAMA_KEEP_ALIVE=-1' \
-e 'OLLAMA_VULKAN=0' \
-e 'OLLAMA_HOST=0.0.0.0:11434' \
-e 'OLLAMA_NUM_PARALLEL=4' \
-d --gpus=all \
-v /usr/share/ollama/.ollama/:/root/.ollama \
--restart unless-stopped \
-p 11434:11434 \
--name ollama ollama/ollama:0.40.13
Notes from the comments
The comments on the gist are part of the story — they’re where I parked the “what do I do about this” thinking while the setup above was coming together. Nothing in this section is implemented; it’s all things I’m chewing on.
k8s finally caught up
From Oct 22:
Supposedly k8s supports this too now which I remember asking about it long time ago and they didn’t seem to have plans for fixing the only rootless options I could find (didn’t work on the lxd host that I had so..) kubernetes.io/docs/concepts/workloads/pods/user-namespaces/ Really surprised Nvidia-ctk works seamlessly without privileged or needing uid 0, somebody must’ve cared.
Landlock
From Oct 24 — long comment, kept because the analysis is directionally right (and yes, it started life as an AI paste; I haven’t done the audit it keeps calling for):
Landlock is a Linux LSM that lets an unprivileged process restrict its own — and its descendants’ — filesystem access via inheritable rulesets. It’s a good fit for hardening the ollama container because you can lock the process down to only the paths it actually needs (model directory, config, socket) without needing CAP_SYS_ADMIN or a privileged profile.
What Landlock gives you:
- Path-based sandboxing: allow/deny
read,write,execute,create,truncate, etc. on specific directories/files. - Inheritable: rulesets apply to children, so you can set them in an entrypoint wrapper before
execing ollama. - Unprivileged: works inside a user namespace (my userns-remap setup) — though rulesets are scoped to the creating user namespace.
Key constraints with this stack:
- User namespace scoping: in a userns-remapped container, container root is mapped to a subordinate UID on the host. Landlock works inside the container’s user namespace, but it won’t by itself restrict access outside that namespace unless the host also enforces it.
- nvidia-ctk / GPU access: ollama needs
/dev/nvidia*and the NVIDIA CUDA libraries. Landlock only controls filesystem paths — it can’t restrictioctl/mmapon device nodes. You’ll still need device cgroup rules, seccomp, or the runtime’s GPU device passthrough to limit that surface. - Not a network sandbox: Landlock does not control sockets, so you still need Docker network modes or seccomp/AppArmor for egress restrictions.
Practical approaches:
Option A, an entrypoint wrapper (simplest): a small C/Rust/Python wrapper that creates a Landlock ruleset; adds rules for the model dir (read + execute), the config dir (read/write), the Unix socket directory (read/write/create), proc/sys (deny or very limited); applies the ruleset to itself (landlock_restrict_self); then execs ollama. Set the wrapper as the container ENTRYPOINT.
Option B, an existing Landlock-enabled launcher: firejail with Landlock support (ongoing/experimental), or a custom runsc/youki shim — though the wrapper gives the most control over ollama’s exact paths.
Example ruleset sketch (conceptual C, not something I’m shipping yet):
/* Pseudo-code; link against liblandlock or use raw syscalls */
struct landlock_ruleset_attr ruleset_attr = {
.handled_access = LANDLOCK_ACCESS_FS_READ
| LANDLOCK_ACCESS_FS_EXECUTE
| LANDLOCK_ACCESS_FS_WRITE
| LANDLOCK_ACCESS_FS_CREATE
| LANDLOCK_ACCESS_FS_REMOVE
| LANDLOCK_ACCESS_FS_TRUNCATE,
};
int ruleset_fd = landlock_create_ruleset(&ruleset_attr, sizeof(ruleset_attr), 0);
struct landlock_path_beneath_attr beneath = {
.parent_fd = open(models_dir, O_PATH|O_CLOEXEC),
};
landlock_add_rule(ruleset_fd, LANDLOCK_RULE_PATH_BENEATH, &beneath, 0);
/* Repeat for config dir, socket dir, etc. */
landlock_restrict_self(ruleset_fd, 0);
execv("/usr/bin/ollama", argv);
Recommended layering — don’t rely on Landlock alone. Combine it with:
- seccomp-bpf: block
mount,ptrace,open_by_handle_at,openat2withRESOLVE_NO_MAGICLINKS, etc. - read-only root filesystem (
--read-onlyin Docker) with tmpfs only where ollama needs to write. - the Docker user namespace remap (already done) to map container root to an unprivileged host UID.
--cap-drop ALL, and don’t addCAP_SYS_ADMINback.
What to watch for with ollama specifically: it may need to download models (network + writes to the model store) — if you pre-pull models and run offline, you can make the model directory read-only after population. It may create temp files in tmp or its working directory — include those in the ruleset. And if it spawns child processes (e.g. for model runners), Landlock’s inheritance helps, but verify those children don’t need broader paths.
seccomp-bpf
From Oct 24 (edited):
prob should revisit this since I haven’t really checked to see what it actually needs (seemed to think nvidia-ctk will want ioctl and mmap which makes sense
…followed by this rough profile:
{
"defaultAction": "SCMP_ACT_ERRNO",
"architectures": [
"SCMP_ARCH_X86_64",
"SCMP_ARCH_AARCH64"
],
"syscalls": [
{
"names": [
"accept", "accept4", "bind", "clone", "clone3", "close", "connect",
"epoll_create1", "epoll_ctl", "epoll_pwait", "epoll_wait", "exit",
"exit_group", "fcntl", "fstat", "futex", "getdents64", "getpid",
"gettid", "gettimeofday", "ioctl", "listen", "lseek", "madvise",
"mmap", "mprotect", "munmap", "nanosleep", "openat", "pipe2",
"poll", "prctl", "read", "readv", "recvfrom", "recvmsg", "rt_sigaction",
"rt_sigprocmask", "rt_sigreturn", "sched_getaffinity", "sched_yield",
"sendmsg", "sendto", "set_robust_list", "setsockopt", "shutdown",
"stat", "write", "writev", "uname", "brk", "clock_gettime",
"getuid", "geteuid", "getgid", "getegid", "access", "tgkill",
"socket", "getsockopt", "getsockname", "getpeername", "dup", "dup2"
],
"action": "SCMP_ACT_ALLOW"
}
]
}
–cap-add=SYS_ADMIN, just to see
From Oct 24 — the firejail landlock link and this experiment:
root@spark-4d11:/home/sq# docker run --rm --name ollama2 --cap-add=SYS_ADMIN \
--entrypoint /bin/bash ollama/ollama -c 'cat /proc/1/status' | grep CapEff
CapEff: 00000000a82425fb
root@spark-4d11:/home/sq#
this should be an option.
Your new hex mask 00000000a82425fb proves that cap_sys_admin is now active. Comparing the two side-by-side, you can see the exact bit that changed:
- Old Mask:
00000000a80425fb - New Mask:
00000000a82425fb
The 0 flipped to a 2 in that specific hexadecimal position, which represents bit 21 — the exact position assigned to CAP_SYS_ADMIN in the Linux kernel. Your container now has the administrative privileges it was missing.
user namespaces are not a security boundary
From Oct 24: edera.dev — user namespaces are not a security boundary. This is the piece I keep going back and forth on — whether userns-remap is “containment” or just “less root, more inconvenience”. Namespace isolation plus mapped UIDs is real, but if it isn’t the boundary, the interesting questions become what the process actually needs, what it touches, and what mediates that — which is where the Landlock and seccomp items above come in, and where I still owe the audit.