Initial Setup
This guide details how the architecture of this project was developed. For streamlined instructions on how to install and start using this project, see the Installation page.
The structure of this project is based on the incredibly helpful guide, Py-Pkgs. Several of the sections below are summarizations of parts of that guide, with details of the changes I made for this specific setup.
Contents
The order of operations
When initially setting up this project, I installed the uv package manager, created the package structure of the project, and added Python package dependencies all before writing the Podman container.
It is likely that one could avoid installing uv on their host system by first setting up the container and, once inside the container, create the package structure.
However, I present those steps first as I have not tested that possibility and thus present them in the manner I followed, outside the container.
I do, however, place the section on the virtual environment and installing package dependencies after creating the container in this document because I have confirmed that adding packages with uv inside the container works.
The uv package manager
In Py-Pkgs Chapter 2, they suggest to use Miniconda to create an environment and use poetry to manage dependencies.
For this project, I decided to use uv instead.
From the instructions for Installing uv, I used Homebrew.
I’ve truncated the output below for brevity.
user@local:~$ brew install uv
==> Auto-updating Homebrew...
Adjust how often this is run with `$HOMEBREW_AUTO_UPDATE_SECS` or disable with
`$HOMEBREW_NO_AUTO_UPDATE=1`. Hide these hints with `$HOMEBREW_NO_ENV_HINTS=1` (see `man brew`).
==> Downloading https://ghcr.io/v2/homebrew/core/portable-ruby/blobs/sha256:7c7830166a509857669c544dcba7a0d08ca656a3da073c68826ca0a5b1b56b12
################################################################################################################ 100.0%
==> Pouring portable-ruby-4.0.2_1.catalina.bottle.tar.gz
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
...
==> Summary
🍺 /usr/local/Cellar/uv/0.11.6: 17 files, 54.0MB
I then ensured that I had the most up-to-date version of uv installed.
user@local:~$ brew upgrade uv
Warning: uv 0.11.6 already installed
Creating a package structure
Using uv init
To start a project, I simply need to use uv to initiate one, following the Working on projects guide.
I chose the name arctichoke to stand for “Sea Ice Choke Points,” trying to balance brevity and descriptiveness.
First, I navigated to the directory in which I want my project to be, <absolute/path/to/project>, then used the uv init command with the name for the project and the --package flag.
user@local:~$ cd /<absolute/path/to/project>
user@local:/<absolute/path/to/project>$ uv init arctichoke --package
Initialized project `arctichoke` at `/<absolute/path/to/project>/arctichoke`
This creates a very simple directory structure for the project.
user@local:/<absolute/path/to/project>$ tree arctichoke
├── .gitignore
├── .python-version
├── README.md
├── main.py
└── pyproject.toml
Version control and GitHub
Following Py-Pkgs Section 3.3. Put your package under version control, I initiated git for my new repository.
user@local:arctichoke$ git init
Initialized empty Git repository in /<absolute/path/to/project>/arctichoke/.git/
I then added and committed the initial structure and pushed to a new GitHub repository for the project. I am working in VSCodium, which I’d already set up to connect to my GitHub account, so the process was as simple as pressing the “push” button in the GUI.
Setting the default branch
For the version of git I have, it still sets the default branch as master instead of main.
I’m following the guide from Geeks for Geeks on How to Change Git Default Branch From Master?
First, I renamed the local branch.
user@local:/<absolute/path/to/project>$ cd arctichoke
user@local:arctichoke$ git branch -m master main
user@local:arctichoke$ git push -u origin main
Total 0 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'main' on GitHub by visiting:
remote: https://github.com/scheemik/arctichoke/pull/new/main
remote:
To https://github.com/scheemik/arctichoke.git
* [new branch] main -> main
Branch 'main' set up to track remote branch 'main' from 'origin'.
Then, on GitHub, I went to the “Settings” tab for the repository, then confirmed I was in the “General” section on the sidebar.
Under the heading “Default branch,” I clicked the button to “Switch to another branch,” selected main, then hit “Update.”
Next, I deleted the master branch from the remote.
user@local:arctichoke$ git push origin --delete master
To https://github.com/scheemik/arctichoke.git
- [deleted] master
Since the repository had not been cloned anywhere else at this point, that was all I needed to do.
Podman
Podman is an open-source tool by Red Hat for creating and managing containers, similar to Docker.
Containerization allows the creation of isolated, reproducible computing environments, making it easier to ensure code will run the same way across different systems.
While virtual environments for Python, such as venv or conda, create reproducible environments for Python packages, containers create reproducible environments for an entire operating system.
This allows the inclusion and management of non-Python software in a project, such as the cdo and esgpull tools used in this project.
Installing Podman
I installed podman on my MacBook following the Podman Installation Instructions.
I downloaded the podman-installer-macos-universal.pkg version from the podman GitHub latest release (v5.8.2) page.
Opening that brought me through the familiar GUI software installation process common to many applications for macOS.
I agreed to the license, selected which users for whom I will install it, then watched the loading bar.
It took less than a minute.
This process also added an item from “Red Hat” in Settings -> Login Items & Extensions -> Allow in the Background.
I assume this is necessary for this to actually run properly.
I then moved the .pkg file to the trash.
The next step in the Podman Installation Instructions for macOS is to setup the Podman virtual machine. This is necessary when running Podman on macOS or Windows as it needs to run on a Linux system, therefore I need to initialize a virtual machine to actually run Podman.
user@local:~$ podman machine init
Looking up Podman Machine image at quay.io/podman/machine-os:5.8 to create VM
Getting image source signatures
Copying blob 5efcf56a5999 done |
Copying config 44136fa355 done |
Writing manifest to image destination
5efcf56a599919c786136faf8e4f48b25bf3865b5e8ea3302f6d705ba750afec
Extracting compressed file: podman-machine-default-amd64.raw: done
Machine init complete
To start your machine run:
podman machine start
That took about 3 minutes. Next, I started the virtual machine.
user@local:~$ podman machine start
Starting machine "podman-machine-default"
This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:
podman machine set --rootful
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
That took about a minute.
Then, I took a look at the podman information.
There is quite a lot of information, so I’ll collapse most of the output.
user@local:~$ podman info
Client:
APIVersion: 5.8.2
BuildOrigin: pkginstaller
Built: 1776189122
BuiltTime: Tue Apr 14 13:52:02 2026
GitCommit: 5b263b5f5b48004a87caac44e67349a8266d9ef4
GoVersion: go1.26.2
Os: darwin
OsArch: darwin/amd64
Version: 5.8.2
...
Expand for more output
...
host:
arch: amd64
buildahVersion: 1.43.1
cgroupControllers:
- cpu
- io
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.2.1-2.fc43.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.2.1, commit: '
cpuUtilization:
idlePercent: 92.76
systemPercent: 4.49
userPercent: 2.75
cpus: 4
databaseBackend: sqlite
distribution:
distribution: fedora
variant: coreos
version: "43"
emulatedArchitectures:
- linux/arm64
- linux/arm64be
eventLogger: journald
freeLocks: 2048
hostname: localhost.localdomain
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 1000000
uidmap:
- container_id: 0
host_id: 502
size: 1
- container_id: 1
host_id: 100000
size: 1000000
kernel: 6.19.7-200.fc43.x86_64
linkmode: dynamic
logDriver: journald
memFree: 1404866560
memTotal: 2046451712
networkBackend: netavark
networkBackendInfo:
backend: netavark
defaultNetwork: podman
dns:
package: aardvark-dns-1.17.0-1.fc43.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.17.0
package: netavark-1.17.2-1.fc43.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.17.2
ociRuntime:
name: crun
package: crun-1.25.1-1.fc43.x86_64
path: /usr/bin/crun
version: |-
crun version 1.25.1
commit: 156ae065d4a322d149c7307034f98d9637aa92a2
rundir: /run/user/502/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20260120.g386b5f5-1.fc43.x86_64
version: |
pasta 0^20260120.g386b5f5-1.fc43.x86_64
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: unix:///run/user/502/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.3.1-3.fc43.x86_64
version: |-
slirp4netns version 1.3.1
commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
libslirp: 4.9.1
SLIRP_CONFIG_VERSION_MAX: 6
libseccomp: 2.6.0
swapFree: 0
swapTotal: 0
uptime: 0h 1m 23.00s
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- docker.io
store:
configFile: /var/home/core/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /var/home/core/.local/share/containers/storage
graphRootAllocated: 106769133568
graphRootUsed: 4263370752
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 0
runRoot: /run/user/502/containers
transientStore: false
volumePath: /var/home/core/.local/share/containers/storage/volumes
...
...
version:
APIVersion: 5.8.2
BuildOrigin: 'Copr: packit/containers-podman-28501'
Built: 1776038400
BuiltTime: Sun Apr 12 20:00:00 2026
GitCommit: 5b263b5f5b48004a87caac44e67349a8266d9ef4
GoVersion: go1.25.9 X:nodwarf5
Os: linux
OsArch: linux/amd64
Version: 5.8.2
I can also simply check the version of podman.
user@local:~$ podman version
Client: Podman Engine
Version: 5.8.2
API Version: 5.8.2
Go Version: go1.26.2
Git Commit: 5b263b5f5b48004a87caac44e67349a8266d9ef4
Built: Tue Apr 14 13:52:02 2026
Build Origin: pkginstaller
OS/Arch: darwin/amd64
Server: Podman Engine
Version: 5.8.2
API Version: 5.8.2
Go Version: go1.25.9 X:nodwarf5
Git Commit: 5b263b5f5b48004a87caac44e67349a8266d9ef4
Built: Sun Apr 12 20:00:00 2026
OS/Arch: linux/amd64
I can also see that I now have podman files in my user’s configuration folder.
user@local:~$ ls -la ~/.config/containers/
total 8
drwxr-xr-x 5 <user> staff 160 Apr 28 10:57 .
drwx------ 8 <user> staff 256 Apr 28 10:55 ..
drwxr-xr-x 3 <user> staff 96 Apr 28 10:55 podman
-rw-r--r-- 1 <user> staff 440 Apr 28 10:57 podman-connections.json
-rw-r--r-- 1 <user> staff 0 Apr 28 10:57 podman-connections.json.lock
Testing Podman
Next, I followed the Basic Setup and Use of Podman guide.
To start, I ran the sample nginx container, calling it basic_httpd.
user@local:~$ podman run --name basic_httpd -d -p 8080:80/tcp docker.io/nginx
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob sha256:677c631968686eeb23ab8dd436d49bde041266df5d8952f03d7a8c418643d4b5
Copying blob sha256:ce776bbcda0d6bf4da8df324b82066a03f45bfbbbe520df535293ae069994e84
Copying blob sha256:4677c2a9a3d4f9290cb784d95a9e16378655ecdd7df9e77668d3915262730d0b
Copying blob sha256:85c66128325abc04138f6944d943e5279375665f6dbefe7f4f6b5e9646d31998
Copying blob sha256:ff048f1f2159a060f69b1861ea262b839cc6e77a9389848929f70275eb7c9e29
Copying blob sha256:3531af2bc2a9c8883754652783cf96207d53189db279c9637b7157d034de7ecd
Copying blob sha256:801a1ad15b4e00add388aca409568400fb8071019d6ba83995f43170af7656fe
Copying config sha256:6c3a6ea6608c89c79027066654a2ef4f0fe58a7bf2c08cc3894733406e476602
Writing manifest to image destination
774a97cf2828429b1feeddae152869417a57cbdcc1e13c0b97ba777aafc762fc
This pulls the image from the web, builds it, and starts a container running that image. If the above command is run a second time, it will be able to skip the part where it downloads and builds the image. I can now list my running containers.
user@local:~$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
774a97cf2828 docker.io/library/nginx:latest nginx -g daemon o... About a minute ago Up About a minute 0.0.0.0:8080->80/tcp basic_httpd
Adding the -a flag shows “all” containers, but that doesn’t result in any different output than without it at this moment because I just have the one container.
Then, I can inspect that running container.
user@local:~$ podman inspect basic_httpd | grep IPAddress
"IPAddress": "10.88.0.3",
"IPAddress": "10.88.0.3",
This is different than what is shown in the guide which says:
“As the container is running in rootless mode, an IP address is not assigned and the value will be listed as ‘none’ in the output from inspect.”
Next, I tested the httpd server to make sure it is running on the expected port, displaying the index page.
user@local:~$ curl http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, nginx is successfully installed and working.
Further configuration is required for the web server, reverse proxy,
API gateway, load balancer, content cache, or other features.</p>
<p>For online documentation and support please refer to
<a href="https://nginx.org/">nginx.org</a>.<br/>
To engage with the community please visit
<a href="https://community.nginx.org/">community.nginx.org</a>.<br/>
For enterprise grade support, professional services, additional
security features and capabilities please refer to
<a href="https://f5.com/nginx">f5.com/nginx</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Using the Container ID from the podman ps command above, I can see the logs of this container.
user@local:~$ podman logs 774a97cf2828
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2026/04/28 15:13:47 [notice] 1#1: using the "epoll" event method
2026/04/28 15:13:47 [notice] 1#1: nginx/1.29.8
2026/04/28 15:13:47 [notice] 1#1: built by gcc 14.2.0 (Debian 14.2.0-19)
2026/04/28 15:13:47 [notice] 1#1: OS: Linux 6.19.7-200.fc43.x86_64
2026/04/28 15:13:47 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 524288:524288
2026/04/28 15:13:47 [notice] 1#1: start worker processes
2026/04/28 15:13:47 [notice] 1#1: start worker process 24
2026/04/28 15:13:47 [notice] 1#1: start worker process 25
2026/04/28 15:13:47 [notice] 1#1: start worker process 26
2026/04/28 15:13:47 [notice] 1#1: start worker process 27
10.88.0.2 - - [28/Apr/2026:15:21:21 +0000] "GET / HTTP/1.1" 200 896 "-" "curl/8.7.1" "-"
I can also see the httpd pid with top using the same container ID.
user@local:~$ podman top 774a97cf2828
USER PID PPID %CPU ELAPSED TTY TIME COMMAND
root 1 0 0.000 10m43.083334622s ? 0s nginx: master process nginx -g daemon off;
nginx 24 1 0.000 10m42.083710443s ? 0s nginx: worker process
nginx 25 1 0.000 10m42.083812615s ? 0s nginx: worker process
nginx 26 1 0.000 10m42.083879016s ? 0s nginx: worker process
nginx 27 1 0.000 10m42.083942087s ? 0s nginx: worker process
Next, I’ll be following part of the guide How to run your first rootless container with Podman.
One of the benefits of Podman is the ability to run containers as “rootless” which adds an extra layer of security.
The podman exec <container_name> command allows you to execute commands within a container.
I will use this to execute the whoami command to ask who the container thinks they are.
user@local:~$ podman exec basic_httpd whoami
root
So, the container is running as root.
Next, I’ll see who owns the process.
user@local:~$ podman top basic_httpd user huser
USER HUSER
root 502
nginx 100100
nginx 100100
nginx 100100
nginx 100100
I can see here that the user id between root and the nginx container don’t match, so it seems like that confirms that even though, inside the container, the service is running as root, outside the container, it is not.
Next, I can stop the container.
user@local:~$ podman stop basic_httpd
basic_httpd
Then, I’ll remove this container and check to make sure it is gone.
user@local:~$ podman rm basic_httpd
basic_httpd
user@local:~$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Pod Manager extension for VSCodium
I went to VSCodium and installed the “Pod Manager” extension from user dreamcatcher45.
When I first went into the extension, I got the following error:
Failed to get containers: Error: Command failed: podman container ls -a --format "{{.ID}}|{{.Names}}|{{.Status}}|{{.Labels}}"
/bin/sh: podman: command not found
I figured out that VSCodium had been open since before I installed podman, and I couldn’t actually run podman version in the VSCodium terminal panel.
After I restarted VSCodium, both the podman version command and the Pod Manager extension worked.
In the sidebar of VSCodium, I clicked on the Pod Manger icon, which looks like a stylized seal. Currently, the dropdown for “Containers” is empty because I remove the ones I was testing above. However, under the “Images” dropdown, I see “docker.io/library/nginx:latest (7aaca76c508f).” This is the image that was built when Testing Podman.
Containers get added to the list in the Pod Manager sidebar when they start running.
I can start the basic_httpd container again.
user@local:arctichoke$ podman run --name basic_httpd -d -p 8080:80/tcp docker.io/nginx
9ab04fb2f7c04e2a1cf263ee46b33a00cde5fc7c89e71926f7a65aeae4e35dd7
Since this image was already built, all that needed to be done was start a container with the image.
Hitting the refresh button next to the overall “Resources” dropdown at the top of the Pod Manager sidebar reveals this basic_httpd container.
The container name has a long alphanumeric string in parentheses appended which changes every time the container is started.
When hovering over the name of the container, several options appear. One of these is to “Open in Terminal” which does the equivalent of opening a new terminal inside VSCodium and entering the container, all in one click. From there, commands can be executed inside the container.
podman exec -it 9ab04fb2f7c0 /bin/sh
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
user@local:arctichoke$ podman exec -it 9ab04fb2f7c0 /bin/sh
# basic_httpd whoami
root
#
Then, to clean up, I will stop and remove this basic_httpd from outside the container.
user@local:~$ podman stop basic_httpd
basic_httpd
user@local:~$ podman rm basic_httpd
basic_httpd
user@local:~$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
If any terminals had been left open inside the container, they will have automatically exited when the container is removed.
Building a simple container
Above, I downloaded an existing image docker.io/nginx from a repository.
For this project, I want to define my own.
The setup of a container is defined by a scripted that is named Containerfile with no file extension.
A good way to start a container is to load an existing minimal distribution.
I chose the trixie-slim version of Debian as it is lightweight and uses Python 3.13 by default.
In Py-Pkgs Chapter 2, they suggest using Miniconda to create an environment and the poetry package to manage dependencies.
For this project, I decided to use uv instead as it has the ability to manage tools and add ephemeral packages when testing out builds.
I can create a minimal Containerfile to test running trixie-slim with Python 3.13 and uv installed.
# ---- Stage 1: get uv binary ----
FROM ghcr.io/astral-sh/uv:latest AS uv
# ---- Stage 2: main image ----
FROM debian:trixie-slim
ENV DEBIAN_FRONTEND=noninteractive
# ---- System dependencies ----
RUN apt-get update && apt-get install -y python3.13
# ---- Make python command available ----
RUN ln -s /usr/bin/python3.13 /usr/bin/python
# ---- Copy uv from official image ----
COPY --from=uv /uv /usr/local/bin/uv
# ---- Set working directory ----
WORKDIR /workspace
# Default shell
CMD ["/bin/bash"]
Next, I’ll build the image. A lot of information is output to the console in the build process, so I’ll collapse most of the output for brevity.
user@local:arctichoke$ podman build -f Containerfile -t test_trixie .
[1/2] STEP 1/1: FROM ghcr.io/astral-sh/uv:latest AS uv
Trying to pull ghcr.io/astral-sh/uv:latest...
Getting image source signatures
Copying blob sha256:dfd617f69b3af15e1fad323e893c535ef022c9efb9528fb53ad6c8ec44741d5a
Copying blob sha256:b4aebd139799aa429f45564ceac662ba2bc66115fb8c0318cd3b2368ea7517e4
Copying config sha256:b960411dc937f9b4d9762349f5f77772d36dead003baa3bc01330abe8e1f38a6
Writing manifest to image destination
--> b960411dc937
[2/2] STEP 1/7: FROM debian:trixie-slim
Resolved "debian" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/debian:trixie-slim...
...
Expand for more output
...
Getting image source signatures
Copying blob sha256:5b4d6ff92fc4e14e911b7753c954fac965d48c40fe1075758d284148ccace970
Copying config sha256:f283d70f878433b889e4b9252110fad858e0e0887df5bac91cd2ad4ccb2b3a2a
Writing manifest to image destination
[2/2] STEP 2/7: ENV DEBIAN_FRONTEND=noninteractive
--> 5808b4b9aad3
[2/2] STEP 3/7: RUN apt-get update && apt-get install -y python3.13
Get:1 http://deb.debian.org/debian trixie InRelease [140 kB]
Get:2 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Get:3 http://deb.debian.org/debian-security trixie-security InRelease [43.4 kB]
Get:4 http://deb.debian.org/debian trixie/main amd64 Packages [9671 kB]
Get:5 http://deb.debian.org/debian trixie-updates/main amd64 Packages [5412 B]
Get:6 http://deb.debian.org/debian-security trixie-security/main amd64 Packages [192 kB]
Fetched 10.1 MB in 2s (4109 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
ca-certificates libexpat1 libffi8 libgpm2 libncursesw6 libpython3.13-minimal
libpython3.13-stdlib libreadline8t64 media-types netbase openssl
python3.13-minimal readline-common
Suggested packages:
gpm python3.13-venv python3.13-doc binutils binfmt-support readline-doc
The following NEW packages will be installed:
ca-certificates libexpat1 libffi8 libgpm2 libncursesw6 libpython3.13-minimal
libpython3.13-stdlib libreadline8t64 media-types netbase openssl python3.13
python3.13-minimal readline-common
0 upgraded, 14 newly installed, 0 to remove and 0 not upgraded.
Need to get 8020 kB of archives.
After this operation, 27.1 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian trixie/main amd64 libexpat1 amd64 2.7.1-2 [108 kB]
Get:2 http://deb.debian.org/debian trixie/main amd64 libpython3.13-minimal amd64 3.13.5-2+deb13u2 [862 kB]
Get:3 http://deb.debian.org/debian trixie/main amd64 python3.13-minimal amd64 3.13.5-2+deb13u2 [2217 kB]
Get:4 http://deb.debian.org/debian trixie/main amd64 netbase all 6.5 [12.4 kB]
Get:5 http://deb.debian.org/debian trixie/main amd64 readline-common all 8.2-6 [69.4 kB]
Get:6 http://deb.debian.org/debian trixie/main amd64 openssl amd64 3.5.6-1~deb13u1 [1503 kB]
Get:7 http://deb.debian.org/debian trixie/main amd64 ca-certificates all 20250419 [162 kB]
Get:8 http://deb.debian.org/debian trixie/main amd64 media-types all 13.0.0 [29.3 kB]
Get:9 http://deb.debian.org/debian trixie/main amd64 libffi8 amd64 3.4.8-2 [24.1 kB]
Get:10 http://deb.debian.org/debian trixie/main amd64 libgpm2 amd64 1.20.7-11+b2 [14.4 kB]
Get:11 http://deb.debian.org/debian trixie/main amd64 libncursesw6 amd64 6.5+20250216-2 [135 kB]
Get:12 http://deb.debian.org/debian trixie/main amd64 libreadline8t64 amd64 8.2-6 [169 kB]
Get:13 http://deb.debian.org/debian trixie/main amd64 libpython3.13-stdlib amd64 3.13.5-2+deb13u2 [1958 kB]
Get:14 http://deb.debian.org/debian trixie/main amd64 python3.13 amd64 3.13.5-2+deb13u2 [757 kB]
Preconfiguring packages ...
Fetched 8020 kB in 1s (6819 kB/s)
Selecting previously unselected package libexpat1:amd64.
(Reading database ... 4936 files and directories currently installed.)
Preparing to unpack .../00-libexpat1_2.7.1-2_amd64.deb ...
Unpacking libexpat1:amd64 (2.7.1-2) ...
Selecting previously unselected package libpython3.13-minimal:amd64.
Preparing to unpack .../01-libpython3.13-minimal_3.13.5-2+deb13u2_amd64.deb ...
Unpacking libpython3.13-minimal:amd64 (3.13.5-2+deb13u2) ...
Selecting previously unselected package python3.13-minimal.
Preparing to unpack .../02-python3.13-minimal_3.13.5-2+deb13u2_amd64.deb ...
Unpacking python3.13-minimal (3.13.5-2+deb13u2) ...
Selecting previously unselected package netbase.
Preparing to unpack .../03-netbase_6.5_all.deb ...
Unpacking netbase (6.5) ...
Selecting previously unselected package readline-common.
Preparing to unpack .../04-readline-common_8.2-6_all.deb ...
Unpacking readline-common (8.2-6) ...
Selecting previously unselected package openssl.
Preparing to unpack .../05-openssl_3.5.6-1~deb13u1_amd64.deb ...
Unpacking openssl (3.5.6-1~deb13u1) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../06-ca-certificates_20250419_all.deb ...
Unpacking ca-certificates (20250419) ...
Selecting previously unselected package media-types.
Preparing to unpack .../07-media-types_13.0.0_all.deb ...
Unpacking media-types (13.0.0) ...
Selecting previously unselected package libffi8:amd64.
Preparing to unpack .../08-libffi8_3.4.8-2_amd64.deb ...
Unpacking libffi8:amd64 (3.4.8-2) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../09-libgpm2_1.20.7-11+b2_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.7-11+b2) ...
Selecting previously unselected package libncursesw6:amd64.
Preparing to unpack .../10-libncursesw6_6.5+20250216-2_amd64.deb ...
Unpacking libncursesw6:amd64 (6.5+20250216-2) ...
Selecting previously unselected package libreadline8t64:amd64.
Preparing to unpack .../11-libreadline8t64_8.2-6_amd64.deb ...
Adding 'diversion of /lib/x86_64-linux-gnu/libhistory.so.8 to /lib/x86_64-linux-gnu/libhistory.so.8.usr-is-merged by libreadline8t64'
Adding 'diversion of /lib/x86_64-linux-gnu/libhistory.so.8.2 to /lib/x86_64-linux-gnu/libhistory.so.8.2.usr-is-merged by libreadline8t64'
Adding 'diversion of /lib/x86_64-linux-gnu/libreadline.so.8 to /lib/x86_64-linux-gnu/libreadline.so.8.usr-is-merged by libreadline8t64'
Adding 'diversion of /lib/x86_64-linux-gnu/libreadline.so.8.2 to /lib/x86_64-linux-gnu/libreadline.so.8.2.usr-is-merged by libreadline8t64'
Unpacking libreadline8t64:amd64 (8.2-6) ...
Selecting previously unselected package libpython3.13-stdlib:amd64.
Preparing to unpack .../12-libpython3.13-stdlib_3.13.5-2+deb13u2_amd64.deb ...
Unpacking libpython3.13-stdlib:amd64 (3.13.5-2+deb13u2) ...
Selecting previously unselected package python3.13.
Preparing to unpack .../13-python3.13_3.13.5-2+deb13u2_amd64.deb ...
Unpacking python3.13 (3.13.5-2+deb13u2) ...
Setting up libexpat1:amd64 (2.7.1-2) ...
Setting up media-types (13.0.0) ...
Setting up libgpm2:amd64 (1.20.7-11+b2) ...
Setting up libpython3.13-minimal:amd64 (3.13.5-2+deb13u2) ...
Setting up libncursesw6:amd64 (6.5+20250216-2) ...
Setting up libffi8:amd64 (3.4.8-2) ...
Setting up python3.13-minimal (3.13.5-2+deb13u2) ...
Setting up netbase (6.5) ...
Setting up openssl (3.5.6-1~deb13u1) ...
Setting up readline-common (8.2-6) ...
Setting up ca-certificates (20250419) ...
Updating certificates in /etc/ssl/certs...
150 added, 0 removed; done.
Setting up libreadline8t64:amd64 (8.2-6) ...
Setting up libpython3.13-stdlib:amd64 (3.13.5-2+deb13u2) ...
Setting up python3.13 (3.13.5-2+deb13u2) ...
Processing triggers for libc-bin (2.41-12+deb13u3) ...
Processing triggers for ca-certificates (20250419) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
--> 4f123f8d89fe
[2/2] STEP 4/7: RUN ln -s /usr/bin/python3.13 /usr/bin/python
--> 6dd8a6c94600
[2/2] STEP 5/7: COPY --from=uv /uv /usr/local/bin/uv
--> b907eed1ae8b
[2/2] STEP 6/7: WORKDIR /workspace
--> 55b854bc8dcd
[2/2] STEP 7/7: CMD ["/bin/bash"]
[2/2] COMMIT test_trixie
--> 7bab3cfc2aa3
...
...
Successfully tagged localhost/test_trixie:latest
7bab3cfc2aa34bf43d9a63ccb8428a859c9f64a807186a4262b17bc5ffe0b4eb
Then, I’ll run the container with the following flags (see the Podman run docs for details):
-i: Interactive“When set to true, make
stdinavailable to the contained process. If false, thestdinof the contained process is empty and immediately closed.”
-t: TTY“Allocate a pseudo-TTY. The default is false. When set to true, Podman allocates a pseudo-tty and attach to the standard input of the container. This can be used, for example, to run a throwaway interactive shell.”
--rm: Remove upon exit“Automatically remove the container and any anonymous unnamed volume associated with the container when it exits. The default is false.”
--name: Container name“Assign a name to the container.” This can be completely different from the name of the image it is built from.
user@local:arctichoke$ podman run -it --rm --name container_name test_trixie
root@<container_id>:/workspace#
Where <container_id> is a 12-digit identifier of the container.
While this container is running, I can hit the refresh button in the Pod Manager sidebar to see that there is a new container named container_name.
I can also verify that Python and uv are installed inside the container.
root@<container_id>:/workspace# python
Python 3.13.5 (main, May 5 2026, 21:05:52) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
root@<container_id>:/workspace# uv --version
uv 0.11.17 (x86_64-unknown-linux-musl)
root@<container_id>:/workspace#
When I am done, I can exit the container.
root@<container_id>:/workspace# exit
exit
user@local:arctichoke$
Upon exiting the container, I can refresh the Pod Manager sidebar again to see that the container removed itself upon exit because of the --rm flag.
This is important to keep from building up a large number of idle containers when starting a particular image many times.
With this working, the next step is to build out the Containerfile to set up the development environment with all the necessary packages, tools, and data access.
The Containerfile
Below is the current Containerfile used to build the image for this project.
# Build with the following command
# podman build -f .devcontainer/Containerfile -t <image_name> . | tee .devcontainer/<log_file_name>.log
# ---- Stage 1: get uv image ----
FROM ghcr.io/astral-sh/uv:trixie-slim@sha256:5cbec7ab7753a6c763c6dda6a38f085c8c585ec9f53cfb4e7368b79ca30bc881 AS uv
# ---- Stage 2: main image ----
FROM debian:trixie-slim@sha256:e18da95f66066b7c5fa31491b524e83121271eca59a3d140f4906c8d0a090367
## Define environment variables
ENV DEBIAN_FRONTEND=noninteractive
# Rename the default directory for the `uv` virtual environment to avoid
# conflicts with using the environment in/outside the container
ENV UV_PROJECT_ENVIRONMENT=/workspace/.cvenv
ENV UV_LINK_MODE=copy
# System + scientific dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
git \
build-essential \
pkg-config \
libnetcdf-dev \
netcdf-bin \
libhdf5-dev \
libcurl4-openssl-dev \
libssl-dev \
cdo \
nco \
python3.13 \
python3.13-venv \
python3-pip \
chromium \
chromium-driver \
fonts-liberation \
&& rm -rf /var/lib/apt/lists/*
# Make `python` command available via symlink
RUN ln -s /usr/bin/python3.13 /usr/bin/python
# Copy uv from official image
COPY --from=uv /uv /usr/local/bin/uv
# Working directory
WORKDIR /workspace
# Copy dependency files for the venv
COPY pyproject.toml uv.lock ./
COPY README.md ./
COPY src ./src
# Install dependencies via `uv` and start a kernel for Jupyter notebooks
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync \
&& uv run python -m ipykernel install --sys-prefix --name python3 --display-name "arctichoke (container)"
# Trigger downloads of commonly used Natural Earth datasets
RUN /workspace/.cvenv/bin/python - <<'EOF'
import cartopy.io.shapereader as shp
shp.natural_earth(resolution='110m', category='physical', name='coastline')
EOF
# Set up the `esgpull` install
COPY .devcontainer/esgpull_entrypoint.sh /esgpull_entrypoint.sh
RUN chmod +x /esgpull_entrypoint.sh
ENTRYPOINT ["/esgpull_entrypoint.sh"]
# Expose the port for the Jupyter server
EXPOSE 8888
CMD ["bash", "-lc", "exec uv run jupyter lab \
--ip=0.0.0.0 \
--port=8888 \
--no-browser \
--allow-root \
--IdentityProvider.token='' \
--ServerApp.password=''"]
Note that this is meant to be executed via the start_container.sh script.
However, if testing a new build, it can be useful to pipe the output of podman build to a log file using tee.
user@local:arctichoke$ podman build -f .devcontainer/Containerfile -t <image_name> . | tee .devcontainer/<log_file_name>.log
I’ll explain each section of the Containerfile in detail below.
Pinning the versions of uv and trixie-slim
For reproducibility purposes, I decided to pin the exact versions of uv and trixie-slim that my container uses.
After Building a simple container, I now see the following images in the Pod Manger sidebar:
ghcr.io/astral-sh/uv:latestdocker.io/library/debian:trixie-slim
I can get the exact hashes of the versions of uv and trixie-slim from their manifests.
user@local:arctichoke$ podman image inspect debian:trixie-slim --format '{{.Digest}}'
sha256:e18da95f66066b7c5fa31491b524e83121271eca59a3d140f4906c8d0a090367
user@local:arctichoke$ podman image inspect ghcr.io/astral-sh/uv --format '{{.Digest}}'
sha256:5cbec7ab7753a6c763c6dda6a38f085c8c585ec9f53cfb4e7368b79ca30bc881
Defining environment variables
The Containerfile defines a few environment variables to facilitate building the image.
DEBIAN_FRONTEND=noninteractiveAccording to the askUbuntu post DEBIAN_FRONTEND environment variable, this prevents installations from getting stuck on interactive processes, such as when the user is asked to confirm or select something.
ENV UV_PROJECT_ENVIRONMENT=/workspace/.cvenvAccording to the Python Developer Tooling Handbook on How to customize uv’s virtual environment location, this renames the default directory for the
uvvirtual environment from.venvto.cvenvto avoid conflicts with using the environment inside vs. outside the container.Note that the use of the directory
/workspaceis detailed below.
ENV UV_LINK_MODE=copyAccording to the Python Developer Tooling Handbook on How to use
uvin a Dockerfile, this “tells uv to copy files instead of hard-linking them. When using Docker cache mounts, the cache and the target directory live on separate filesystems, so uv falls back to copying anyway. Setting this explicitly avoids the warning message.”
Installing system and scientific dependencies
The next block in the Containerfile uses apt-get to install the necessary system-level dependencies for the project.
The flags used here are:
-y: Assume yesFrom the Linux manual page for
apt-get, “Automatic yes to prompts. Assume ‘yes’ as answer to all prompts and run non-interactively. If an undesirable situation, such as changing a held package or removing an essential package, occurs thenapt-getwill abort.”
--no-install-recommends:According to the askUbuntu post How to not install recommended and suggested packages?, this flag prevents
apt-getfrom automatically installing recommended packages, keeping the container’s stack to a minimum, installing only what is required.
The packages installed in this block are:
ca-certificatescurlgitFor version control.
build-essentialpkg-configunziplibnetcdf-devFor working with netCDF files.
netcdf-binFor working with netCDF files.
libhdf5-devFor working with netCDF files.
libcurl4-openssl-devFor establishing Secure Sockets Layer (SSL) for internet connections.
libssl-devFor establishing Secure Sockets Layer (SSL) for internet connections.
cdoncopython3.13The version of Python used in this project.
python3.13-venvFor using Python in a virtual environment.
python3-pipFor installing packages that
uvcannot handle natively.
chromiumFor taking “screenshots” of
htmlmaps to produce.pngimages.
chromium-driverFor taking “screenshots” of
htmlmaps to produce.pngimages.
fonts-liberation
Preparing Python, uv, and the working directory
RUN ln -s /usr/bin/python3.13 /usr/bin/pythonThis command creates a symlink such that using the command
pythoncalls Python 3.13.This removes the need to specify the version of Python to use every time.
COPY --from=uv /uv /usr/local/bin/uvFrom Using
uvin Docker, this command copiesuvfrom where it was downloaded into theusrdirectory for ease of use.
WORKDIR /workspaceThis sets the working directory to be named
/workspace.The choice of the name is arbitrary, however sets a specific file path that can be expected by other parts of the project.
This essentially renames the root of the project directory on your computer (
arctichoke) to be/workspaceinside the container.
The next three commands set up dependency files for the virtual environment.
COPY pyproject.toml uv.lock ./COPY README.md ./COPY src ./src
uv dependencies and Jupyter kernel
The next block in the Containerfile is:
...
# Install dependencies via `uv` and start a kernel for Jupyter notebooks
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync \
&& uv run python -m ipykernel install --sys-prefix --name python3 --display-name "arctichoke (container)"
...
This installs and/or updates all the dependencies defined in the pyproject.toml file using uv sync.
Specifying --mount=type=cache,target=/root/.cache/uv will cache the packages with uv, making subsequent builds much faster.
Then, it starts a kernel for using the Python virtual environment in Jupyter notebooks.
Downloading commonly used Natural Earth
When making maps, it is often useful to plot coastlines or other shapes to give context.
A common way of doing this in Python is from Natural Earth shape files.
There are a couple of common files that the next block of the Containerfile downloads so that it won’t need to be downloaded the first time a plot is made that requires them.
...
# Trigger downloads of commonly used Natural Earth datasets
RUN /workspace/.cvenv/bin/python - <<'EOF'
import cartopy.io.shapereader as shp
shp.natural_earth(resolution='110m', category='physical', name='coastline')
EOF
...
If this is not present in the image, the following warning occurs when making a plot that requires downloading shape files.
/workspace/.venv/lib/python3.13/site-packages/cartopy/io/__init__.py:242: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/110m_physical/ne_110m_land.zip
warnings.warn(f'Downloading: {url}', DownloadWarning)
Setting up esgpull install
This project uses the esgpull tool to download HighResMIP data files.
This next block sets up an install of esgpull on an external volume.
...
# Set up the `esgpull` install
COPY .devcontainer/esgpull_entrypoint.sh /esgpull_entrypoint.sh
RUN chmod +x /esgpull_entrypoint.sh
ENTRYPOINT ["/esgpull_entrypoint.sh"]
...
This uses the separate script included with this project at .devcontainer/esgpull_entrypoint.sh which is shown below.
#!/usr/bin/env bash
# Set shell behavior to exit immediately if non-zero exit code encountered
set -e
# Make sure to start in the /workspace directory of the container
cd /workspace
# Activate venv
if [[ -f ".cvenv/bin/activate" ]]; then
source .cvenv/bin/activate
else
echo "ERROR: Could not activate virtual environment at .cvenv/bin/activate"
exit 1
fi
# Run esgpull install (idempotent, only takes affect the first time it is executed)
if [[ -d "/arctichoke_data" ]]; then
cd /arctichoke_data
# Use `|| true` to ensure the following line always returns exit status 0
# even if the `esgpull` command fails
uv run esgpull self install bergybits || true
else
echo "WARNING: Directory /arctichoke_data not found. No `esgpull` install created."
fi
# Go back and start default CMD
cd /workspace
# Ensure the rest of the CMD in the Containerfile runs
exec "$@"
The .devcontainer/esgpull_entrypoint.sh script assumes that the directory in which the data on the external volume is stored has been defined as /arctichoke_data, something which is set when executing the podman run command (see The start_container script).
For more information on esgpull, see Initializing esgpull.
Setting up the Jupyter server
The last block of the Containerfile exposes port 8888 of the container and sets up the Jupyter server to run on that port with no password or identity token.
...
# Expose the port for the Jupyter server
EXPOSE 8888
CMD ["bash", "-lc", "exec uv run jupyter lab \
--ip=0.0.0.0 \
--port=8888 \
--no-browser \
--allow-root \
--IdentityProvider.token='' \
--ServerApp.password=''"]
Note that port 8888 in the container will be set to connect to a different port (8889) on the host machine later.
The start_container script
The Containerfile defines how the image for this project should be built.
I created a script called start_container.sh which ultimately starts the container from that image, but first checks to make sure the Podman virtual machine is running and whether the image exists already.
The script is shown below:
#!/usr/bin/env bash
# This script should be run from the project main directory
set -euo pipefail
# ---- Ensure podman machine is running (macOS) ----
if ! podman machine inspect >/dev/null 2>&1; then
echo "No podman machine found. Initializing..."
podman machine init
fi
MACHINE_STATE=$(podman machine inspect --format '{{.State}}')
if [[ "$MACHINE_STATE" != "running" ]]; then
echo "Starting podman machine..."
podman machine start
fi
# ---- Set parameters ----
IMAGE="arctichoke"
CONTAINER_NAME="arctichoke_cont"
WORKDIR="/workspace"
# ---- Cleanup old container if it exists ----
podman rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
# ---- Ensure image exists ----
if ! podman image exists "$IMAGE"; then
echo "Image $IMAGE not found. Building from '.devcontainer/Containerfile'..."
podman build -f .devcontainer/Containerfile -t "$IMAGE" . | tee .devcontainer/build_container_log.txt
fi
# ---- Setup external hard drive access ----
export SICP_DATA_DIR=/Volumes/BERGY_BITS/arctichoke_data/
SICP_DATA_DIR="${SICP_DATA_DIR:-}"
if [[ -n "$SICP_DATA_DIR" ]]; then
if [[ ! -d "$SICP_DATA_DIR" ]]; then
echo "ERROR: SICP_DATA_DIR does not exist: $SICP_DATA_DIR"
exit 1
fi
fi
# ---- Build volume args ----
VOLUMES=(-v "$(pwd)":"$WORKDIR")
if [[ -n "$SICP_DATA_DIR" ]]; then
VOLUMES+=(-v "$SICP_DATA_DIR:/arctichoke_data")
fi
# ---- Run container (Jupyter starts automatically from CMD) ----
podman run -it --rm \
--name "$CONTAINER_NAME" \
-p 8889:8888 \
"${VOLUMES[@]}" \
-w "$WORKDIR" \
"$IMAGE"
This script can be run with a simple bash command.
user@local:arctichoke$ bash start_container.sh
Starting podman machine...
Starting machine "podman-machine-default"
This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:
podman machine set --rootful
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
───────────────────────────────────────────────────── esgpull installation ──────────────────────────────────────────────────────
Using existing install at /arctichoke_data/bergybits
Install config added to /root/.config/esgpull/installs.json
Built arctichoke @ file:///workspace
Uninstalled 1 package in 13ms
Installed 1 package in 62ms
[I 2026-06-04 18:23:25.915 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2026-06-04 18:23:25.925 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2026-06-04 18:23:25.934 ServerApp] jupyterlab | extension was successfully linked.
[I 2026-06-04 18:23:25.946 ServerApp] notebook | extension was successfully linked.
[I 2026-06-04 18:23:25.948 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2026-06-04 18:23:27.822 ServerApp] notebook_shim | extension was successfully linked.
[I 2026-06-04 18:23:27.823 ServerApp] panel.io.jupyter_server_extension | extension was successfully linked.
[W 2026-06-04 18:23:27.959 ServerApp] All authentication is disabled. Anyone who can connect to this server will be able to run code.
[I 2026-06-04 18:23:27.960 ServerApp] notebook_shim | extension was successfully loaded.
[I 2026-06-04 18:23:27.966 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2026-06-04 18:23:27.968 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2026-06-04 18:23:28.012 LabApp] JupyterLab extension loaded from /workspace/.cvenv/lib/python3.13/site-packages/jupyterlab
[I 2026-06-04 18:23:28.012 LabApp] JupyterLab application directory is /workspace/.cvenv/share/jupyter/lab
[I 2026-06-04 18:23:28.017 LabApp] Extension Manager is 'pypi'.
[I 2026-06-04 18:23:28.366 ServerApp] jupyterlab | extension was successfully loaded.
[I 2026-06-04 18:23:28.403 ServerApp] notebook | extension was successfully loaded.
[I 2026-06-04 18:23:28.404 ServerApp] panel.io.jupyter_server_extension | extension was successfully loaded.
[I 2026-06-04 18:23:28.406 ServerApp] Serving notebooks from local directory: /workspace
[I 2026-06-04 18:23:28.406 ServerApp] Jupyter Server 2.17.0 is running at:
[I 2026-06-04 18:23:28.406 ServerApp] http://e343a4f95781:8888/lab
[I 2026-06-04 18:23:28.406 ServerApp] http://127.0.0.1:8888/lab
[I 2026-06-04 18:23:28.406 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2026-06-04 18:23:28.620 ServerApp] Skipped non-installed server(s): basedpyright, bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyrefly, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
...
Leave this terminal running to have access to the Jupyter server and see relevant updates. The output that continues to appear in this terminal as the Jupyter server is being used can be very helpful in debugging Jupyter-related issues. The instructions above on how to start the container are also shown on the Starting the Container page.
I’ll explain each section of the start_container.sh script in detail below.
Set the script to fail on common errors
The first line is set -euo pipefail.
The GitHub Gist by akrasic called bash_strict_mode explains that the set command is used to cause a script to fail very explicitly when encountering errors.
This can be helpful to track down where exactly the bugs are, especially if the script calls other scripts.
Ensure the virtual machine is running
The next block checks the status of the virtual machine, which is required for running on macOS. First, it checks whether the machine has been initialized.
...
# ---- Ensure podman machine is running (macOS) ----
if ! podman machine inspect >/dev/null 2>&1; then
echo "No podman machine found. Initializing..."
podman machine init
fi
...
Then, it gets the state of the machine to see whether it has already been started.
...
MACHINE_STATE=$(podman machine inspect --format '{{.State}}')
if [[ "$MACHINE_STATE" != "running" ]]; then
echo "Starting podman machine..."
podman machine start
fi
...
This block allows you to re-run the start_container.sh script even if you already have the Podman machine running.
This is useful when testing the code in a way which requires restarting the container frequently.
Set the parameters
Next, the scripts sets the following parameters.
IMAGE="arctichoke_7"This will be the name of the image that is built.
If an image with this name already exists, a new one will not be built.
If you want to try a new build, change this parameter.
CONTAINER_NAME="sicp_cont"This is the name that the container will have when it is running.
If you want to have multiple containers running at the same time, you might need to change this.
WORKDIR="/workspace"This defines the name of the working directory for the container.
This must be the same name used in the
Containerfile.
Check existing containers and images
The next two blocks clean up any existing images and builds the image, if necessary.
...
# ---- Cleanup old container if it exists ----
podman rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
# ---- Ensure image exists ----
if ! podman image exists "$IMAGE"; then
echo "Image $IMAGE not found. Building from '.devcontainer/Containerfile'..."
podman build -f .devcontainer/Containerfile -t "$IMAGE" . | tee .devcontainer/build_container_log.txt
fi
...
The || true in the first command ensures that it executes successfully, even when there is no old container to clean up.
In the second part, the image building step will be skipped if an image with the given name already exists.
Set up access to external volumes
The next block sets up access to data on an external volume.
The value of SICP_DATA_DIR should be changed to match the absolute file path of your external hard drive where esgpull will store the HighResMIP data.
...
# ---- Setup external hard drive access ----
export SICP_DATA_DIR=/Volumes/BERGY_BITS/arctichoke_data/
SICP_DATA_DIR="${SICP_DATA_DIR:-}"
if [[ -n "$SICP_DATA_DIR" ]]; then
if [[ ! -d "$SICP_DATA_DIR" ]]; then
echo "ERROR: SICP_DATA_DIR does not exist: $SICP_DATA_DIR"
exit 1
fi
fi
...
If you need to change the external volume setup, you will need to restart the Podman machine, not just the container, to see whether the change worked.
user@local:arctichoke$ podman machine stop
Machine "podman-machine-default" stopped successfully
user@local:arctichoke$ podman machine rm
The following files will be deleted:
/Users/<user>/.config/containers/podman/machine/applehv/podman-machine-default.json
/var/folders/30/czkm3xpn6fx2v22bz_r2g37r0000gp/T/podman/podman-machine-default.sock
/var/folders/30/czkm3xpn6fx2v22bz_r2g37r0000gp/T/podman/podman-machine-default-gvproxy.sock
/var/folders/30/czkm3xpn6fx2v22bz_r2g37r0000gp/T/podman/podman-machine-default-api.sock
/var/folders/30/czkm3xpn6fx2v22bz_r2g37r0000gp/T/podman/podman-machine-default.log
Are you sure you want to continue? [y/N] y
Create the list of volumes
The next block creates the list of arguments to be used to define the volumes. This should include the working directory and the external volume.
...
# ---- Build volume args ----
VOLUMES=(-v "$(pwd)":"$WORKDIR")
if [[ -n "$SICP_DATA_DIR" ]]; then
VOLUMES+=(-v "$SICP_DATA_DIR:/arctichoke_data")
fi
...
Run the container
Finally, the script starts the container with the podman run command.
...
# ---- Run container (Jupyter starts automatically from CMD) ----
podman run -it --rm \
--name "$CONTAINER_NAME" \
-p 8889:8888 \
"${VOLUMES[@]}" \
-w "$WORKDIR" \
"$IMAGE"
The arguments of the command are, similar to Building a simple container (see the Podman run docs for details):
-i: Interactive“When set to true, make
stdinavailable to the contained process. If false, thestdinof the contained process is empty and immediately closed.”
-t: TTY“Allocate a pseudo-TTY. The default is false. When set to true, Podman allocates a pseudo-tty and attach to the standard input of the container. This can be used, for example, to run a throwaway interactive shell.”
--rm: Remove upon exit“Automatically remove the container and any anonymous unnamed volume associated with the container when it exits. The default is false.”
--name: Container name“Assign a name to the container.” This can be completely different from the name of the image it is built from.
-p: Publish (Ports)“Publish a container’s port, or range of ports, to the host. Both hostPort and containerPort can be specified as a range of ports. When specifying ranges for both, the number of container ports in the range must match the number of host ports in the range.”
This is set here as
8889:8888to specify that the host system (my laptop) will use port 8889 (to avoid conflicts with existing Jupyter servers) to connect to the container’s port 8888 where it’s Jupyter server is running.
"${VOLUMES[@]}": List of volumesThe list of directories (
/workspaceand the external volume) that the container should have access to.
-w: Working directory“Working directory inside the container. The default working directory for running binaries within a container is the root directory (
/). The image developer can set a different default with theWORKDIRinstruction. The operator can override the working directory by using the-woption.”
"$IMAGE"The name of the image to use for this container.
Testing the container
With the container running, I can test to make sure things were installed correctly.
First, I’ll check the versions of various packages, starting with uv.
user@local:arctichoke$ podman exec -it <container_id> /bin/sh
# bash
root@<container_id>:/workspace# uv --version
uv 0.11.8 (x86_64-unknown-linux-musl)
Next, I’ll check cdo.
root@<container_id>:/workspace# cdo --version
Climate Data Operators version 2.5.1 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -std=gnu++20 -g -O2 -ffile-prefix-map=/build/reproducible-path/cdo-2.5.1=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fopenmp -pthread
CXX version : g++ (Debian 14.2.0-19) 14.2.0
CXX library :
C Compiler: gcc -g -O2 -Werror=implicit-function-declaration -ffile-prefix-map=/build/reproducible-path/cdo-2.5.1=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wall -pedantic -fPIC -fopenmp -pthread -pthread
C version : gcc (Debian 14.2.0-19) 14.2.0
F77 Compiler: f77 -g -O2 -ffile-prefix-map=/build/reproducible-path/cdo-2.5.1=. -fstack-protector-strong -fstack-clash-protection -fcf-protection
F77 version : GNU Fortran (Debian 14.2.0-19) 14.2.0
Features: 1GB 4threads c++20 OpenMP45 Fortran pthreads HDF5 NC4/HDF5/threadsafe OPeNDAP sz udunits2 proj xml2 magics curl fftw3 hirlam_extensions sse2
Libraries: yac/3.4.0 NetCDF/4.9.3 HDF5/1.14.5 proj/9.6.0 xml2/2.9.14 curl/8.14.1(h8.13.0-rc3) magics/4.15.5
CDI data types: SizeType=size_t
CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 nczarr
CDI library version : 2.5.1
ecCodes library version : 2.41.0
NetCDF library version : 4.9.3 of Feb 14 2025 11:24:29 $
exse library version : 2.0.0
FILE library version : 1.9.1
To check nco, I’ll check the version of ncks (NetCDF Kitchen Sink).
root@<container_id>:/workspace# ncks --version
NCO netCDF Operators version 5.3.3 "Sea Shanty" built by sbuild on sbuild at Mar 29 2025 05:28:49
ncks version 5.3.3
Then, I can check Python.
root@<container_id>:/workspace# python
Python 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
Cleaning up old images
During the testing above, several images were generated. In the Pod Manager sidebar, under the “Overview” dropdown, a summary of the images can be seen. Usually, you will need to hit the refresh button next to this dropdown to see any information. For the images, containers, and local volumes, it lists the total number, the number active, the disk space used, and the reclaimable disk space.
When testing out new builds of the Containerfile, it is easy to generate many images.
Podman can list the existing images.
user@local:arctichoke$ podman images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 55b854bc8dcd 2 days ago 199 MB
localhost/test_trixie latest 7bab3cfc2aa3 2 days ago 199 MB
<none> <none> b907eed1ae8b 2 days ago 199 MB
<none> <none> 6dd8a6c94600 2 days ago 142 MB
<none> <none> 4f123f8d89fe 2 days ago 142 MB
<none> <none> 5808b4b9aad3 2 days ago 81.1 MB
ghcr.io/astral-sh/uv latest b960411dc937 6 days ago 58.2 MB
docker.io/library/nginx latest 7aaca76c508f 12 days ago 165 MB
docker.io/library/debian trixie-slim f283d70f8784 2 weeks ago 81.1 MB
localhost/arctichoke_7 latest b05b6acdb72b 3 weeks ago 2.78 GB
While the image for this project is a reasonable 2.78 GB, it adds up quickly when there are dozens of copies.
Podman provides an easy prune command to clean this up.
When this command is run, it will remove all images that do not have an associated container that is currently running.
This makes it convenient as I can just start a container from the image that I want to keep then, from outside that container, run a prune command.
user@local:arctichoke$ podman image prune -a
WARNING! This command removes all images without at least one container associated with them.
Are you sure you want to continue? [y/N] y
7aaca76c508f7d121ff29cbe9dd071012486d00c21e17655eb1a1dfb711e9330
b960411dc937f9b4d9762349f5f77772d36dead003baa3bc01330abe8e1f38a6
f283d70f878433b889e4b9252110fad858e0e0887df5bac91cd2ad4ccb2b3a2a
5808b4b9aad30e55a565efe96b48bb0628439d44dc31883d7be8b24998e52bfd
4f123f8d89feb6111a63528bf05c56cb2831684cda3ef4b7e2cad3c87d567a58
6dd8a6c94600c177a0a4a6f0166574c35d720766c981bbd10f34d77be956f0bc
b907eed1ae8b4ec83fe12df4292630c67ca77d99f5e966dfa0e99e8077d2bae7
55b854bc8dcd53e9a5d01887214276422374802dd565dd165b3b78af8af79f18
7bab3cfc2aa34bf43d9a63ccb8428a859c9f64a807186a4262b17bc5ffe0b4eb
I can then confirm that only one image remains.
user@local:arctichoke$ podman images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/arctichoke_7 latest b05b6acdb72b 3 weeks ago 2.78 GB
Virtual environment and packages
Activating the virtual environment
When initializing the project, uv automatically creates a virtual environment in .venv/.
That environment can be used outside the container.
In the section Defining environment variables, I noted that I changed the default virtual environment location for uv to be .cvenv inside the container.
I can easily activate it by first starting a terminal inside the container.
podman exec -it <container_id> /bin/sh
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
user@local:arctichoke$ podman exec -it <container_id> /bin/sh
#
Then, I activate bash and source the virtual environment directory.
# bash
root@<container_id>:/workspace# source .cvenv/bin/activate
(arctichoke) root@<container_id>:/workspace#
Note that the virtual environment’s name (arctichoke) is now at the beginning of the command prompt.
Adding package dependencies
Using uv to add dependencies works similarly to poetry as described in Py-Pkgs 3.6. Adding dependencies to your package.
When a uv add <package> command is run, that package is automatically added to the pyproject.toml file.
See uv docs for The project environment for more information.
Packages for datasets
I use xarray as the main workhorse to handle datasets.
(arctichoke) root@<container_id>:/workspace# uv add xarray
Resolved 48 packages in 343ms
Prepared 1 package in 318ms
Installed 1 package in 11ms
+ xarray==2026.4.0
In order to load data from NetCDF files into an xarray dataset, I also need to add the netcdf4 package.
(arctichoke) root@<container_id>:/workspace# uv add netcdf4
Resolved 78 packages in 716ms
Prepared 2 packages in 3.74s
Installed 2 packages in 13ms
+ cftime==1.6.5
+ netcdf4==1.7.4
I also specifically added dask so that I can take advantage of lazy loading with xarray.open_dataset().
This allows me to filter a large dataset before actually loading the entire file into memory.
(arctichoke) root@<container_id>:/workspace# uv add dask
Resolved 198 packages in 708ms
Built arctichoke @ file:///workspace
Prepared 1 package in 18ms
Uninstalled 1 package in 13ms
Installed 7 packages in 2.48s
+ cloudpickle==3.1.2
+ dask==2026.3.0
+ fsspec==2026.4.0
+ locket==1.0.0
+ partd==1.4.2
~ arctichoke==0.1.0 (from file:///workspace)
+ toolz==1.1.0
Packages for making plots
For plots, I added the hvplot package to be able to make html maps of irregular gridded data without interpolating onto a regular grid first.
(arctichoke) root@<container_id>:/workspace# uv add hvplot
Resolved 170 packages in 2.70s
Built arctichoke @ file:///workspace
Prepared 16 packages in 6.01s
Uninstalled 1 package in 14ms
Installed 16 packages in 18.01s
+ bokeh==3.9.0
+ colorcet==3.2.1
+ contourpy==1.3.3
+ holoviews==1.22.1
+ hvplot==0.12.2
+ linkify-it-py==2.1.0
+ markdown==3.10.2
+ narwhals==2.20.0
+ panel==1.8.10
+ param==2.3.3
+ pillow==12.2.0
+ pyviz-comms==3.0.6
~ arctichoke==0.1.0 (from file:///workspace)
+ tqdm==4.67.3
+ uc-micro-py==2.0.0
+ xyzservices==2026.3.0
Then, I added cartopy to have access to map projections through the submodule cartopy.crs.
(arctichoke) root@<container_id>:/workspace# uv add cartopy
Resolved 178 packages in 1.40s
Built arctichoke @ file:///workspace
Prepared 9 packages in 6.54s
Uninstalled 1 package in 10ms
Installed 9 packages in 5.07s
+ cartopy==0.25.0
+ cycler==0.12.1
+ fonttools==4.62.1
+ kiwisolver==1.5.0
+ matplotlib==3.10.9
+ pyproj==3.7.2
+ pyshp==3.0.3
~ arctichoke==0.1.0 (from file:///workspace)
+ shapely==2.1.2
I also added the geoviews package for handling physical features on maps.
(arctichoke) root@<container_id>:/workspace# uv add geoviews
Resolved 179 packages in 643ms
Built arctichoke @ file:///workspace
Prepared 2 packages in 503ms
Uninstalled 1 package in 13ms
Installed 2 packages in 859ms
+ geoviews==1.15.1
~ arctichoke==0.1.0 (from file:///workspace)
When tyring to plot data from the HadGEM3-GC models over time, I got the following error.
ImportError: Plotting of arrays of cftime.datetime objects or arrays indexed by cftime.datetime objects requires the optional `nc-time-axis` (v1.2.0 or later) package.
It turns out that these models use the type cftime.Datetime360Day instead of numpy.datetime64, which causes issues for matplotlib when using time values on one of the axes.
In the GitHub issue How to convert cftime.Datetime360Day() object to python datetime?, one of the maintainers of xarray mentioned that:
“We fixed this very recently in xarray by adding an optional dependency on nc-time-axis, a package that enables plotting cftime dates in matplotlib. The changes will take effect in the next version, which has yet to be released (version 0.12.0).”
Adding nc-time-axis indeed fixed the issue.
(arctichoke) root@<container_id>:/workspace# uv add nc-time-axis
Resolved 200 packages in 697ms
Built arctichoke @ file:///workspace
Prepared 2 packages in 174ms
Uninstalled 1 package in 20ms
Installed 2 packages in 139ms
+ nc-time-axis==1.4.1
~ arctichoke==0.1.0 (from file:///workspace)
I had also tried to use xarray.convert_calendar() instead of installing nc-time-axis, but with no luck.
They have notes in the documentation about how to deal with “360_day” calendars.
However, I still found that some dates being dropped or set to the missing value, even though all the dates are the 16th of each month.
I’m sticking with just using nc-time-axis as my solution.
Packages for saving plots
The html plots that are made with hvplot cannot be directly saved to a png file.
As a workaround, I added the packages chromium and chromium-driver to the Containerfile in order to open an html plot in a browser within the container, take a “screenshot”, and save that as a png image.
In order for that process to work, I added the selenium and bokeh packages.
(arctichoke) root@<container_id>:/workspace# uv add selenium
Resolved 187 packages in 910ms
Built arctichoke @ file:///workspace
Prepared 9 packages in 1.64s
Uninstalled 1 package in 31ms
Installed 9 packages in 2.54s
+ outcome==1.3.0.post0
+ pysocks==1.7.1
~ arctichoke==0.1.0 (from file:///workspace)
+ selenium==4.43.0
+ sniffio==1.3.1
+ sortedcontainers==2.4.0
+ trio==0.33.0
+ trio-websocket==0.12.2
+ wsproto==1.3.2
(arctichoke) root@<container_id>:/workspace# uv add bokeh
Resolved 187 packages in 323ms
Built arctichoke @ file:///workspace
Prepared 1 package in 18ms
Uninstalled 1 package in 14ms
Installed 1 package in 45ms
~ arctichoke==0.1.0 (from file:///workspace)
I also added the pillow package for additional png manipulation tools.
(arctichoke) root@<container_id>:/workspace# uv add pillow
Resolved 187 packages in 331ms
Built arctichoke @ file:///workspace
Prepared 1 package in 20ms
Uninstalled 1 package in 14ms
Installed 1 package in 60ms
~ arctichoke==0.1.0 (from file:///workspace)
Packages for Jupyter notebooks
In order to use Jupyter notebooks with the .cvenv virtual environment, I added ipykernel and jupyter.
(arctichoke) root@<container_id>:/workspace# uv add ipykernel jupyter
Resolved 134 packages in 1.15s
Built arctichoke @ file:///workspace
Prepared 51 packages in 3.38s
Uninstalled 1 package in 19ms
Installed 51 packages in 14.63s
+ anyio==4.13.0
+ argon2-cffi==25.1.0
+ argon2-cffi-bindings==25.1.0
+ arrow==1.4.0
+ async-lru==2.3.0
+ beautifulsoup4==4.14.3
+ bleach==6.3.0
+ cffi==2.0.0
+ defusedxml==0.7.1
+ fqdn==1.5.1
+ h11==0.16.0
+ httpcore==1.0.9
+ httpx==0.28.1
+ ipywidgets==8.1.8
+ isoduration==20.11.0
+ json5==0.14.0
+ jsonpointer==3.1.1
+ jupyter==1.1.1
+ jupyter-console==6.6.3
+ jupyter-events==0.12.1
+ jupyter-lsp==2.3.1
+ jupyter-server==2.17.0
+ jupyter-server-terminals==0.5.4
+ jupyterlab==4.5.6
+ jupyterlab-pygments==0.3.0
+ jupyterlab-server==2.28.0
+ jupyterlab-widgets==3.0.16
+ lark==1.3.1
+ mistune==3.2.0
+ nbconvert==7.17.1
+ notebook==7.5.5
+ notebook-shim==0.2.4
+ pandocfilters==1.5.1
+ prometheus-client==0.25.0
+ pycparser==3.0
+ python-json-logger==4.1.0
+ rfc3339-validator==0.1.4
+ rfc3986-validator==0.1.1
+ rfc3987-syntax==1.1.0
~ arctichoke==0.1.0 (from file:///workspace)
+ send2trash==2.1.0
+ setuptools==82.0.1
+ soupsieve==2.8.3
+ terminado==0.18.1
+ tinycss2==1.4.0
+ tzdata==2026.2
+ uri-template==1.3.0
+ webcolors==25.10.0
+ webencodings==0.5.1
+ websocket-client==1.9.0
+ widgetsnbextension==4.0.15
Packages for external tools
I added the Python package for cdo to be able to call it’s functions from Python scripts.
Note that this requires that the cdo CLI is installed, which is done in the Containerfile.
(arctichoke) root@<container_id>:/workspace# uv add cdo
Resolved 199 packages in 2.61s
Built arctichoke @ file:///workspace
Prepared 2 packages in 177ms
Uninstalled 1 package in 17ms
Installed 2 packages in 68ms
+ cdo==1.6.1
~ arctichoke==0.1.0 (from file:///workspace)
(arctichoke) root@<container_id>:/workspace# python
Python 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cdo
>>> print(cdo.__version__)
1.6.0
Next, I added the esgpull package to be able to download HighResMIP data.
On the esgpull documentation Installation page they show an example using uv tool install which creates an isolated installation.
However, I use uv add so that esgpull becomes a persistent dependency of the project.
I specify the source with git+https://github.com/ESGF/esgf-download in order to get the latest release of the package to resolve an issue I was encountering.
root@<container_id>:/workspace# uv add git+https://github.com/ESGF/esgf-download
Resolved 198 packages in 1.07s
Built arctichoke @ file:///workspace
Updated https://github.com/ESGF/esgf-download (726ef1166114eadd085c24c6e1542ec0be052e03)
Built esgpull @ git+https://github.com/ESGF/esgf-download@726ef1166114eadd085c24c6e1542ec0be052e03
Prepared 21 packages in 3.08s
Uninstalled 1 package in 13ms
Installed 21 packages in 2.54s
+ aiofiles==25.1.0
+ aiostream==0.7.1
+ alembic==1.18.4
+ annotated-types==0.7.0
+ cattrs==26.1.0
+ click-params==0.5.0
+ cryptography==48.0.0
+ deprecated==1.3.1
+ esgpull==0.9.6 (from git+https://github.com/ESGF/esgf-download@726ef1166114eadd085c24c6e1542ec0be052e03)
+ mako==1.3.12
+ pydantic==2.13.4
+ pydantic-core==2.46.4
+ pydantic-settings==2.14.0
+ pyopenssl==26.2.0
+ python-dotenv==1.2.2
+ rich==15.0.0
~ arctichoke==0.1.0 (from file:///workspace)
+ tomlkit==0.14.0
+ typing-inspection==0.4.2
+ validators==0.22.0
+ wrapt==2.1.2
The esgpull maintainer svenrdz replied to the issue that I posted on the esgpull GitHub.
He found a fix, which was to put in a short sleep timer before trying to make a new httpx.AsyncClient.
This fix can be seen in .cvenv/lib/python3.13/site-packages/esgpull/context.py:
...
async def _fetch_one(self, result: RT) -> RT:
host = result.request.url.host
semaphore = self.get_or_create_semaphore(host)
# The bridge API tends to produce non-standard errors when too many
# new connections open in a short time span. With no sleep, the 4th
# connection is always where it breaks. 50ms sleep seems to fix that.
if self.client is None:
await asyncio.sleep(0.005)
...
This should be included in future updates to esgpull which should make it unnecessary to specify the exact commit when adding it with uv as shown above.
Using esgpull requires additional steps as noted on the Installation page.
I detail how I managed that set up in the Initializing esgpull section later in this guide.
Packages for testing
In order to run tests, following Py-Pkgs Section 3.7.2. Running tests and Py-Pkgs Section 3.7.3. Code coverage, I installed pytest and pytest-cov as development dependencies by specifying the --dev group.
This means that, if someone where to install arctichoke as a package for their own purposes, the packages in the --dev group would not be installed by default.
root@<container_id>:/workspace# uv add --dev pytest
Resolved 190 packages in 2.15s
Built arctichoke @ file:///workspace
Prepared 4 packages in 210ms
Uninstalled 1 package in 14ms
Installed 4 packages in 499ms
+ iniconfig==2.3.0
+ pluggy==1.6.0
+ pytest==9.0.3
~ arctichoke==0.1.0 (from file:///workspace)
root@<container_id>:/workspace# uv add --dev pytest-cov
Resolved 192 packages in 512ms
Built arctichoke @ file:///workspace
Prepared 3 packages in 181ms
Uninstalled 1 package in 16ms
Installed 3 packages in 337ms
+ coverage==7.13.5
+ pytest-cov==7.1.0
~ arctichoke==0.1.0 (from file:///workspace)
Packages for documentation
Following Py-Pkgs 3.8.4. Building documentation, I added the packages necessary to build the documentation you are reading to the --dev group.
(arctichoke) root@<container_id>:/workspace# uv add --dev myst-nb sphinx-autoapi sphinx-rtd-theme
Resolved 112 packages in 1.08s
Built arctichoke @ file:///workspace
Prepared 31 packages in 3.20s
Uninstalled 1 package in 8ms
Installed 35 packages in 317ms
+ alabaster==1.0.0
+ astroid==4.1.2
+ attrs==26.1.0
+ babel==2.18.0
+ click==8.3.2
+ docutils==0.22.4
+ fastjsonschema==2.21.2
+ greenlet==3.4.0
+ imagesize==2.0.0
+ importlib-metadata==9.0.0
+ jsonschema==4.26.0
+ jsonschema-specifications==2025.9.1
+ jupyter-cache==1.0.1
+ myst-nb==1.4.0
+ myst-parser==5.0.0
+ nbclient==0.10.4
+ nbformat==5.10.4
+ referencing==0.37.0
+ roman-numerals==4.1.0
+ rpds-py==0.30.0
~ arctichoke==0.1.0 (from file:///workspace)
+ snowballstemmer==3.0.1
+ sphinx==9.1.0
+ sphinx-autoapi==3.8.0
+ sphinx-rtd-theme==3.1.0
+ sphinxcontrib-applehelp==2.0.0
+ sphinxcontrib-devhelp==2.0.0
+ sphinxcontrib-htmlhelp==2.1.0
+ sphinxcontrib-jquery==4.1
+ sphinxcontrib-jsmath==1.0.1
+ sphinxcontrib-qthelp==2.0.0
+ sphinxcontrib-serializinghtml==2.0.0
+ sqlalchemy==2.0.49
+ tabulate==0.10.0
+ zipp==3.23.1
The pyproject.toml file
After adding all the above packages as dependencies, the pyproject.toml file now appears as below.
[project]
name = "arctichoke"
version = "0.1.0"
description = "Investigate sea ice choke points in the Canadian Arctic Archipelago using high-resolution models."
authors = [
{ name = "Mikhail Schee", email = "mikhail.schee@alumni.utoronto.ca" }
]
license = "MIT"
readme = "README.md"
requires-python = ">=3.13,<3.14"
dependencies = [
"bokeh>=3.9.0",
"cartopy>=0.25.0",
"cdo>=1.6.1",
"dask>=2026.3.0",
"esgpull",
"geoviews>=1.15.1",
"hvplot>=0.12.2",
"ipykernel>=7.2.0",
"jupyter>=1.1.1",
"nc-time-axis>=1.4.1",
"netcdf4>=1.7.4",
"pillow>=12.2.0",
"selenium>=4.43.0",
"xarray>=2026.4.0",
]
[dependency-groups]
dev = [
"myst-nb>=1.4.0",
"pytest>=9.0.3",
"pytest-cov>=7.1.0",
"sphinx-autoapi>=3.8.0",
"sphinx-rtd-theme>=3.1.0",
]
[project.scripts]
arctichoke = "arctichoke:main"
[build-system]
requires = ["uv_build>=0.9.15,<0.12.0"]
build-backend = "uv_build"
[tool.uv.sources]
esgpull = { git = "https://github.com/ESGF/esgf-download" }
When new packages are added to the project, they will be cached the next time the container is run.
Building the package
As shown in Astral’s Build systems documentation for uv, I had used the --package flag when initializing the repository with uv.
In Astral’s Building your package section, I used the
user@local:arctichoke$ podman exec -it <container_id> /bin/sh
# bash
root@<container_id>:/workspace# source .cvenv/bin/activate
(arctichoke) root@<container_id>:/workspace# uv sync
Resolved 200 packages in 550ms
Built arctichoke @ file:///workspace
Prepared 1 package in 53ms
Installed 1 package in 12ms
+ arctichoke==0.1.0 (from file:///workspace)
(arctichoke) root@<container_id>:/workspace# uv lock
Resolved 200 packages in 21ms
(arctichoke) root@<container_id>:/workspace# uv build
Building source distribution (uv build backend)...
Building wheel from source distribution (uv build backend)...
Successfully built dist/arctichoke-0.1.0.tar.gz
Successfully built dist/arctichoke-0.1.0-py3-none-any.whl
(arctichoke) root@<container_id>:/workspace# uv version
arctichoke 0.1.0
I can now import the package in the Python interpreter.
(arctichoke) root@<container_id>:/workspace# python
Python 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import arctichoke
>>> print(arctichoke.__version__)
0.1.0
Using a Jupyter notebook in the container
Instructions on how to test whether you can access the Jupyter server inside the container are shown in the Jupyter Notebook Test guide.
Once that is working, I can test to see whether the Jupyter server has access to the arctichoke package by executing a cell with the following code.
import arctichoke
print(arctichoke.__version__)
0.1.0
Initializing esgpull
Data from the CMIP6 HighResMIP models can be downloaded from the ESGF Federated Nodes webportal, which has great search functionality but tedious manual downloading, or through Globus, which augments the webportal through an application you can install on your system.
However, I decided to use esgpull for this project, which offers a command line interface for searching and downloading.
For detailed reasoning on choosing esgpull, see the Downloading model data with esgpull guide.
Creating an esgpull install
In the section Packages for external tools, I added esgpull as a dependency for the Python environment.
As noted in the Installation page under “Setup,” an additonal step must be taken before esgpull can be used, namely running the esgpull self install command.
“The reason is that esgpull is prevented from writing anything on disk until installed.
Installing esgpull equates choosing a directory in which it is allowed to write anything it needs to run properly. It also creates all the required files/directories in that directory and fetches some metadata from ESGF that is required to run properly.”
In the esgpull_entrypoint.sh script, (see Setting up esgpull install), I run uv run esgpull self install bergybits from the directory where I want the data to be (in this case, /seaice_data which is connected to my external drive) to create an install called bergybits.
The first time this is run, the output will indicate that a new install has been created.
(arctichoke) root@<container_id>:/workspace# cd /arctichoke_data/
(arctichoke) root@<container_id>:/arctichoke_data# uv run esgpull self install bergybits
──────────────────────────────────── esgpull installation ────────────────────────────────────
Creating install directory and files at /arctichoke_data/bergybits
Install config added to /root/.config/esgpull/installs.json
On subsequent times running the start_container.sh script, esgpull will automatically recognize and use the existing install that has been set up with whatever data has already been downloaded.
...
───────────────────────────────── esgpull installation ─────────────────────────────────
Using existing install at /arctichoke_data/bergybits
Install config added to /root/.config/esgpull/installs.json
If no name for the install is given, (i.e., running just uv run esgpull self install), then you will be taken through an interactive setup process shown below where you can specify the name and location of the install.
(arctichoke) root@<container_id>:/workspace# uv run esgpull self install
──────────────────────────────────────────────────── esgpull installation ────────────────────────────────────────────────────
Install location (/workspace/.esgpull): .esgpull
Name (optional): new_install
Creating install directory and files at /workspace/.esgpull
Install config added to /root/.config/esgpull/installs.json
I can view all the installs I have setup where the one marked with * is the one that is currently selected.
(arctichoke) root@<container_id>:/workspace# uv run esgpull self choose
Install locations
/arctichoke_data/bergybits
* /workspace/.esgpull
I can choose a different install by specifying the name at the end of that command.
(arctichoke) root@<container_id>:/workspace# uv run esgpull self choose /arctichoke_data/bergybits/
(arctichoke) root@<container_id>:/workspace# uv run esgpull self choose
Install locations
* /arctichoke_data/bergybits
/workspace/.esgpull
To delete an install, select it first, then use the esgpull self delete command.
To remove the associated data, run the rm command suggested by the output.
(arctichoke) root@<container_id>:/workspace# uv run esgpull self choose /workspace/.esgpull/
(arctichoke) root@<container_id>:/workspace# uv run esgpull self choose
Install locations
/arctichoke_data/bergybits
* /workspace/.esgpull
(arctichoke) root@<container_id>:/workspace# uv run esgpull self delete
You are going to delete: /workspace/.esgpull
Please enter '.esgpull' to continue: .esgpull
Deleting /workspace/.esgpull from config...
To remove all files from this install, run:
$ rm -rf /workspace/.esgpull
(arctichoke) root@<container_id>:/workspace# rm -rf /workspace/.esgpull/
(arctichoke) root@<container_id>:/workspace#
Configuring the esgpull install
The first time I tried to search with esgpull, I got the following error.
(arctichoke) root@<container_id>:/workspace# uv run esgpull search
(
'fetch',
[
HTTPStatusError("Server error '500 500' for url
'https://esgf-node.ipsl.upmc.fr/esg-search/search?type=File&offset=0&limit=0&format=applicatio
n%2Fsolr%2Bjson&fields=instance_id&distrib=true&latest=true&replica=true&retracted=false'\nFor
more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500")
]
)
ERROR 'esgf-node.ipsl.upmc.fr' is not responding
Aborted!
This clearly indicates that the French node is not responding currently.
However, in an older version of esgpull, the error was more cryptic.
Expand for an older version of the error message
(arctichoke) root@<container_id>:/workspace# uv run esgpull search
[2026-04-14 18:33:45] ERROR root
+ Exception Group Traceback (most recent call last):
...
| raise HTTPStatusError(message, request=request, response=self)
| httpx.HTTPStatusError: Server error '500 500' for url 'https://esgf-node.ipsl.upmc.fr/esg-search/search?type=File&offset=0&limit=0&format=application%2Fsolr%2Bjson&fields=instance_id&distrib=true&latest=true&retracted=false'
| For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
+------------------------------------
ExceptionGroup: fetch (1 sub-exception)
See /workspace/bergybits/log/esgpull-search-2026-04-14_22-33-44.log for error log.
Aborted!
The line with Server error '500 500' for url 'https://esgf-node.ipsl.upmc.fr/... led me to be confused as to why it would be checking a French web domain.
I checked the configuration for this esgpull install.
(arctichoke) root@<container_id>:/workspace# uv run esgpull config
──────────────────────────── /arctichoke_data/bergybits/config.toml ────────────────────────────
[paths]
data = "/arctichoke_data/bergybits/data"
db = "/arctichoke_data/bergybits/db"
log = "/arctichoke_data/bergybits/log"
tmp = "/arctichoke_data/bergybits/tmp"
plugins = "/arctichoke_data/bergybits/plugins"
[credentials]
filename = "credentials.toml"
[cli]
page_size = 20
[db]
filename = "esgpull.db"
[download]
chunk_size = 67108864
http_timeout = 20
max_concurrent = 5
disable_ssl = false
disable_checksum = false
show_filename = false
[api]
index_node = "esgf-node.ipsl.upmc.fr"
http_timeout = 20
max_concurrent = 5
page_limit = 50
default_query_id = ""
use_custom_distribution_algorithm = false
[api.default_options]
distrib = "true"
latest = "true"
replica = "none"
retracted = "false"
[plugins]
enabled = false
This confirms install I created has index_node = "esgf-node.ipsl.upmc.fr".
I can check the status of the available nodes.
This information should also be available at the ESGF Nodes Status Summary page, however that site is currently not working.
(arctichoke) root@<container_id>:/workspace# uv run esgpull index-nodes
node │ status
════════════════════════════════════╪═════════════
esgf-node.ornl.gov/esgf-1-5-bridge │ OK
us-index │ no response
esgf.ceda.ac.uk │ OK
esgf.nci.org.au │ no response
esgf-node.ipsl.upmc.fr │ no response
esgf-data.dkrz.de │ OK
And here’s the problem. The French node is indeed not responding. I changed the index node to be the one recommended for the USA region.
(workspace) root@<container_id>:/arctichoke_data# uv run esgpull config api.index_node esgf-node.ornl.gov/esgf-1-5-bridge
[api]
index_node = "esgf-node.ornl.gov/esgf-1-5-bridge"
👍 New config file created at /arctichoke_data/bergybits/config.toml.
Then, I was able to perform a search:
(workspace) root@<container_id>:/workspace# uv run esgpull search project:CMIP6 activity_id:HighResMIP
Found 66629 datasets.
id │ dataset │ # │ size │ data_node
════╪═════════════════════════════════════════════════════════════════════════╪════╪═════════════╪════════════════════
0 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.control-1950.r1i1p1f1.CFmon.clt… │ 10 │ 2.9 GiB │ esgf-data.ucar.edu
1 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.control-1950.r1i1p1f1.Lmon.mrro… │ 10 │ 1,002.9 MiB │ esgf-data.ucar.edu
2 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.control-1950.r1i1p1f1.Lmon.tsl.… │ 10 │ 9.9 GiB │ esgf-node.ornl.gov
3 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.control-1950.r1i1p1f1.Omon.soga… │ 99 │ 2.8 MiB │ esgf-node.ornl.gov
4 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Amon.ta… │ 1 │ 719.4 MiB │ esgf-data.ucar.edu
5 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Amon.ta… │ 1 │ 738.9 MiB │ esgf-node.ornl.gov
6 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Amon.ta… │ 1 │ 752.3 MiB │ esgf-node.ornl.gov
7 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.CFmon.c… │ 1 │ 36.0 GiB │ esgf-data.ucar.edu
8 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.CFmon.c… │ 1 │ 980.0 MiB │ esgf-node.ornl.gov
9 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.CFmon.c… │ 1 │ 964.7 MiB │ esgf-data.ucar.edu
10 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.CFmon.p… │ 1 │ 985.8 MiB │ esgf-data.ucar.edu
11 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.CFmon.p… │ 1 │ 985.8 MiB │ esgf-node.ornl.gov
12 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Emon.mr… │ 1 │ 3.2 GiB │ eagle.alcf.anl.gov
13 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Emon.ps… │ 1 │ 711.3 MiB │ eagle.alcf.anl.gov
14 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Lmon.mr… │ 1 │ 324.7 MiB │ eagle.alcf.anl.gov
15 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Lmon.tr… │ 1 │ 310.3 MiB │ esgf-node.ornl.gov
16 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Omon.so… │ 36 │ 1.0 MiB │ esgf-node.ornl.gov
17 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.Omon.to… │ 36 │ 1.0 MiB │ esgf-data.ucar.edu
18 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.SImon.s… │ 1 │ 59.9 kiB │ eagle.alcf.anl.gov
19 │ CMIP6.HighResMIP.NCAR.CESM1-CAM5-SE-HR.highres-future.r1i1p1f1.SImon.s… │ 1 │ 1.6 GiB │ eagle.alcf.anl.gov
I also set the replica option to true.
(workspace) root@<container_id>:/workspace# uv run esgpull config api.default_options.replica true
[api.default_options]
replica = "true"
Previous value: none
I’m not certain this step is strictly necessary, but it works.
Here is what the configuration for my esgpull install looks like.
(arctichoke) root@<container_id>:/workspace# uv run esgpull config
──────────────────────────── /arctichoke_data/bergybits/config.toml ────────────────────────────
[paths]
data = "/arctichoke_data/bergybits/data"
db = "/arctichoke_data/bergybits/db"
log = "/arctichoke_data/bergybits/log"
tmp = "/arctichoke_data/bergybits/tmp"
plugins = "/arctichoke_data/bergybits/plugins"
[credentials]
filename = "credentials.toml"
[cli]
page_size = 20
[db]
filename = "esgpull.db"
[download]
chunk_size = 67108864
http_timeout = 20
max_concurrent = 5
disable_ssl = false
disable_checksum = false
show_filename = false
[api]
index_node = "esgf-node.ornl.gov/esgf-1-5-bridge"
http_timeout = 20
max_concurrent = 5
page_limit = 50
default_query_id = ""
use_custom_distribution_algorithm = false
[api.default_options]
distrib = "true"
latest = "true"
replica = "true"
retracted = "false"
[plugins]
enabled = false
Documentation
Building documentation
In the section, Packages for documentation, I added the myst-nb sphinx-autoapi and sphinx-rtd-theme packages to the development group of the project.
To build the documentation, activate the virtual environment, go into the docs/ directory, and run the make command.
This will produce a lot of output, especially the first time running, so I have hidden most of it.
(arctichoke) root@<container_id>:/workspace# cd docs
(arctichoke) root@<container_id>:/workspace/docs# make html
Running Sphinx v9.1.0
loading translations [en]... done
loading pickled environment... The configuration has changed (3 options: 'html_permalinks_icon', 'jquery_use_sri', 'mathjax3_config')
done
...
Expand for more output.
...
myst v5.0.0: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions={'amsmath', 'dollarmath'}, disable_syntax=[], all_links_external=False, links_external_new_tab=False, url_schemes=('http', 'https', 'wiki', 'doi', 'gh-issue'), ref_domains=None, fence_as_directive=set(), number_code_blocks=[], title_to_header=False, heading_anchors=0, heading_slug_func=None, html_meta={}, footnote_sort=True, footnote_transition=True, words_per_minute=200, substitutions={}, linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', enable_checkboxes=False, suppress_warnings=[], highlight_code_blocks=True)
myst-nb v1.4.0: NbParserConfig(custom_formats={}, metadata_key='mystnb', cell_metadata_key='mystnb', kernel_rgx_aliases={}, eval_name_regex='^[a-zA-Z_][a-zA-Z0-9_]*$', execution_mode='auto', execution_cache_path='', execution_excludepatterns=(), execution_timeout=30, execution_in_temp=False, execution_allow_errors=False, execution_raise_on_error=False, execution_show_tb=False, merge_streams=False, render_plugin='default', remove_code_source=False, remove_code_outputs=False, scroll_outputs=False, code_prompt_show='Show code cell {type}', code_prompt_hide='Hide code cell {type}', number_source_lines=False, output_stderr='show', render_text_lexer='myst-ansi', render_error_lexer='ipythontb', render_image_options={}, render_figure_options={}, render_markdown_format='commonmark', output_folder='build', append_css=True, metadata_to_fm=False)
Using jupyter-cache at: /workspace/docs/_build/.jupyter_cache
[AutoAPI] Reading files... [100%] /workspace/src/arctichoke/params/var_params.py
[AutoAPI] Mapping Data... [100%] /workspace/src/arctichoke/params/var_params.py
[AutoAPI] Rendering Data... [ 4%] arctichoke
[AutoAPI] Rendering Data... [ 7%] arctichoke.plot
[AutoAPI] Rendering Data... [ 11%] arctichoke.path
[AutoAPI] Rendering Data... [ 15%] arctichoke.verify
[AutoAPI] Rendering Data... [ 19%] arctichoke.params
[AutoAPI] Rendering Data... [ 22%] arctichoke.dataset
[AutoAPI] Rendering Data... [ 26%] arctichoke.arctichoke
[AutoAPI] Rendering Data... [ 30%] arctichoke.dataset.tmp
[AutoAPI] Rendering Data... [ 33%] arctichoke.plot.hvplots
[AutoAPI] Rendering Data... [ 37%] arctichoke.path.file_lists
[AutoAPI] Rendering Data... [ 41%] arctichoke.plot.time_series
[AutoAPI] Rendering Data... [ 44%] arctichoke.path.model_paths
[AutoAPI] Rendering Data... [ 48%] arctichoke.plot.save_hvplots
[AutoAPI] Rendering Data... [ 52%] arctichoke.plot.limit_extent
[AutoAPI] Rendering Data... [ 56%] arctichoke.dataset.date_type
[AutoAPI] Rendering Data... [ 59%] arctichoke.dataset.grid_type
[AutoAPI] Rendering Data... [ 63%] arctichoke.params.var_params
[AutoAPI] Rendering Data... [ 67%] arctichoke.verify.verify_path
[AutoAPI] Rendering Data... [ 70%] arctichoke.dataset.field_mean
[AutoAPI] Rendering Data... [ 74%] arctichoke.plot.seasonal_cycle
[AutoAPI] Rendering Data... [ 78%] arctichoke.path.variable_paths
[AutoAPI] Rendering Data... [ 81%] arctichoke.dataset.get_variable
[AutoAPI] Rendering Data... [ 85%] arctichoke.dataset.trim_dataset
[AutoAPI] Rendering Data... [ 89%] arctichoke.params.latlon_params
[AutoAPI] Rendering Data... [ 93%] arctichoke.dataset.set_date_type
[AutoAPI] Rendering Data... [ 96%] arctichoke.path.manipulate_paths
[AutoAPI] Rendering Data... [100%] arctichoke.dataset.example_dataset
[autosummary] generating autosummary for: autoapi/index.rst, autoapi/arctichoke/dataset/date_type/index.rst, autoapi/arctichoke/dataset/example_dataset/index.rst, autoapi/arctichoke/dataset/field_mean/index.rst, autoapi/arctichoke/dataset/get_variable/index.rst, autoapi/arctichoke/dataset/grid_type/index.rst, autoapi/arctichoke/dataset/index.rst, autoapi/arctichoke/dataset/set_date_type/index.rst, autoapi/arctichoke/dataset/tmp/index.rst, autoapi/arctichoke/dataset/trim_dataset/index.rst, ..., docs_setup/HighResMIP_choices.md, docs_setup/data_dir_structure.md, docs_setup/esgpull_downloads.md, docs_setup/globus_downloads.md, docs_setup/initial_setup.md, docs_setup/installation.md, docs_setup/jupyter_test.md, docs_setup/start_container.md, example.ipynb, index.md
building [mo]: targets for 0 po files that are out of date
writing output...
building [html]: targets for 28 source files that are out of date
updating environment: 0 added, 28 changed, 0 removed
reading sources... [100%] autoapi/arctichoke/verify/verify_path/index
/workspace/docs/autoapi/arctichoke/dataset/field_mean/index.rst:29: WARNING: duplicate object description of arctichoke.dataset.field_mean.cdo, other instance in autoapi/arctichoke/dataset/field_mean/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/dataset/index.rst:144: WARNING: duplicate object description of arctichoke.dataset.cdo, other instance in autoapi/arctichoke/dataset/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/dataset/index.rst:146: WARNING: duplicate object description of arctichoke.dataset.meta_vars, other instance in autoapi/arctichoke/dataset/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/dataset/index.rst:161: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/dataset/trim_dataset/index.rst:30: WARNING: duplicate object description of arctichoke.dataset.trim_dataset.cdo, other instance in autoapi/arctichoke/dataset/trim_dataset/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/dataset/trim_dataset/index.rst:43: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/path/index.rst:269: WARNING: duplicate object description of arctichoke.path.get_model_path, other instance in autoapi/arctichoke/path/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/path/index.rst:301: WARNING: duplicate object description of arctichoke.path.list_available_models, other instance in autoapi/arctichoke/path/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/path/index.rst:388: WARNING: duplicate object description of arctichoke.path.get_variable_path, other instance in autoapi/arctichoke/path/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/plot/hvplots/index.rst:38: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/index.rst:67: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/index.rst:101: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/index.rst:116: WARNING: duplicate object description of arctichoke.plot.get_limited_extent, other instance in autoapi/arctichoke/plot/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/plot/index.rst:127: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/index.rst:141: WARNING: duplicate object description of arctichoke.plot.save_hvplot, other instance in autoapi/arctichoke/plot/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/plot/index.rst:194: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/index.rst:200: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/limit_extent/index.rst:29: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/seasonal_cycle/index.rst:37: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/seasonal_cycle/index.rst:43: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/time_series/index.rst:34: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/plot/time_series/index.rst:40: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils]
/workspace/docs/autoapi/arctichoke/verify/index.rst:27: WARNING: duplicate object description of arctichoke.verify.verify_path, other instance in autoapi/arctichoke/verify/verify_path/index, use :no-index: for one of them
/workspace/docs/autoapi/arctichoke/verify/verify_path/index.rst:4: WARNING: duplicate object description of arctichoke.verify.verify_path, other instance in autoapi/arctichoke/verify/index, use :no-index: for one of them
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets...
copying static files...
Writing evaluated template result to /workspace/docs/_build/html/_static/basic.css
Writing evaluated template result to /workspace/docs/_build/html/_static/language_data.js
Writing evaluated template result to /workspace/docs/_build/html/_static/documentation_options.js
Writing evaluated template result to /workspace/docs/_build/html/_static/js/versions.js
copying static files: done
copying extra files...
copying extra files: done
copying assets: done
writing output... [100%] index
generating indices... genindex py-modindex done
highlighting module code... [100%] arctichoke.verify.verify_path
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
...
...
build succeeded, 24 warnings.
The HTML pages are in _build/html.
Then, I went into the docs/html/index.html in VSCodium, hit the preview button, then the top right menu to open the page in a browser.
The test webpage seems like it rendered properly.
When troubleshooting, it can be helpful to try making the html directory clean and recreating those documents, in case some cached data is causing issues.
(arctichoke) root@<container_id>:/workspace/docs# make clean
Removing everything under '_build'...
Then, run the make html command again.
Hosting documentation
Following Py-Pkgs section 3.8.5. Hosting documentation online to be able to have the documentation you are reading hosted on Read the Docs.
First, a few changes need to be made to the .readthedocs.yml file to ensure the documentation can successfully be hosted.
Upon generating the cookiecutter files, the configuration looked like this:
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the OS, Python version and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.13.5"
jobs:
post_create_environment:
# Install poetry
# https://python-poetry.org/docs/#installing-manually
- pip install poetry
post_install:
- VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --all-groups
# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/conf.py
Two changes need to be made here.
The first is to specify the version of Python as 3.13 instead of 3.13.5 as Read the Docs only recognizes up to the minor version number, not the patch number.
Second, the jobs section needs to be changed to use uv instead of poetry.
I followed an example shown in the GitHub issue for Read the Docs #11289, Support uv.
The .readthedocs.yml file now reads as follows.
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the OS, Python version and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.13"
jobs:
pre_create_environment:
- asdf plugin add uv
- asdf install uv latest
- asdf global uv latest
create_environment:
- uv venv $READTHEDOCS_VIRTUALENV_PATH
install:
# Use a cache dir in the same mount to halve the install time
- VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH uv pip install --cache-dir $READTHEDOCS_VIRTUALENV_PATH/../../uv_cache -r docs/requirements.txt
# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/conf.py
Next, I went to Read the Docs and logged in. On the “Projects” dashboard, I clicked “Add project.” On the next page, I selected “Configure manually” then filled in this information:
Name
arctichoke
Repository URL
https://github.com/scheemik/arctichoke
Default branch
main
Language
English
Then, I clicked “Next” and confirmed that the file .readthedocs.yaml exists already.
Upon building, I was greeted with confirmation that the build succeeded.
Version latest / Builds / #32314755
git clone --depth 1 https://github.com/scheemik/arctichoke .
git fetch origin --force --prune --prune-tags --depth 50 refs/heads/main:refs/remotes/origin/main
git checkout --force origin/main
cat .readthedocs.yml
asdf global python 3.14.0
asdf plugin add uv
asdf install uv latest
asdf global uv latest
uv venv $READTHEDOCS_VIRTUALENV_PATH
VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH uv pip install --cache-dir $READTHEDOCS_VIRTUALENV_PATH/../../uv_cache -r docs/requirements.txt
python -m sphinx -T -j auto -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html
Now, the documentation for this project is live and available to view at https://arctichoke.readthedocs.io/en/latest/. I put that URL in the “About” section of the GitHub repository under “Website” to make it more visible to people who find the project.
Enabling \(\LaTeX\) math syntax
In order to be able to use dollar signs to quickly indicate a mathematical symbol from \(\LaTeX\), I added the dollarmath and amsmath extensions to the docs/conf.py file’s myst_enable_extensions list.
myst_enable_extensions = [
"dollarmath",
"amsmath"
]
This allows me to render mathematical syntax, such as a speed of $\sim$ 1 km day$^{−1}$ which renders as \(\sim\) 1 km day\(^{−1}\).
See the MyST-Parser documentation page on Syntax Extensions
Enabling easy DOI links
When making citations in the documentation, it can be cumbersome to link to a DOI.
For example, to cite the EC-Earth3P-HR model, I would type out the link [doi:10.22033/ESGF/CMIP6.2323](https://doi.org/10.22033/ESGF/CMIP6.2323) which renders as doi:10.22033/ESGF/CMIP6.2323.
In the documentation of my HighResMIP Choices, I make around 50 citations.
To set up a URL scheme that makes DOI links shorter, I added the following to docs/conf.py:
myst_url_schemes = {
"http": None,
"https": None,
"wiki": "https://en.wikipedia.org/wiki/{{path}}#{{fragment}}",
"doi": "https://doi.org/{{path}}",
"gh-issue": {
"url": "https://github.com/executablebooks/MyST-Parser/issue/{{path}}#{{fragment}}",
"title": "Issue #{{path}}",
"classes": ["github"],
},
}
Now, I can simply type out <doi:10.22033/ESGF/CMIP6.2323> which renders as doi:10.22033/ESGF/CMIP6.2323, the same link but with many fewer characters.
See the MyST-Parser documentation page on Customizing external URL resolution for more information.