Recently I’ve been working on a pretty big rust project and to my surprise I couldn’t get tests to work properly.
Running cargo test would start running all the tests in the repo and after a
couple of milliseconds every single test would start to fail because of an error
that I’m not very familiar with
Io(Os { code: 24, kind: Other, message: "Too many open files" })Fortunately, the error is pretty explicit and straightfoward so I was able to understand what was going on in a reasonable time. I’ve started digging a bit and learned some stuff along the way.
Ever wondered how your programs juggle multiple tasks - reading files, sending data over the network, or even just displaying text on your screen - all at once? File descriptors are what make this all possible (in Unix systems).
At its core, a file descriptor (often abbreviated as fd) is simply a positive integer used by the operating system kernel to identify an open file. In Unix, "everything is a file." Contrary to what the word says, a file descriptor doesn’t just refer to regular files on your disk. It can represent:
-
Regular files: The documents, images, and code files you interact with daily.
-
Directories: Yes, even directories are treated like files to some extent, allowing programs to list their contents.
-
Pipes: Used for inter-process communication, allowing one program’s output to become another’s input.
-
Sockets: The endpoints for network communication, whether it’s talking to a web server or another application on your local machine.
-
Devices: Hardware devices like your keyboard, mouse, and printer are also accessed via file descriptors.
When a program wants to interact with any of these resources, it first asks the kernel to "open" it. If successful, the kernel returns a file descriptor, which the program then uses for all subsequent operations (reading, writing, closing, etc.).
By convention, every Unix process starts with at least three standard file descriptors automatically opened:
-
0: Standard Input (stdin) - Typically connected to your keyboard for user input. -
1: Standard Output (stdout) - Usually connected to your terminal for displaying normal program output. -
2: Standard Error (stderr) - Also usually connected to your terminal, but specifically for displaying error messages.
On macOS we can quickly check this, open your favorite terminal and run ls
/dev/fd.
$ ls -lah /dev/fd
Permissions Size User Date Modified Name
crw--w---- 16,2 mattrighetti 4 Jun 00:44 0
crw--w---- 16,2 mattrighetti 4 Jun 00:44 1
crw--w---- 16,2 mattrighetti 4 Jun 00:44 2
dr--r--r-- - root 24 May 08:23 3On Linux we can do something similar but the repository is different and usually
follows the current pattern /proc/<pid>/fd. Running the same on Linux gives me this:
$ echo $$ // prints the current process id
2806524
$ sudo ls -lah /proc/2806524/fd
total 0
dr-x------ 2 root root 11 Jun 4 00:40 .
dr-xr-xr-x 9 pi pi 0 Jun 4 00:39 ..
lrwx------ 1 root root 64 Jun 4 00:40 0 -> /dev/null
lrwx------ 1 root root 64 Jun 4 00:40 1 -> /dev/null
lrwx------ 1 root root 64 Jun 4 00:40 10 -> /dev/ptmx
lrwx------ 1 root root 64 Jun 4 00:40 11 -> /dev/ptmx
lrwx------ 1 root root 64 Jun 4 00:40 2 -> /dev/null
lrwx------ 1 root root 64 Jun 4 00:40 3 -> 'socket:[14023056]'
lrwx------ 1 root root 64 Jun 4 00:40 4 -> 'socket:[14023019]'
lrwx------ 1 root root 64 Jun 4 00:40 5 -> 'socket:[14022300]'
lrwx------ 1 root root 64 Jun 4 00:40 6 -> 'socket:[14023037]'
lrwx------ 1 root root 64 Jun 4 00:40 7 -> /dev/ptmx
l-wx------ 1 root root 64 Jun 4 00:40 8 -> /run/systemd/sessions/1501.refAs you can see, we have 0, 1 and 2 as expected, but we also have a bunch of
other file descriptors.
Another useful command to check for open file descriptors is lsof, which
stands for "list open files".
$ lsof -p $(echo $$)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
zsh 39367 mattrighetti cwd DIR 1,17 2496 250127 /Users/mattrighetti
zsh 39367 mattrighetti txt REG 1,17 1361200 1152921500312522433 /bin/zsh
zsh 39367 mattrighetti txt REG 1,17 81288 1152921500312535786 /usr/share/locale/en_US.UTF-8/LC_COLLATE
zsh 39367 mattrighetti txt REG 1,17 170960 1152921500312525313 /usr/lib/zsh/5.9/zsh/zutil.so
zsh 39367 mattrighetti txt REG 1,17 118896 1152921500312525297 /usr/lib/zsh/5.9/zsh/terminfo.so
zsh 39367 mattrighetti txt REG 1,17 171344 1152921500312525281 /usr/lib/zsh/5.9/zsh/parameter.so
zsh 39367 mattrighetti txt REG 1,17 135696 1152921500312525255 /usr/lib/zsh/5.9/zsh/datetime.so
zsh 39367 mattrighetti txt REG 1,17 135568 1152921500312525291 /usr/lib/zsh/5.9/zsh/stat.so
zsh 39367 mattrighetti txt REG 1,17 338592 1152921500312525247 /usr/lib/zsh/5.9/zsh/complete.so
zsh 39367 mattrighetti txt REG 1,17 136880 1152921500312525293 /usr/lib/zsh/5.9/zsh/system.so
zsh 39367 mattrighetti txt REG 1,17 593088 1152921500312525303 /usr/lib/zsh/5.9/zsh/zle.so
zsh 39367 mattrighetti txt REG 1,17 134928 1152921500312525287 /usr/lib/zsh/5.9/zsh/rlimits.so
zsh 39367 mattrighetti txt REG 1,17 117920 1152921500312525263 /usr/lib/zsh/5.9/zsh/langinfo.so
zsh 39367 mattrighetti txt REG 1,17 2289328 1152921500312524246 /usr/lib/dyld
zsh 39367 mattrighetti txt REG 1,17 208128 1152921500312525249 /usr/lib/zsh/5.9/zsh/complist.so
zsh 39367 mattrighetti txt REG 1,17 118688 1152921500312525285 /usr/lib/zsh/5.9/zsh/regex.so
zsh 39367 mattrighetti txt REG 1,17 118288 1152921500312525305 /usr/lib/zsh/5.9/zsh/zleparameter.so
zsh 39367 mattrighetti 0u CHR 16,1 0t17672 1643 /dev/ttys001
zsh 39367 mattrighetti 1u CHR 16,1 0t17672 1643 /dev/ttys001
zsh 39367 mattrighetti 2u CHR 16,1 0t17672 1643 /dev/ttys001
zsh 39367 mattrighetti 10u CHR 16,1 0t5549 1643 /dev/ttys001According to the lsof documentation:
-
cwd: The current working directory of the process. -
txt: Executable files or shared libraries loaded into memory (e.g., /bin/zsh, modules like zutil.so, or system libraries like /usr/lib/dyld). -
0u,1u,2u: Standard input (0), output (1), and error (2) streams, respectively. Theumeans the descriptor is open for both reading and writing. These are tied to/dev/ttys001(my current terminal device). -
10u: Another file descriptor (also tied to/dev/ttys001`), likely used for additional terminal interactions.
We now know that file descriptors are a way for the operating system to keep track of open files and other resources, nice!
Have you ever wondered how many file descriptors can be open at the same time? The most common answer in software engineering applies here too: It depends.
Each operating system has its own limits on the number of file descriptors a process can open simultaneously. These limits are in place to prevent a single misbehaving program from hogging all available resources and crashing the system.
On macOS, we can easily inspect these limits using the sysctl and
ulimit commands in your terminal.
$ sysctl kern.maxfiles
kern.maxfiles: 245760
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 122880
$ ulimit -n
256
-
kern.maxfilesrepresents the absolute maximum number of file descriptors that can be open across the entire macOS system at any given moment. It’s a global governor, preventing the system from running out of file descriptor resources, even if many different applications are running. -
kern.maxfilesperprocis the hard limit on the number of file descriptors that a single process can have open. Think of it as the ultimate ceiling for an individual application. No matter what, a process cannot open more files than this hard limit set by the kernel. -
ulimit -nis your shell’s "soft" limit for the number of open file descriptors. If a process tries to open more files than its soft limit, the operating system will typically return an error (e.g., "Too many open files"). The good news is that a process can raise its own soft limit, but only up to its hard limit.
Enough with the theory, let’s get back to the problem I was having with my rust
tests. My assumption was that since cargo test gets executed in my terminal,
it inevitably reaches a point where it tries to open more files than the soft
limit set by my shell, which is 256 in this case. When that happens, the
operating system screams at cargo and tells it that it can’t open any more
files, cargo then propagates that error to the tests and they all fail.
I wanted to confirm this hypothesis, so I created this monitoring script that
watches for cargo test PID and prints the number of open file descriptors at
different intervals.
#!/bin/bash
# This function exits the script gracefully
function cleanup() {
echo -e "\nstopping."
exit 0
}
# This function encapsulates the logic for formatting and printing the monitoring output.
# Arguments:
# $1: Initial PID
# $2: Total number of open files
print_status() {
local initial_pid="$1"
local total_open_files="$2"
echo "$(date '+%H:%M:%S') - Main PID ($initial_pid) - open: ${total_open_files}"
}
PROCESS_NAME="cargo"
COMMAND_ARGS="test"
echo "press ctrl+c to stop."
# Find the Process ID (PID) of the initial command.
INITIAL_PID=$(pgrep -f "$PROCESS_NAME.*$COMMAND_ARGS" | head -n 1)
if [ -z "$INITIAL_PID" ]; then
echo "waiting for '$PROCESS_NAME $COMMAND_ARGS' to start..."
# If the process isn't found immediately, loop and wait for it.
sleep 0.01
while [ -z "$INITIAL_PID" ]; do
INITIAL_PID=$(pgrep -f "$PROCESS_NAME.*$COMMAND_ARGS" | head -n 1)
done
fi
echo "Found '$PROCESS_NAME $COMMAND_ARGS' with PID: $INITIAL_PID"
# trap command catches the INT signal (triggered by Ctrl+C)
# and calls the cleanup function to exit gracefully.
trap cleanup INT
while true; do
# check if the main process (INITIAL_PID) is still running.
if ! ps -p "$INITIAL_PID" > /dev/null; then
echo "PID $INITIAL_PID no longer running. bye!"
break
fi
# `sudo lsof -p "$INITIAL_PID"` lists all open files for this specific PID.
# `2>/dev/null` redirects stderr (errors like "process not found") to null.
# `grep -v " txt "` filters out loaded executable code and libraries for a more relevant count.
# `wc -l` counts the lines, effectively the number of open files.
# `tr -d ' '` removes any leading/trailing spaces for clean arithmetic.
OPEN_FILES_COUNT=$(sudo lsof -p "$INITIAL_PID" 2>/dev/null | grep -v " txt " | wc -l | tr -d ' ')
# Ensure COUNT is not empty (it might be if lsof returns nothing)
if [ -z "$OPEN_FILES_COUNT" ]; then
OPEN_FILES_COUNT=0
fi
print_status "$INITIAL_PID" "$TOTAL_OPEN_FILES"
doneNote that usually to get an accurate count of open file descriptors you would also need to consider the entire process tree, not just the main process. This is because child processes can also open files, and their file descriptors contribute to the total count. In my case, there was only one process (`cargo test`) running, so I didn't.
I can now run this script in one terminal and run cargo test in another
terminal. I actually had to do this a couple of times to get a good sample of
data, rust runs pretty fast once the code is compiled and this monitor script
does not run fast enough to catch all the open file descriptors changes.
$ sudo ./monitor.sh
press ctrl+c to stop.
waiting for 'cargo test' to start...
Found 'cargo test' with PID: 44152
01:46:21 - Main PID (44152) - open: 14
01:46:21 - Main PID (44152) - open: 32
01:46:21 - Main PID (44152) - open: 78
01:46:21 - Main PID (44152) - open: 155
01:46:21 - Main PID (44152) - open: 201
01:46:21 - Main PID (44152) - open: 228
01:46:21 - Main PID (44152) - open: 231
01:46:21 - Main PID (44152) - open: 237 # errors started happening here
01:46:21 - Main PID (44152) - open: 219
01:46:21 - Main PID (44152) - open: 205
01:46:21 - Main PID (44152) - open: 180
01:46:21 - Main PID (44152) - open: 110
01:46:21 - Main PID (44152) - open: 55
01:46:21 - Main PID (44152) - open: 28
01:46:21 - Main PID (44152) - open: 15
01:46:21 - Main PID (44152) - open: 0
PID 44152 no longer running. bye!I couldn’t get the script to catch the exact moment the process reached the soft limits, but I can clearly see that the tests starts failing when the number of open file descriptors reaches 237, which is pretty close to the soft limit of 256.
Time to fix this! This is a bit underwhelming, but the solution is to just bump
the soft limit of open file descriptors in my shell. I can do this by using the
ulimit command again.
$ ulimit -n 8192
$ ulimit -n
8192Running cargo test now works as expected and no "Too many open files" error is thrown.
The above chart shows the number of open file descriptors with the new soft limit. As you can see the max value reached is around 1600, which is way above the previous limit of 256.
All in all, this was a fun exercise that taught me a lot about file descriptors and how they work in Unix-like systems. Now you know how to troubleshoot this error that might pop up in your own projects!
