Golang garbage collection of open file descriptors

Bruce Ediger

2025-02-20 (Last Modified: 2025-02-22)

When you write Go code, you use os.Stdout and os.Stderr instead of our old C friends, stdout and stderr from stdio.h. os.Stdin, os.Stdout and os.Stderr have type *os.File. This Go type encapsulates encapsulates Linux file descriptors (small integer values representing open files).

os.Stdin, os.Stdout and os.Stderr act almost exactly like the stdio.h header file’s stdin, stdout, stderr, which are C type FILE *. The C type also encapsulate (small integer value) file descriptors.

These file descriptors are a scarce resource. Linux (and other Unix-like or Posix operating systems) have a fairly low system wide limit on the number of open file descriptors.

$ cat /proc/sys/fs/file-max
9223372036854775807

Well, open file descriptors have been a scarce resource in the past. Seriously, 9223372036854775807 open file descriptors can’t be correct. Each open file descriptor, just an integer value in its process, is a struct in the kernel, and potentially not a small one, what with readahead and write behind and all kinds of fancy locks and so forth.

It happens that 9223372036854775807₁₀ = 0x7FFFFFFFFFFFFFFF, the largest number representable in a positive, twos-complement 64-bit integer.

I’ve got Linux kernel 6.13 running right now. The per-process limit on open file descriptors is 1024. There’s still some limit on open file descriptors. That means there’s a reason for processes to close open file descriptors when they’re no longer needed. In C programs, that would look like fclose(fin); at the stdio level of abstraction, or close(fd); at the system call level. In Go, it would look like fin.Close(), vaguely object oriented.

A few days ago, I found a long-running program that did not call Close on a particular *os.File instance, but nevertheless did not have a large number of open file descriptors.

One of the benefits of Linux: you can look at a process’ open file descriptors by knowing its Process ID (PID), and listing the files in /proc/$PID/fd:

$ ls -l /proc/$(pgrep some-identifier)/fd
total 0
lrwx------ 1 bediger bediger 64 Feb 22 17:40 0 -> /dev/pts/0
lrwx------ 1 bediger bediger 64 Feb 22 17:40 1 -> /dev/pts/0
lrwx------ 1 bediger bediger 64 Feb 22 17:40 10 -> /dev/pts/0
lrwx------ 1 bediger bediger 64 Feb 22 17:40 2 -> /dev/pts/0

The long-running program only had 5 or so open file descriptors, typical for a Go program. The program’s code definitely did not call Close() on one particular set of *os.File instances. The program created and wrote to temporary files, all set up by calls to os.CreateTemp(). None of the long-running program’s open file descriptors in /proc/$PID/fd/ was associated with any of the files resulting from o.CreateTemp() calls.

I wrote a little program of my own to see if something I didn’t comprehend was closing open file descriptors. This program creates a file named by a certain pattern in a goroutine (a.k.a., a thread). The thread exits without calling Close() on the *os.File instance resulting from calling os.OpenFile(). The underlying Linux file descriptor should remain open, and appear in /proc/$WHATEVERPID/fd/ as a link to the real file. My little program read link names from /proc/$WHATEVERPID/fd/, then read the linked-to file names. Counted linked-to file names with a regular expression that should accept only the file names that the os.OpenFile() call made.

Files opened as:

const fileNamePrefix = "Glump"
f, err := os.OpenFile(fmt.Sprintf("%s%04d", fileNamePrefix, n), os.O_RDWR|os.O_CREATE, 0777)

The regular expression used to match linked-to file names:

var fileNamePattern = regexp.MustCompile(fmt.Sprintf("^/..*/%s[0-9]+$", fileNamePrefix))

Since the link files are in /proc/$PID/fd/, they must contain the fully-qualified path of the file created by my program. The regular expression matches whatever path to those files, plus the file name prefix (“Glump” in the code fragments above), and the suffix comprised of digits.

This technique allows the program to detect if it’s leaving open file descriptors behind.

Code repository

Here’s what my program prints out:

1m0.001265712s - created 1 files, 1 open file descriptors
2m0.001348759s - created 2 files, 2 open file descriptors
3m0.000981728s - created 3 files, 0 open file descriptors
4m0.001607638s - created 4 files, 1 open file descriptors
5m0.000647874s - created 5 files, 0 open file descriptors
6m0.00085456s - created 6 files, 1 open file descriptors
...

Every minute, it creates a new file, does NOT close the *os.File, then counts open file descriptors. You can see that the process does not accumulate open file descriptors. As if deleted by an Occult Hand, the open file descriptors disappear.

As it happens, type *os.File sets up a “finalizer” that closes the open file descriptor when the memory of the *os.File gets garbage collected. Depending on how often the Go runtime does garbage collection, any leftover open file descriptors get closed. It’s not an Occult Hand at all.

I don’t think that a cautious programmer will rely on this. It is documented behavior if you look hard enough, but exactly if and when a Go program’s runtime executes garbage collection is somewhat hard to figure out. I had some vague recollection that the Go runtime would let 60 seconds pass before starting a garbage collection. I no longer think this is true.

I got different garbage collection behavior from my program with slightly different code.

If I put in code to use the Go runtime tracer, my program never got more than 2 open file descriptors.

If I put in code to call runtime.GC() after opening 10 files (forcing garbage collection), my program would dutifully create 10 files, leaving all 10 file descriptors open. After running runtime.GC(), all 10 open file descriptors disappeared.

If I had my program create a file every 20 seconds, and not execute runtime.GC() until it had created 100 files, it created 100 files, then closed every single open file descriptor when runtime.GC() executed.

When I had my program open a file every 10 seconds, and never run runtime.GC(), it created 127 open file descriptors. Sometime between 127 and 128 open file descriptors, it closed 30 leaving 97 open. At 129 open files, it had only 1 open file descriptor.

The moral of the story is to close all open *os.File instances, lest weird resource problems plague you.