When we first learn to program, we are taught to be mindful of the computer’s resources; they must be used sparingly and released as soon as possible. So you know the drill: close files, release database connections, close sockets, free memory, etc.
But have you ever wondered who sets the limits of what can be done?
Open files limit
Before answering that question, let’s see these ideas in action by running the following program. It’s a simple Java application to open 1 million files (without closing them!)
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
public class MainJava {
public static void main(String[] args) throws IOException {
int count = 1000000;
FileWriter[] arr = new FileWriter[count];
for (int i=0; i<count; i++) {
File f = new File("/Users/franciscoalvarez/myfiles/f_"+String.format("%05d",i));
FileWriter fw = new FileWriter(f);
arr[i] = fw;
}
}
}
When I run this program on my laptop, I get this:
Exception in thread "main" java.io.FileNotFoundException: /Users/franciscoalvarez/myfiles/f_10235 (Too many open files) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at java.io.FileOutputStream. (FileOutputStream.java:162) at java.io.FileWriter. (FileWriter.java:90) at MainJava.main(MainJava.java:12)
The program creates 10,235 files (from f_00000 to f_10234) before complaining about ‘too many open files’.
You may get different results though. In order to understand why, let’s repeat this experiment using different versions of Java.
Java 8
franciscoalvarez@franciscos filelimit % java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Srver VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)
franciscoalvarez@franciscos filelimit % java MainJava
Exception in thread "main" java.io.FileNotFoundException: /Users/franciscoalvarez/myfiles/f_10235 (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:162)
at java.io.FileWriter.(FileWriter.java:90)
at MainJava.main(MainJava.java:12)
Java 11
franciscoalvarez@franciscos filelimit % java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
Eclipse OpenJ9 VM AdoptOpenJDK-11.0.11+9 (build openj9-0.26.0, JRE 11 Mac OS X amd64-64-Bit Compressed References 20210421_957 (JIT enabled, AOT enabled)
OpenJ9 - b4cc246d9
OMR - 162e6f729
JCL - 7796c80419 based on jdk-11.0.11+9)
franciscoalvarez@franciscos filelimit % java MainJava
Exception in thread "main" java.io.FileNotFoundException: /Users/franciscoalvarez/myfiles/f_49146 (Too many open files)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:237)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:187)
at java.base/java.io.FileWriter.(FileWriter.java:96)
at MainJava.main(MainJava.java:12)
Java 16
franciscoalvarez@franciscos filelimit % java -version
openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (build 16.0.1+9)
Eclipse OpenJ9 VM AdoptOpenJDK-16.0.1+9 (build openj9-0.26.0, JRE 16 Mac OS X amd64-64-Bit Compressed References 20210421_24 (JIT enabled, AOT enabled)
OpenJ9 - b4cc246d9
OMR - 162e6f729
JCL - cea22090ecf based on jdk-16.0.1+9)
franciscoalvarez@franciscos filelimit % java MainJava
Exception in thread "main" java.io.FileNotFoundException: /Users/franciscoalvarez/myfiles/f_49146 (Too many open files)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:291)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:234)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:184)
at java.base/java.io.FileWriter.(FileWriter.java:96)
at MainJava.main(MainJava.java:12)
Whereas in Java 8 we are limited to a maximum of 10,235 files, in Java 11 and 16 that amount goes up to 49,146.
Some theory
Java 8
To understand the behaviour of our little program, we need to explain some theory (hopefully, not too much).
Operating systems (here we’ll limit our discussion to POSIX-compliant operating systems, the likes of Unix, Linux, macOS) set limits on some system resources like the number of files that can be held open by a process.
These limits are governed by different standards and specs: in particular, the maximum number of open files held by a process must be at least 20
On my Mac, that value can be found in the header /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include/limits.h:
#define _POSIX_OPEN_MAX 20
The key words of the previous statement are ‘at least’: that means that different systems are free to increase that value. And that’s the case on my Mac, where we can find the following definition in the header /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include/sys/syslimits.h:
#define OPEN_MAX 10240 /* max open files per process - todo, make a config option? */
10,240! That makes sense, it’s close to the value 10,235 obtained when running Java 8.
We didn’t mention it before, in reality the number of open files was not 10,235 but 10,238. These 3 extra files are standard input, standard output and standard error, which are opened by default by all processes. As to the remaining 2 files to reach 10,240, they can be anything really (processes open all sorts of file descriptors to execute their tasks).
Java 11+
What about the results obtained with the other Java versions? Well, processes are free to increase the limits up to a certain value called ‘hard limit’. By contrast, the current value of a limit is called ‘soft limit’. So we are to assume that the JVM that runs our program has increased the soft limit beyond 10,240. But, how can we prove that?
Let’s run the program again and take a core dump with “Ctrl-\”. Here’s a fragment of the core dump:
1CIUSERLIMITS User Limits (in bytes except for NOFILE and NPROC) NULL ------------------------------------------------------------------------ NULL type soft limit hard limit 2CIUSERLIMIT RLIMIT_AS unlimited unlimited 2CIUSERLIMIT RLIMIT_CORE unlimited unlimited 2CIUSERLIMIT RLIMIT_CPU unlimited unlimited 2CIUSERLIMIT RLIMIT_DATA unlimited unlimited 2CIUSERLIMIT RLIMIT_FSIZE unlimited unlimited 2CIUSERLIMIT RLIMIT_MEMLOCK unlimited unlimited 2CIUSERLIMIT RLIMIT_NOFILE 49152 unlimited 2CIUSERLIMIT RLIMIT_NPROC 5568 5568 2CIUSERLIMIT RLIMIT_RSS unlimited unlimited 2CIUSERLIMIT RLIMIT_STACK 8388608 67104768
There it is, the soft limit of the number of open files is 49,152, that matches (except for a few units) the value 49,146 + 3
2CIUSERLIMIT RLIMIT_NOFILE 49152 unlimited
Other examples
It’s important to be aware of the varied nature of these limits, depending on the process. For instance, in the shell process, the open files limit is 256 as can be checked by running either one of the following commands
franciscoalvarez@franciscos out % getconf OPEN_MAX 256 franciscoalvarez@franciscos out % ulimit -n 256
As a consequence, any new process started from the shell will inherit this value (unless the process itself changes it as Java does). This can be seen by running this C version of the Java program:
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <stdio.h>
int main(int argc, char const *argv[])
{
if (argc < 2 || strcmp(argv[1], "--help") == 0)
{
printf("USAGE: %s num_files \n", argv[0]);
exit(EXIT_FAILURE);
}
size_t count = atoi(argv[1]);
size_t i;
for (i = 0; i < count; i++)
{
char file_name[20];
sprintf(file_name, "f_%03zu", i);
int inputFd = open(file_name, O_CREAT | O_RDONLY, S_IRUSR | S_IWUSR);
if (inputFd == -1)
{
printf("error while opening file %s\n", file_name);
perror("");
break;
}
}
exit(EXIT_SUCCESS);
}
franciscoalvarez@franciscos out % ./file_desc_limit 300 error while opening file f_253 Too many open files
Where 253+3 equals the 256 limit established by the shell.
However, if the same program is run from the terminal of Intellij, the result is very different.
franciscoalvarez@franciscos out % ./file_desc_limit 11000 error while opening file myfiles/f_10237 Too many open files
Here’s Intellij’s limit of open files:
franciscoalvarez@franciscos out % getconf OPEN_MAX 10240
And if you are curious, this is the result when using VSCode’s terminal:
franciscoalvarez@franciscos out % getconf OPEN_MAX 10496
What’s more, the open files limit can be changed in the shell with the command
ulimit -n <integer>
Now we see why it made sense for the Java process (and any other process) to set their own limits, so that they do not depend on the values set in the environment where they are executed. With that in mind, we can modify our C program to set its own limit on the amount of open files.
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <stdio.h>
#include <sys/resource.h>
int main(int argc, char const *argv[])
{
if (argc < 2 || strcmp(argv[1], "--help") == 0)
{
printf("USAGE: %s num_files \n", argv[0]);
exit(EXIT_FAILURE);
}
struct rlimit rl = {1000, RLIM_INFINITY};
if(setrlimit(RLIMIT_NOFILE, &rl) == -1){
perror("error setting open files limit");
}
size_t count = atoi(argv[1]);
size_t i;
for (i = 0; i < count; i++)
{
char file_name[20];
sprintf(file_name, "f_%03zu", i);
int inputFd = open(file_name, O_CREAT | O_RDONLY, S_IRUSR | S_IWUSR);
if (inputFd == -1)
{
printf("error while opening file %s\n", file_name);
perror("");
break;
}
}
exit(EXIT_SUCCESS);
}
We have set the limit to 1,000 and therefore, no matters whether we execute it in the shell or in Intellij’s terminal, the result is the same
franciscoalvarez@franciscos out % ./file_desc_limit 5000 error while opening file myfiles/f_997 Too many open files