Issue
I'm trying to read files from a folder and its subfolders but after reading certain files, it throws
java.io.IOException: Too many open files
I am having more than 80k files which I have to read. I am unable to close files. Maybe I am having a wrong approach. Please guide me to the right path. Thanks Here is my Logcat
Caused by: java.io.IOException: Too many open files
at java.base/sun.nio.ch.IOUtil.makePipe(Native Method)
at java.base/sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:83)
at java.base/sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:167)
Code
try (Stream<Path> filePathStream=Files.walk(Paths.get(FOLDER_PATH))) {
filePathStream.forEach(filePath -> {
if (Files.isRegularFile(filePath)) {
System.out.println(filePath.toFile().getAbsoluteFile().toString());
try {
String content = Files.readString(filePath);
System.out.println(filePath.getFileName());
JSONObject jsonObject1 = new JSONObject(content);
// System.out.println(file.getName());
HashMap yourHashMap1 = new Gson().fromJson(jsonObject1.toString(), HashMap.class);
request.add(new Config().client().prepareIndex("recipe_json", "default").setSource(yourHashMap1));
} catch (IOException ignore) {
}
Solution
There are a couple of possible reasons for the problem:
You may simply be opening too many files at the same time. I don't see how, because
Files.readString(filePath)
should not leak. However, I notice that you are squashingIOException
, and that might be throwing away some significant evidence.You may be opening files ... or pipes or sockets somewhere else. Maybe here:
request.add(new Config().client() .prepareIndex("recipe_json", "default") .setSource(yourHashMap1));
I am using Elasticsearch Bulk API and adding the content of the file to bulk at this line of code. does that break the rule?
Probably yes. Check the API documentation. I expect that you need to "close" or "disconnnect" the client. If you don't, that is liable to leak sockets.
Answered By - Stephen C
Answer Checked By - Dawn Plyler (JavaFixing Volunteer)