java.net.SocketException: Too many open files

[jpollard]

All of them.

Your problem appears to be that the parent thread opens all files... then starts all
threads - this will duplicate any open file within each process - and use up all of the
possible file descriptors. If the parent thread is opening them, then any undesired
fid should be closed by the child thread after it starts.

[newbie14]

Dear Jpollard,
Yes I do agree with you I have one which is keep listening once is accepted then run a thread which in it will write to a text file and insert into database. I have already verified and make sure all the closing it done well. Because now I am confuse as I know ulimit-n is for the particular user or proces limit? I would like to know for my application what will be the limit for file descriptors?

[jpollard]

The normal limit is 1024 open files.

This is partially because the select system call can only handle 1024 file descriptors
for event support.

In the normal client/server model, the server child will close file descriptors inherited
from the parent that are not relevent to the child. The parent server, must close
any file descriptors passed to the child thare are not relevent to the parent.

One problem I have seen with threads is that the "child" does not necessarily
terminate when it's processing has completed - instead, it goes into a wait
state for the next event to be serviced. If the child does not explicitly close the
file descriptors, they will remain open while processing the next descriptor. this
causes the thread to accumulate open descriptors until there are no more
available. This means that care must be taken to ensure the cleanup after
the thread has performed its service.

I don't know if the java threads actually terminate when they complete their
processing, or are put into a work/event queue for reuse. Using a work queue allows
for a much faster thread startup than actually terminating a thread and creating
a new one.

[newbie14]

Dear Jpollard,
So based on your experience do you think by increasing the 1024 to a bigger number will it help out? Another thing I notice is that each time I restart my server then it takes some time before I get a high number of Established connection. It goes like first hours is like 100 then next hour become 200 and so on. Is this normal that big number connection can not come in the first hour? Thank you.

[giulix]

I have tested the example code provided. In that example, parent and children share the same log file, handled by log4j. Each single child opens one input file to read the socket input from and one output file to write to. Provided it reads the EOF character from the input stream, it closes the files always. What can happen is that no EOF character is received, thus the files never get closed. The code could be made more robust by, say, waiting for a predetermined amount of time and then closing the files, terminating the connection. It's easy to test it by running the application provided:

java -cp path_to_log4j/log4j-1.2.16.jar:. ThreadTest &

Use

jobs -l

to find out the process id of the background process, and observe the messages that are logged with

tail -f threadtest.log

From other terminals, open several telnet connections to the listening socket with

telnet -e"#" localhost 9876

Enter # on a new line to make telnet switch to command mode and Ctrl-D to send the EOF character. When a new connection is established, the process number of open files is increased by two and a message is logged; when the EOF character is sent along the line, the number is decreased by two and another message is logged . If the EOF is not received, only the initial message (files open) is logged and the number of open files keeps growing until the limit is reached. To test, set the limit to a low value, say 35 or 40 (the limit can be reduced by any user but can only be increased by root), with

ulimit -n 40

before starting the application. Use

lsof -p n |wc -l

, n being the process number output by `jobs -l', to monitor the number of open files

[jpollard]

Originally Posted by newbie14

Dear Jpollard,
So based on your experience do you think by increasing the 1024 to a bigger number will it help out? Another thing I notice is that each time I restart my server then it takes some time before I get a high number of Established connection. It goes like first hours is like 100 then next hour become 200 and so on. Is this normal that big number connection can not come in the first hour? Thank you.

It won't help - the problem depends on how frequently connections are
made, and that is outside the control of the server.

It it increases at a rate of 100 per hour, then that would be 2400 per day,
16800 per week - so to keep it down you would have to restart the
server at a minimum of every day. You still get a failure if the rate increases
to 200 per hour, or gets burst rates of up to 1000.

---------- Post added at 07:46 AM CDT ---------- Previous post was at 07:30 AM CDT ----------

I have tested the example code provided. In that example, parent and children share the same log file, handled by log4j. Each single child opens one input file to read the socket input from and one output file to write to. Provided it reads the EOF character from the input stream, it closes the files always. What can happen is that no EOF character is received, thus the files never get closed. The code could be made more robust by, say, waiting for a predetermined amount of time and then closing the files, terminating the connection. It's easy to test it by running the application provided:

Good point - Another thing I noticed is that all of the exception handlers ignore
the file. They do nothing. If the connection is aborted by the client, does the
server recieve an EOF?, or does it get a "connection reset" exception?

By the way.... the phrase:

(m=r.read()) != 0)

Isn't really looking for an end of file (my java is very rusty, but from C eof is a -1).
This would seem to hang the thread, causing the number of threads to increase.

I'm used to explicitly looking for an end of file, as a 0 value may be a valid part
of the data.

[giulix]

Yes, you're right. I was referring to my example attached to post #21 of this thread. I think the OP is presently running a modified version of the code originally posted by him/her... sorry for the confusion

[jpollard]

I like that one much better - though even it seems to ignore some errors. For
instance:

        } finally {
            try {
                if(log.isDebugEnabled())
                    log.debug("Closing input/output files");
                if(r != null) {
                    r.close();
                    r = null;
                }
                if(fw != null) {
                    fw.flush();
                    fw.close();
                    fw = null;
                }
            } catch(IOException e) {
                log.fatal(e.getMessage(), e);
            }
        }

If the last "try" block here catches an exception based on the "r.close();", does the
file "fw" get closed?

[stevea]

If you want to see the files opened by the process use 'lsof' and -p or grep the process name or pid. That should tell you the number of open files/sockets and the specific files/sockets.

[giulix]

Originally Posted by jpollard

If the last "try" block here catches an exception based on the "r.close();", does the
file "fw" get closed?

Good point. There should be 2 try blocks, one for each file. If I remember correctly, the finally is guaranteed to execute all included statements, even if an exception is raised.

---------- Post added at 03:03 PM CDT ---------- Previous post was at 02:39 PM CDT ----------

Confirmed (tested).

[newbie14]

Dear Jpollard,
Ok like you suggest I wont be incerasing the value as it wont be helping me either rite. About the phrase I have already change it to (m=r.read()) != -1) that is what my current system is running on. Ok I will run this statement too.
if(r != null) {
r.close();
r = null;
}
if(fw != null) {
fw.flush();
fw.close();
fw = null;
}

---------- Post added at 08:30 AM CDT ---------- Previous post was at 08:29 AM CDT ----------

Dear Giulix,
I dont understand when you say "There should be 2 try blocks, one for each file. " Which one do you meant? Thank you.

---------- Post added at 08:36 AM CDT ---------- Previous post was at 08:30 AM CDT ----------

Dear Stevea,
Ok I run like this lsof -w -p 20480 it give me a list non stop. How if I just want to get the total? Another thing what is the command for me to stop it from the list keep going and where I do paging? Thank you.

[jpollard]

The code in question should have try blocks around both files

Something like:

        } finally {
            try {
                if(log.isDebugEnabled())
                    log.debug("Closing input/output files");
                if(r != null) {
                    r.close();
                    r = null;
                }
              } catch(IOException e) {
                      log.fatal(e.getMessage(), e);
              }
              try { 
                if(fw != null) {
                    fw.flush();
                    fw.close();
                    fw = null;
                }
               } catch(IOException e) {
                log.fatal(e.getMessage(), e);
            }
        }

[giulix]

Ah, the beauty of open source and peer review...

[giulix]

Originally Posted by newbie14

For the log file can I have separate log files on daily basis?

I found a working example here (velocityreviews.com). Just change the name of the file to whatever you like...

<param name="File" value="whateveryoulike.log" />

Name the XML file log4j.xml

[newbie14]

Dear Giulix & Jpollard,
I would like to share with you all I just found out and I guess that is the problem of mine. The problem is that my device once it is connected it does not break the connection. So now I am going to use setTimeout. Will update you all with my progress ok. Anyway I will still do the fundamental testing as suggested earlier.