Java

J2EE, HP/UX threads, and OutOfMemoryError

Posted on by rkischuk in Java

We recently experienced regular java.lang.OutOfMemoryError exceptions on HP/UX any time our J2EE application came under significant load. We have since determined that it is due to a native issue with threading.

The HP/UX kernel has a parameter named max_thread_proc, which sets the maximum number of threads that a process can have. Apparently the default is 64. Considering that JBoss starts up with almost 60 threads when it’s doing nothing, this cap of 64 is quite low.

There are 2 options to fix the problem. One is to pass the -green parameter to the JVM when you start your app server. This causes the JVM to use its own green threading model rather than that of the operating system. The advantage of this approach is that it can be tested and implemented quickly. The disadvantage is that Java’s green threading model isn’t as sophisticated as the native threading model of the underlying operating system (Sun has said as much), so you may pay a performance penalty for this.

The other option is to recompile the HP/UX kernel with a higher max_thread_proc value. We set it to 3000, just to be safe, although we haven’t exceeded 250 yet. The biggest disadvantage to this is that you have to recompile the kernel. The other disadvantage is that you haven’t completely avoided the problem, you’ve only moved the threshold at which it will occur. The advantage is that you get the performance, monitoring, and tuning of using the native thread model.

I don’t think that Linux users need to worry. I believe that Linux’s threading model is native green threading. Note that this is still different from the JVM’s green threading, so -green should still not be used in this case either – Linux will handle it for you.

Advertisement

5 thoughts on “J2EE, HP/UX threads, and OutOfMemoryError”

  1. How did you determine that it was a threading issue? I’m curious because we have a handful of HPUX machines here that were once production and are now development test machines. Every now and then, Websphere will stop working and I see the OutOfMemory error in the SystemErr.log file. I have to restart WAS to get it working again. I have full admin rights to the box, but I have no idea how to recompile the kernal for it. Any HPUX hints and tips sites out there?

  2. It was largely a process of elimination. an OutOfMemory error can actually be one of a few things: it can be that the JVM is out of memory – this is configured with the -Xmx parameter. The OS itself may have run out of memory. In either of these 2 cases, check for a memory leak.

    It may be that you’ve run out of file descriptors – once again, this can be at the OS or JVM level, and once again, check for a leak.

    Neither of these things were our problem, so we did some more digging, and watched the thread count in a system utility. As soon as it hit 64, the app stopped responding.

    I can’t exactly suggest tools to you, because we have sysadmins who were instrumental in chasing down the problem using their sets of tools.

  3. Thanks for the tips. Unfortunately our sysadmins are too busy with production issues to look at something minor like this. I’ll see if I can’t try and find some answers on my own. I suppose it’s the best way to learn something…

  4. Understood – our issue was a production issue, so it was high priority for us. If the target machines are development/test, why not just use green threading? Threading performance won’t be critical, and you won’t spend as much time chasing an evasive answer.

  5. Note: Java 1.4 doesn’t support green threading on HPUX
    If you want to try the green threads before modifying the kernel you need to run your app under 1.3 (if possible)

Comments are closed.