For the last few weeks on one of our ColdFusion 8 servers I have been noticing an increase in performance issues. Requests would regularly start taking longer and timing out and it would often be happening to all the running requests and not just some. Strangely this also seemed to happen whenever I modified a security sandbox, which would usually take a long time to process the request and would also cause all other requests to slowdown and timeout as above.
One common factor I noticed while stack tracing these requests with FusionReactor and the server monitor was that they all seemed to be using Fusebox and there was a lot of class loading going on.
Now I know that frameworks like Fusebox and ModelGlue can be slow to initialise the first time as they have a lot of classes to load, but this shouldn't happen again unless an application is re-initialised or CF is restarted.
I had Charlie Arehart helping me look at this issue and he reminded me of the problem with Java 6, which I had totally forgotten about. For all the other performance improvements and increased functionality in Java 6, it introduced a bug in the class loader that causes substantially slower class loading. You can see a discussion of the problem on Sun's forums.
This bug will affect any application using a lot of CFC's, especially those using frameworks such as fusebox, ModelGlue, Transfer etc, due to the large number of classes that must be loaded as a result. So for those of you not caching your CFC's, perhaps it time to start doing so.
This lead us to the conclusion that whenever you make any changes to a security sandbox this causes all existing classes to be reloaded, which seemed to be confirmed by the stack traces which did show a lot of class loading happening in the slow running processes.
As the aforementioned bug is supposed to have been fixed in the current release of Java 6, I decided to go with Charlie's suggestion and give this a try and see if it resolved the problem. So I downloaded and installed the JDK 6 Update 12, set ColdFusion to use this version and so far I have not been able to repeat the previous issues when modifying security sandboxes, so it seems as though this solution has worked. It is however early days as I only made the change last night, so time will tell if this gives an overall performance boost to ColdFusion in general.
Updating ColdFusion to use a newer version of Java is very straight forward, so here are the steps for anyone interested in doing so.
Download the latest version of the JDK (or whatever version you need) from http://java.sun.com/javase/downloads/index.jsp
Now install this on your server in your desired location, but don't forget that you may need to setup additional permissions if you are not running ColdFusion in the standard configuration and have it running under anything other than SYSTEM.
now you need to edit your jvm.config file and modify the java.home path to point to the newly installed JDK.
On a standard installation this file can be found in c:\coldfusion8\runtime\bin\jvm.config, if you are using a J2EE or multi-server installation then your path will be different.
Comment out the existing java.home by adding a # to the start of the line and then enter your new path like so:-
#java.home=C:/ColdFusion8/runtime/jrejava.home=C:/Program Files/Java/jdk1.6.0_12/jre
Please note the path is using back slashes and not the default forward slashes that you will get if you copy and paste the path from windows explorer. This is required or ColdFusion will not start. If you are running multiple instances of ColdFusion with each using their own JVM.config then you will need to make this change in each jvm.config file.
Now simply restart ColdFusion, then login to your cfadministrator and go to the system information page, where it shows which version of Java is being used, which should now reflect your changes.
ColdFusion 8 Server Monitor
I also wanted to mention that while trying to diagnose these issues I tried using the built in server monitor, which unfortunately caused more of a hindrance than a help. It seems that enabling the Profiling and Memory Tracking on a live production server may not a good idea and could well bring it to its knees within minutes. While these options were enabled the JRUN memory usage started to climb and continued to climb until the max 1024mb had been consumed, at which point CF will stop responding. I also noted the memory tracking will also incorrectly report the memory usage of complex FuseBox variables, stating that they are several TerraBytes in size, so this seems to imply that again the problems were related to frameworks and class loading.
I have however enabled the server monitor briefly since updating Java and the memory consumption problem does seem to have gone but the incorrect reporting of memory usage in application and request scopes is still there.
Sadly I have never been able to get the "Sessions by memory usage" or "CF Threads by memory usage" to work, they have always been blank, so whether these have the same issues I do not know.
Mar 20, 2009 at 12:26 PM Also worth installing the Cumulative Hot Fix 2 for ColdFusion 8.0.1 http://kb.adobe.com/selfservice/viewContent.do?externalId=kb403781 because of:
"Fix for memory leaks with CFCs stored in memory scopes."
This pretty much applies to any application framework you're using!
Mar 20, 2009 at 12:55 PM Already installed John :-)
Mar 23, 2009 at 12:37 AM Thanks for sharing the observation with everyone, Russ, and glad to have helped. For those who don't know, this is what I do for a living, helping solve problems related to CF and CF server processing. If anyone might ever hit a problem that stumps them, feel free to get in touch. More at carehart.org/consulting/.
Mar 24, 2009 at 8:42 PM I am not a developer, but work as a sysadmin for a development house. We have not had much success in the past with changing JVM versions. Here is some we were running CF8 (patch 3) with 1.6.0_01 (the defualt JVM level) and tried 1.6.0_10 and this caused requests to slowly bank up and queue to the point of crashing our webservers. We reverted to back to default JVM and all run ok again. We then upgraded to CF8.1 (patch 2, JVM1.6.0_04) and after reading this blog decided to try J1.6.0_12. But again immediately our requests begin to slowly bank up and crash our webservers, reverted back to 1.6.0_04 and all ok. We are running default configs on all servers. Thought this may be helpful to others
Mar 25, 2009 at 12:01 AM Joshua, here's a suggestion: you refer to requests "banking up", as if you think they're hung up. Here's some info that may help you get further toward solving your problem.
First, do you have any information to confirm that, other than requests not running? I mean, are you viewing either CFSTAT, perfmon stats, or the JRun Metrics to tell how many requests are running (and are queued)?
Beyond that, you'll want to know what those hung requests are, and what they're doing. There are solutions that can help, for free.
You mention you're running CF8. Is that Enterprise? If so, use the CF 8 Server Monitor. If one was on 6 or 7, they could use FusionReactor or Seefusion, both of which have free trials.
Any of these tools can tell you exactly what requests are running, and they also all let you drill into each request to see exactly what line of code it's running. It need no longer be a mystery of what's backing up. :-)
Now, FWIW, with Russ's challenge, he wasn't seeing requests really "backup". They weren't "hung". They would all be running whenever we looked (he was using FusionReactor). But requests were indeed taking a long time to run, which is what led to him trying what solved things for him.
It sounds like your challenge is different and perhaps off-topic, but I'm sure other readers would also look forward to hearing what you find.
Mar 25, 2009 at 3:19 AM Sorry i should have been more specific - we use FusionReactor and the requests do backup and never complete. Without the JVM update, they are fine. Just posting as a word of warning for others going to try it to be careful. Thanks
Mar 25, 2009 at 10:26 AM Josh,
Have you tried stack tracing your hanging requests to see where exactly they are hanging?
Apr 1, 2009 at 7:52 PM Joshua, did you ever try the stack tracing that I and Russ were referring you to?
Apr 21, 2009 at 11:43 PM No, unfortunately we have not had the time to troubleshoot the issue in great depth, its a pity FR does not log to DB so history is not lost on a service hang.
Apr 22, 2009 at 6:43 PM Joshua, you say, "its a pity FR does not log to DB so history is not lost on a service hang." It sounds like you're lamenting that FR's logging, which is to a file, can grow to a size that the log file fills (defaults to 10meg in FR 3), and that the rotation (which defaults to 5) might cause you to lose older logs (perhaps even within a day for some of the logs).
Well, did you realize that the log file size and rotation can be changed? In FR 2, the default was just 1meg (yikes). I lobbied for the increase to 10meg. Even then, 10meg times 5 rotations is only 50 meg, and as there are 5 kinds of logs in FR, that's a total of 250meg. I think most could afford to let the logs grow still larger than that. :-)
So I recommend people change them to 100 meg. Then it's 500 meg * 5 kinds of logs, for a total of 2.5 gig. Most can afford that, and then you rarely lose logs. Even then, if that sounds like "too much" for logs, only some really pump large volumes of data quickly (primarily the jdbc log, and maybe the request log), so maybe one could keep the FR and CP logs smaller if pressed for space.
Hope that's helpful.
BTW, just to be clear, the stack trace (that we suggested you use) is never logged, but if you enable Crash Protection notification and set the settings to send you email, you get the stack trace in an email.