Problem of freeMemory()?

Hi,

I am trying to use the freeMemory() of Runtime object to determine the available memory in JVM, so I wrote a simple program to allocate some memory, then free them and call GC to notify system to collect them. Ideally, most allocated memory should be collected by GC. However, every time there are always about half of the memory uncollected, and I don't know if it is the problem of freeMemory() or the GC process! Could someone please take a look at the program below and tell me why the result?

Thanks a lot for your help!
import java.util.ArrayList; import java.util.List; public class MemoryTest { /** * @param args */ public static void main(String[] args) { // TODO Auto-generated method stub long size1 = Runtime.getRuntime().freeMemory(); List list = new ArrayList(); for (long i=0;i<100000;i++) { double rand = Math.random()*5000; String str = "Id"+Integer.toString((int)rand); list.add(str); } long size2 = Runtime.getRuntime().freeMemory(); System.out.println("Space used:"+(size1-size2)); list.clear(); list = null; System.gc(); // wait for a few second try { Thread.sleep(200); } catch (Exception e) {} long size3 = Runtime.getRuntime().freeMemory(); System.out.println("Uncovered space:"+(size1-size3)); } }
I was using Eclipse to run the program and the result was like:

Space used:6091568
Uncovered space:3033752

[ added code tags - Jim ]
[ January 18, 2007: Message edited by: Jim Yingst ]

There is no guarantee for GC to empty memory immediatly. I think this could be the reason.
JVM runs garbage collector. You can tell JVM to run GC but there is no gurantee if JVM does it or not.

As previous poster says, you cannot force Java to do a full GC, so results of freeMemory() are not necessarily what you want.

Perhaps you should take a step back and tell us what you really want to achieve? That is, what is the purpose for which you think you need to know the exact free memory?

If it's just for interest, then I'm afraid the answer is that you cannot do what you are trying to do.

If it's to control some aspect of your application's behaviour, then there may be other ways to achieve what you want. For instance, you can use SoftReference to create caches that automatically get flushed when the system is short of memory. Or you can use JVM parameters like -Xmx, -Xms, -XXminHeapFreeRatio and -XXmaxHeapFreeRatio to get the JVM to adapt the heap size automatically to your application's requirements.

Basically, the JVM is in charge of memory allocation and GC. Attempts by programmers to seize control of it usually fail dismally. Instead, you should work with the hooks like the ones mentioned above.

Thanks for your replies, guys! The reason I am doing the test is for my current project, which processes a large amount of data in memory and always ends up with OutOfMemory exception. One scenario is that I have a list of 830000 objects and need to generate a result string from those objects. One approach I was trying to avoid the memory problem is to free the previous processed objects before doing further processing, but it didn't work. That is the reason why I played with the freeMemory() method to see how GC works for my JVM. I have set the parameter as -Xmx2G (my machine has 3G physical memory) but the problem is still there.

Back to the original program, I believe GC has actually started when the main thread is sleeping (If the statement System.gc() is commented out, you will see that no memory has been collected at all in result). It looks to me that one round of GC does not finish all the work, as only half of the allocated memory is collected. Does anybody know how to make GC collect more memory?

Thanks again for your time!

--James

By the way, I also tried using SoftReference and WeakReference in the array list. The result is a little better than to add string directly to the list, but still there are about 1/3 of the allocated memory uncollected.

--James

Originally posted by James Zhang:
Does anybody know how to make GC collect more memory?

When you get an OutOfMemoryError, the GC already is guaranteed to have tried everything in his power to collect memory.

The next thing to do is using a profiler to find out which objects are taking up the memory, and what holds them from being garbage collected.

James: Using freeMemory() as you have above can be very misleading. The problem is that it only tells you the difference between the heap memory used and the heap memory currently allocated. Both these quantities can change - as you use more memory, the JVM can allocate more memory to the heap (up to whatever your -Xmx setting is). It would be more useful for you to measure heap usage like this:
long sizeUsed = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();

[James]: It looks to me that one round of GC does not finish all the work, as only half of the allocated memory is collected. Does anybody know how to make GC collect more memory?

Although your method for measuring memory usage is unreliable, it is true that System.gc() frequently does not do as much work as it could do when you call it. One simple workaround is to simply call it multiple times. In testing this in the past, I've sometimes seen it take at least four consecutive calls before the memory usage stops changing. For memory testing, typically call it 10 times in a row to be safe:
public static long memoryUsed() { // scrub real hard: for (int i = 0; i < 10; i++) System.gc(); return Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(); }
It should be noted that this is still not guaranteed to free all available memory. Fundamentally you never will get a guarantee of that. But this works well enough, if you don't mind the fact that it's horribly, horribly inefficient and should never ever be used in production code. Generally, modern GC works much better if you don't ever call System.gc() yourself. It's been optimized to run on its own, and if you interrupt its routine with System.gc() you usually screw up its efficiency considerably. So this is fine for testing purposes (as long as you're not testing performance), but a bad idea otherwise.

I fully agree with all of Ilja's last post. Rather than studying how well System.gc() does or does not work, focus on figuring out what's in memory when the OutOfMemoryError is thrown. One part of this is: when is the OOME thrown? If you're adding a bunch of data to a List, and the OOME happens before you've had a chance to clear the list at the end, then the problem is probably just that you're putting too much data in the List. Do you really need to have one huge List like that, or can you modify the processing to eliminate the List and just process one item at a time (allowing each already-processed item to be CG-ed when convenient)?

Also, have you tried simply increasing the maximum memory used by your JVM, using the -Xmx switch? See the documentation for java options for details.
[ January 18, 2007: Message edited by: Jim Yingst ]

Thanks for your valuable comments, Jim and Ilja! I I really appreciate all the helps! We have already used -Xmx2G when starting JVM but seems not helping too much. As your guys pointed out, it is not a good idea to run GC in the codes. My program is just a testing for the GC functionality because I was suspecting that the GC process was not started in our application. Probably a better way is to use tools like JProbe to analyze the memory usage for the application, which is what I am trying to do next.

Again thanks guys!

--James

Originally posted by Jim Yingst:

It should be noted that this is still not guaranteed to free all available memory. Fundamentally you never will get a guarantee of that.

Just to make this totally clear: there is one important exception to this rule. It *is* guaranteed that all available memory is freed before an OutOfMemoryError is thrown.

Well, something like that. Skipping past a semantic discussion of what exactly the specs say (scattered in a number of different places) in practice it seems that the actual behavior is: an OOME will not be thrown unless sufficient memory could not be freed. In some cases the JVM can determine up front that a request would require more memory than it could possibly allocate - e.g. an array that requires more memory than the JVM could possibly have. In such a situation, it may not bother actually freeing up other memory first; it can simply throw OOME right away.

Anyway, it does seem that OOME is not thrown if there is any possible way that sufficient memory could have been allocated. So calling System.gc() still offers nothing that you wouldn't get by default, in terms of freeing up memory when you need it.

Agreed. If the problem is that you are getting OutOfMemoryError, then GC is 100% NOT the cause of the problem. Java GC does work and you NEVER need to invoke it explicitly to prevent OutOfMemoryError.

You either have a memory leak in your program or you need more heap. If, as you say, your max heap is already 2GB, then you either have an incredibly memory-intensive application needing specialised (64-bit) hardware, or (more likely) you have a leak.

Originally posted by Jim Yingst:
Well, something like that.

Good point, thanks for the clarification!