OutOfMemoryException when working with StringBuilders and strings
One of our sites started exibiting some strange behaviour when adding a lot of strings to a StringBuilder. The error was intermittent (best kind there is) and the trace showed the error originating from the StringBuilder class and then from within the String class itself (from an external method).
On the web other people noticed this and one possible explanation was that strings need to be contiguous and thus they cannot grow bigger than the largest free contiguous memory block. Remember defragmentation? Heh.
In order to test this, I added a letter to a StringBuilder, then added itself to it for a few times to see where it breaks. It broke at a size of 2^27 (134,217,728 = 0x8000000)! At first I thought it was a StringBuilder bug, but getting sb.ToString() then adding to it a letter resulted in an error. Then I thought maybe it is a default size that, for whatever reason, is considered the maximum string size. That would suck. I don't usually need 134Mb of string, but what if I do? I have 2.5Gb of memory in this computer. But no, people reported the same problem but with other powers of 2, like 29 and on a Vista computer I got 2^28.
Is it from the fragmentation of memory? Right now I have only 1.8Gb in use. I rather doubt that in the 700Mb of space left there is only one 134Mb contiguous memory block. But I cannot prove it one way or another. My guess is that there is another mechanism that actually interferes with this and effectively limits the maximum string size.
But what does that mean, anyway? For practical purposes it means that you have to plan not to have strings that big in your applications. Creating a custom string builder might work, but it would have to use some other methods than strings.
I have tried the same thing with byte arrays. While declaring a byte array of 2^28 was possible (probably also because a string uses Unicode characters internally and thus takes twice the memory) writing it to a MemoryStream resulted in an error.
So watch these things out, try to always keep single variable blocks to manageable sizes.
On the web other people noticed this and one possible explanation was that strings need to be contiguous and thus they cannot grow bigger than the largest free contiguous memory block. Remember defragmentation? Heh.
In order to test this, I added a letter to a StringBuilder, then added itself to it for a few times to see where it breaks. It broke at a size of 2^27 (134,217,728 = 0x8000000)! At first I thought it was a StringBuilder bug, but getting sb.ToString() then adding to it a letter resulted in an error. Then I thought maybe it is a default size that, for whatever reason, is considered the maximum string size. That would suck. I don't usually need 134Mb of string, but what if I do? I have 2.5Gb of memory in this computer. But no, people reported the same problem but with other powers of 2, like 29 and on a Vista computer I got 2^28.
Is it from the fragmentation of memory? Right now I have only 1.8Gb in use. I rather doubt that in the 700Mb of space left there is only one 134Mb contiguous memory block. But I cannot prove it one way or another. My guess is that there is another mechanism that actually interferes with this and effectively limits the maximum string size.
But what does that mean, anyway? For practical purposes it means that you have to plan not to have strings that big in your applications. Creating a custom string builder might work, but it would have to use some other methods than strings.
I have tried the same thing with byte arrays. While declaring a byte array of 2^28 was possible (probably also because a string uses Unicode characters internally and thus takes twice the memory) writing it to a MemoryStream resulted in an error.
So watch these things out, try to always keep single variable blocks to manageable sizes.
Comments
Be the first to post a comment