Newtonsoft Json OutOfMemoryException
I was trying to log some stuff (a lot of stuff) and I noticed that my memory went to the roof (16GB in a few seconds), then an OutOfMemoryException was thrown. I've finally narrowed it down to the JSON serializer from Newtonsoft.First of all, some introduction on how to serialize any object into JSON: whether you use JsonConvert.SerializeObject(yourObject,new JsonSerializerSettings { <settings> }) or new JsonSerializer { <settings> }.Serialize(writer, object) (where <settings> are some properties set via the object initializer syntax) you will need to consider these properties:
We will use these classes to test the results:
class Parent
{
public string Name { get; set; }
public Child Child1 { get; set; }
public Child Child2 { get; set; }
public Child[] Children { get; set; }
public Parent Self { get; internal set; }
}
class Child
{
public string Name { get; set; }
public Parent Parent { get; internal set; }
}
For this piece of code:
JsonConvert.SerializeObject(new Child { Name = "other child" }, settings)you will get either
{"Name":"other child","Parent":null}or {
"Name": "other child",
"Parent": null
}based on whether we use Formatting.None or Formatting.Indented. The other properties do not affect the serialization (yet).Let's set up the following objects:
var child = new Child
{
Name = "Child name"
};
var parent = new Parent
{
Name = "Parent name",
Child1 = child,
Child2 = child,
Children = new[]
{
child, new Child { Name = "other child"}
}
};
parent.Self = parent;
child.Parent = parent;
As you can see, not only does parent have multiple references to child and one to himself, but the child also references the parent. If we try the code
JsonConvert.SerializeObject(parent, settings)we will get an exception Newtonsoft.Json.JsonSerializationException: 'Self referencing loop detected for property 'Parent' with type 'Parent'. Path 'Child1'.'. In order to avoid this, we can use ReferenceLoopHandling.Ignore, which tells the serializer to ignore circular references. Here is the output when using
var settings = new JsonSerializerSettings
{
Formatting = Formatting.Indented,
ReferenceLoopHandling=ReferenceLoopHandling.Ignore
};
{
"Name": "Parent name",
"Child1": {
"Name": "Child name"
},
"Child2": {
"Name": "Child name"
},
"Children": [
{
"Name": "Child name"
},
{
"Name": "other child",
"Parent": null
}
]
}
If we add NullValueHandling.Ignore we get
{
"Name": "Parent name",
"Child1": {
"Name": "Child name"
},
"Child2": {
"Name": "Child name"
},
"Children": [
{
"Name": "Child name"
},
{
"Name": "other child"
}
]
} (the "Parent": null bit is now gone)The default for the ReferenceLoopHandling property is ReferenceLoopHandling.Error, which throws the serialization exception above, but we can also use ReferenceLoopHandling.Serialize besides Error and Ignore. In that case we get a System.StackOverflowException: 'Exception of type 'System.StackOverflowException' was thrown.' as it tries to serialize at infinitum.
PreserveReferencesHandling is rather interesting. It creates extra properties for objects like $id, $ref or $values and then uses those to define objects that are circularly referenced. Let's use this configuration:
var settings = new JsonSerializerSettings
{
Formatting = Formatting.Indented,
NullValueHandling = NullValueHandling.Ignore,
PreserveReferencesHandling = PreserveReferencesHandling.Objects
};
Then the result will be
{
"$id": "1",
"Name": "Parent name",
"Child1": {
"$id": "2",
"Name": "Child name",
"Parent": {
"$ref": "1"
}
},
"Child2": {
"$ref": "2"
},
"Children": [
{
"$ref": "2"
},
{
"$id": "3",
"Name": "other child"
}
],
"Self": {
"$ref": "1"
}
}
Let's try PreserveReferencesHandling.Arrays:
var settings = new JsonSerializerSettings
{
Formatting = Formatting.Indented,
NullValueHandling = NullValueHandling.Ignore,
ReferenceLoopHandling=ReferenceLoopHandling.Ignore,
PreserveReferencesHandling = PreserveReferencesHandling.Arrays
};
The result will then be
{
"Name": "Parent name",
"Child1": {
"Name": "Child name"
},
"Child2": {
"Name": "Child name"
},
"Children": {
"$id": "1",
"$values": [
{
"Name": "Child name"
},
{
"Name": "other child"
}
]
}
}which annoyingly adds an $id to the Children array. There is one more possible value, PreserveReferencesHandling.All, which causes this output: {
"$id": "1",
"Name": "Parent name",
"Child1": {
"$id": "2",
"Name": "Child name",
"Parent": {
"$ref": "1"
}
},
"Child2": {
"$ref": "2"
},
"Children": {
"$id": "3",
"$values": [
{
"$ref": "2"
},
{
"$id": "4",
"Name": "other child"
}
]
},
"Self": {
"$ref": "1"
}
}I personally recommend using PreserveReferencesHandling.Objects, which doesn't need setting the ReferenceLoopHandling property at all. Unfortunately, it adds an $id to every object, even if it is not circularly defined. However, it creates an object that can be safely deserialized back into the original, but if you just want a quick and dirty output of the data in an object, use ReferenceLoopHandling.Ignore with NullValueHandling.Ignore. Note that object references cannot be preserved when a value is set via a non-default constructor such as types that implement ISerializable.
Warning, though, this is still not enough! In my logging code I had used ReferenceLoopHandling.Ignore and the exception was quite different, an OutOfMemoryException. It seems that even with circular references checked, JsonSerializer will messes up some times.
The culprits? Task<T> (or async lambdas send as parameters) and an Entity Framework context object. The solution I employed was to check the type of the objects I send to the serializer and, if any of the offending types, replace them with the full names of their types.
Hope it helps!
