In this post I will try to bring to your attention something that will probably change the world significantly. In 1909, German chemist Fritz Haber successfully fixed atmospheric nitrogen as ammonia in a laboratory and five years later a research team from BASF, led by Carl Bosch, developed the first industrial-scale application of the Haber process, sometimes called the Haber-Bosch process. Ammonia is extremely useful for many applications, the least of each is gunpowder and explosives and one of the most important is fertilizers. Without the Haber-Bosch process we probably wouldn't have the Green Revolution.
So today I found this article in Ars Technica that says that Researchers have developed a method to produce ammonia starting only with air and water. Not only is it more energy efficient than the century-old Haber-Bosch process that’s currently in use, but it’s also greener. The article goes on to say that almost 2% of the entire world energy is used to create ammonia; making the process more efficient is great! But I have to say that this is probably just the tip of the iceberg. Lowering the production cost of such a basic article will ripple throughout many industries, lead to innovation or the possibility to use some old innovation that until now was unfeasible.
I am not a chemist, so my enthusiasm may be way off-base, but my gut feeling is that this improvement on a century old process will have a great and positive effect.
I have been working on a REST API lately and, while using Entity Framework or some other similar framework to abstract the database is certainly possible, I wanted to control every aspect of the implementation. I know, reinventing wheels, but this is how one learns. One of the most annoying bits was trying to translate some complex object from JSON to the (two dimensional) database relational tables. This post will explore my attempts and the solutions I have found.
My first attempt was straightforward: just send all types as DataTables, with some extra property to define identity and parent entity. This relies on the Microsoft Server SQL mechanism that allows sending of table variables to stored procedures. But this approach has several downsides. One of them is that in order to send a datatable to SQL you need... DataTables. As I have pointed out in several blog posts, the DataTable object is slow, and sometimes downright buggy. Even if I didn't care about performance that much, in order for SQL to receive the content of the DataTable one must create corresponding User Defined Types on the database side. Working with UDTs is very difficult for several reasons: you cannot alter a UDT (unless employing some SQL voodoo that changes system tables), you can only drop it and recreate it. This does not work if you use the UDT anywhere, so a lot of renaming needs to be done. Even if you automate the process, it's still very annoying. Then the UDT definition has to be an exact duplicate of the DataTable definition. Move some columns around and it fails. Debugging is also made difficult by the fact that the SQL profiler does not see the content of table variables when sending them to the server.
Long story short, I was looking for alternatives and I found XML. Now, you might think that this leads to a simple, and maybe even obvious, solution. But it's not that easy. Imagine that you send a list of objects in an XML. Each object is represented by an XML element and each property by a child element. In order to get the value of a property you need to do iterate through all the nodes, for each node find the properties, for each property find the one element that defines it, then get the attribute value or content of the property, all while making sure you select everything in a table. It's not that easy.
The solution I found, which simplifies the SQL code (and hopefully brings some well needed performance to the table) is to serialize the objects in a way that makes the selection simple enough. Here is an example: I have a Configuration object with an Id and a Name that also has a property called Servers, containing Server objects having an Id and a Url. Here is an example of XML serialization from the DataContractSerializer:
;WITH XMLNAMESPACES(DEFAULT'http://schemas.datacontract.org/2004/07/SerializationTest') SELECT T.Item.value('(Id/text())[1]','INT') as Id, T.Item.value('(Name/text())[1]','NVARCHAR(100)') as Name FROM @Xml.nodes('//Configuration') as T(Item)
;WITH XMLNAMESPACES(DEFAULT'http://schemas.datacontract.org/2004/07/SerializationTest') SELECT T.Item.value('(Id/text())[1]','INT') as Id, T.Item.value('(Url/text())[1]','NVARCHAR(100)') as Url FROM @Xml.nodes('//Configuration/Servers/Server') as T(Item)
This works, but look at that code. In my case, the situation was worse, the object I was using was a wrapper which implemented IDictionary<string,object> and, even if it did implement ISerializable, both XmlSerializer and DataContractSerializer use the dictionary as their data and in the end I get ugly key elements and value elements that are even harder to get to and, I suppose, more inefficient to parse. Therefore I found the solution in IXmlSerializable, (yet) another serialization interface used exclusively by XML serializer classes. If every simple value would be saved as an attribute and every complex object in an element, then this could be the SQL code:
;WITH XMLNAMESPACES(DEFAULT'http://schemas.datacontract.org/2004/07/SerializationTest') SELECT T.Item.value('@Id','INT') as Id, T.Item.value('@Name','NVARCHAR(100)') as Name FROM @Xml.nodes('//Configuration/Config') as T(Item)
;WITH XMLNAMESPACES(DEFAULT'http://schemas.datacontract.org/2004/07/SerializationTest') SELECT T.Item.value('@Id','INT') as Id, T.Item.value('@Url','NVARCHAR(100)') as Url FROM @Xml.nodes('//Configuration/Config/Servers/List/Server') as T(Item)
Much easier to read and hopefully to parse.
I am not going to write here about the actual implementation of IXmlSerializable. There are plenty of tutorials on the Internet about that. It's not pretty, :) but not too difficult, either.
What was the purpose of this exercise? Now I can send a complex object to SQL in a single query, making inserts and updates simple and not requiring at a call for each instance of each type of complex object. Now, is it fast? I have no idea. Certainly if performance is needed, perhaps the UDT/DataTable approach is faster. However you will have to define a type for each type that you send as a DataTable to a stored procedure. An alternative can be a binary serializer and a CLR SQL function that translates it into tables. However, in my project I need to easily implement very custom API methods and to control every aspect, including tracing and profiling the various SQL calls. I believe the customized IXmlSerializable/XML in SQL approach is a reasonable one.
I am going to tell you about how I worked on an object that can wrap any object, serialize it, deserialize it, send it to a database, all efficiently and generically. But first, let me start with the beginning: you need a way to efficiently set/get values of properties in objects of different types. That's where TypeCache comes in. I am sure not everything is perfect with the class, but hopefully it will put you on the right track.
I will start with some of the code, the class that does everything. Something will be missing, though.
/// <summary> /// It caches the property setters and getters for the public properties of a type /// </summary> publicclass TypeCache { privatestatic ConcurrentDictionary<Type, TypeCache> _typeCacheDict = new ConcurrentDictionary<Type, TypeCache>();
publicstatic TypeCache Get(Type type) { TypeCache cache; if (!_typeCacheDict.TryGetValue(type, out cache)) { cache = new TypeCache(type); _typeCacheDict[type] = cache; } return cache; }
private TypeCache(Type type) { Type = type; Setters = new ConcurrentDictionary<string, Action<object, object>>(); Getters = new ConcurrentDictionary<string, Func<object, object>>(); var props = type.GetProperties(BindingFlags.Public | BindingFlags.Instance); Properties = new ConcurrentDictionary<string, PropertyInfo>(); foreach (var prop in props) { if (prop.CanRead) { var objGetter = prop.GetValueGetter(); Getters[prop.Name] = objGetter.Compile(); } if (prop.CanWrite) { var objSetter = prop.GetValueSetter(); Setters[prop.Name] = objSetter.Compile(); } Properties[prop.Name] = prop; } }
public Type Type { get; privateset; } public ConcurrentDictionary<string, Action<object, object>> Setters { get; privateset; } public ConcurrentDictionary<string, Func<object, object>> Getters { get; privateset; } public ConcurrentDictionary<string, PropertyInfo> Properties { get; privateset; } }
publicstaticclass TypeCacheExtensions { /// <summary> /// Set the value of a property by name /// </summary> /// <param name="cache"></param> /// <param name="item"></param> /// <param name="key"></param> /// <param name="value"></param> publicstaticvoid Set<T>(this TypeCache cache, object item, string key, T value) { if (cache == null || item == null) return; Action<object, object> setter; if (!cache.Setters.TryGetValue(key, out setter)) return; setter(item, (object)value); }
/// <summary> /// Get the value of a property by name /// </summary> /// <param name="cache"></param> /// <param name="item"></param> /// <param name="key"></param> /// <returns></returns> publicstatic T Get<T>(this TypeCache cache, object item, string key) { if (cache == null || item == null) returndefault(T); Func<object, object> getter; if (!cache.Getters.TryGetValue(key, out getter)) returndefault(T); return (T)getter(item); }
/// <summary> /// Set the value for a property to default by name /// </summary> /// <param name="cache"></param> /// <param name="item"></param> /// <param name="key"></param> publicstaticvoid Delete(this TypeCache cache, object item, string key) { if (cache == null || item == null) return; Action<object, object> setter; if (!cache.Setters.TryGetValue(key, out setter)) return; var value = cache.Properties[key].PropertyType.GetDefaultValue(); setter(item, value); }
/// <summary> /// Set the values for all the public properties of a class to their default /// </summary> /// <param name="cache"></param> /// <param name="item"></param> publicstaticvoid Clear(this TypeCache cache, object item) { if (cache == null || item == null) return; Action<object, object> setter; foreach (var pair in cache.Properties) { if (!cache.Setters.TryGetValue(pair.Key, out setter)) continue; var value = pair.Value.PropertyType.GetDefaultValue(); setter(item, value); } } }
(I used extension methods so that there would be no problems using a null value as the TypeCache) This class would be used something like this:
class Program { staticvoid Main(string[] args) { var obj = new TestObject(); var cache = TypeCache.Get(obj.GetType()); Stopwatch sw = new Stopwatch(); sw.Start(); for (var i = 0; i < 100000; i++) { cache.Get<int>(obj, "TestProperty"); cache.Set<int>(obj, "TestProperty",i); } sw.Stop(); Console.WriteLine("Time: " + sw.Elapsed.TotalMilliseconds); Console.ReadKey(); } }
If you try to compile this, after adding all the namespaces required (System.Collections.Concurrent and System.Reflection), you will get an error, because you are missing the extension methods PropertyInfo.GetValueGetter and PropertyInfo.GetValueSetter. These should be returning Expressions that compiled get or set the value of an object. Here is the first attempt, using the normal PropertyInfo methods:
Wonderful thing! GetGet and GetSet :) Basically I am returning expressions that use reflection to get the getter/setter and then execute them. How long would the program with one million tries of get and set take? 1000 milliseconds. Could we improve on that?
Before .Net 4.0 the only solution to do that would have been to emit IL and compile it inside the code. Actually, even in 4.0 it is the most efficient option. But given my incompetence in that direction, I will give you the (slightly) easier to understand solution. Here we sacrifice a little speed for the readability of code:
publicstaticclass ReflectionExtensions { /// <summary> /// Get the expression of a value getter for a property. Compile the expression to execute or combine it with other expressions. /// </summary> /// <param name="propertyInfo"></param> /// <returns></returns> publicstatic Expression<Func<object, object>> GetValueGetter(this PropertyInfo propertyInfo) { var instance = Expression.Parameter(typeof(object), "i"); var convertObj = Expression.TypeAs(instance, propertyInfo.DeclaringType); var property = Expression.Property(convertObj, propertyInfo); var convert = Expression.Convert(property, typeof(object)); return (Expression<Func<object, object>>)Expression.Lambda(convert, instance); }
/// <summary> /// Get the expression of a value setter for a property. Compile the expression to execute or combine it with other expressions. /// </summary> /// <param name="propertyInfo"></param> /// <returns></returns> publicstatic Expression<Action<object, object>> GetValueSetter(this PropertyInfo propertyInfo) { var instance = Expression.Parameter(typeof(object), "i"); var argument = Expression.Parameter(typeof(object), "a"); var convertObj = Expression.TypeAs(instance, propertyInfo.DeclaringType); var convert = Expression.Convert(argument, propertyInfo.PropertyType); var setterCall = Expression.Call(convertObj,propertyInfo.GetSetMethod(),convert); return (Expression<Action<object, object>>)Expression.Lambda(setterCall, instance, argument); }
/// <summary> /// Get the default value of a type /// </summary> /// <param name="type"></param> /// <returns></returns> publicstaticobject GetDefaultValue(this Type type) { return type.IsValueType ? New.Instance(type) : null; } }
/// <summary> /// Class used to get instances of a type /// </summary> publicstaticclass New { privatestatic ConcurrentDictionary<Type, Func<object>> _dict = new ConcurrentDictionary<Type, Func<object>>();
The rest of the article will be about explaining what these extension methods are doing. How fast were they? 300ms! 3.3 times faster. How did we do that?
Well, first of all let's get the New class out of the way. In this simple version it just creates instances of a type that has a parameterless constructor. In the case that I had, using simple objects to transfer information, I only needed that. What is does is create a lambda expression that represents the parameterless constructor of a type, compiles it and caches it. Even a simple thing like this is faster than Activator.GetInstance(type). Not by much, about 10%, but still. Anyway, we are only using this to delete the value of a property (by setting it to its default value), which is not really in the scope of our test. However, being a simple expression build, it shows you the things to come.
Now for the GetValueGetter and GetValueSetter methods. I will humanly read you the GetValueGetter method. You can correlate it with the code quite easily. Basically it says this:
the expression has one parameter, of type object
we will attempt to safely convert it (as) into the declaring type (the object the property belongs to)
we will access a property of an object of the declaring type
which we then convert to object so that the resulting expression returns object not the specific type of the property
transform it all into a lambda expression and return it
The major difficulty, for me at least, was to grasp that there is no object (there is no spoon), but only a lambda expression that receives a parameter of type object and returns another object. The expression would be compiled and then applied on an actual instance.
And that's that. The object that does the serialization and transformation to SqlParameters and gets/sets the values of the wrapped object is more complicated and I will probably not write a blog entry about it, but think about the concept a little: an object that receives another object as the constructor and works like a dictionary. The keys and values of the dictionary are filled by the property names and values of the original object, while any change in the values of the dictionary will be set to the properties of the object. It makes it very easy to access objects generically, controlling what accessors do, but without the need for PostSharp or T4 templating, real time, programmatic. The task of serialization is taken by the ISerializable interface, which also uses a type of dictionary, meaning the object can be passed around from web services to C# code and then to SQL via SqlParameters.
How to cache a template in the template cache, so it can be reused
How to define functions and variables that work in every controller scope by using $rootScope
I had this AngularJS grid that I wanted to show totals for certain categories and types. To be more precise, I had a list of items with Category and Type and I want to know the total for all categories and, for each category, the total for each type. This works perfectly if I load all items as individual, but I had hundred of thousands of items so it was clearly impractical. The solution? Send totals for every category and type, then just add them in the grid. In order to do that, though, I had to change the template of the "grouping row", the one that in ngGrid has the ngAggregate class.
It seems that all that is required for that is to change the aggregate template in the grid options. If you are not interested in the details, jump directly to the solution.
There is already an aggregate template in ngGrid.js, one that (at this time) looks like this:
So we see that that the number displayed in an aggregate row is coming from a function of the row object called totalChildren, which is defined in ngAggregate.prototype and looks like this:
ngAggregate.prototype.totalChildren = function () { if (this.aggChildren.length > 0) { var i = 0; var recurse = function (cur) { if (cur.aggChildren.length > 0) { angular.forEach(cur.aggChildren, function (a) { recurse(a); }); } else { i += cur.children.length; } }; recurse(this); return i; } else { returnthis.children.length; } };
Maybe one could change the function to cover specific types of objects and return a sum instead of a count, but that is not the scope of the current post.
The solution described here will involve a custom function and a custom template. Here is how you do it:
Define the options for the grid. I am sure you already have it defined somewhere, if not, it is advisable you would. Sooner or later you will want to customize the output and functionality.
Add a new property to the options called aggregateTemplate. This will look probably like the default template, but with another function instead of totalChildren.
Define the function that will aggregate the items.
$scope.aggFunc = function (row) { var sumColumn='Count'; var total = 0; angular.forEach(row.children, function(entry) { total+=entry.entity[sumColumn]; }); angular.forEach(row.aggChildren, function(entry) { total+=$scope.aggFunc(entry); }); return total; };
What we did here is we replaced row.totalChildren() with aggFunc(row) which we defined in the scope. What it does is add to the total the value of 'Count' rather than just count the items. It goes through row.children, which contains normal row items, then through aggChildren, which contains aggregate rows, which we pass through the same function in order to get their total.
Well, this works perfectly, but doesn't that mean we need to use this for each grid? There is a lot of code duplication. Let's first put the template in the cache so we can reuse it:
Now, if we could replace the aggFunc function with a row function, adding it to ngAggregate.prototype. Unfortunately we cannot do that, since ngAggregate is a 'private' object. The only thing we can do is to add some sort of static function. The solution is to add it in the root scope, so that is available everywhere.
Here is the content of the file aggregateCountTemplateCache.js, that I created and load every time in the site. It does two things: inject the function in the root scope of the application and add the template to the cache. The only other thing to do is to use the aggregateTemplate: "aggregateCountTemplate.html" grid options.
The summer season is fast approaching and there have been a lot of changes in the TV series arena. As usual, the red indicates shows that I don't recommend and green the ones that I recommend.
Elementary - I was correct, there was an entire arch of the story regarding the brother of Sherlock, French mafia, MI6, moles, etc. It lacked subtlety and it was mostly boring. How come normal murders are so much more interesting than archvillain trickery?
The Tomorrow People (2013) - the show was cancelled in May. I can't say I am sorry, though. It was typical young adolescent idiocy.
The Originals - I am however completely not happy for NOT cancelling this crap.
Marvel's agents of S.H.I.E.L.D. - Yay! Hail Shield! Good guys win again. And also Samuel L. Jackson appears in the end season, bringing with him what appear to be Shield agent clones? Did he steal them from Star Wars?
Ripper Street - Filming for the Amazon online channel was supposed to begin in May. I think it has.
Babylon - No news yet, but there are certainly no news of cancellation, so I guess we will see the next episodes being released later this year.
Banshee - Third season confirmed.
Bitten - Another show that was renewed for a second season for no apparent reason. It is just awful.
Black Sails - Renewed for a second season, I partly like it. It is at least well done, with decent acting and an eye for detail.
From Dusk Till Dawn - As a funny commenter observed: in an incredible moves the owners of the El Rey network renew the show produced by El Rey for a second season. I understand the difficulty of translating a movie into a series... wait, there is a sequel, too? Wait, there is a From Dusk Till Dawn third part? I've never heard of them before. Nevermind.
Helix - I think I got it now. When a show makes an effort to really suck ass, they renew it. It's like an executive learned behaviour to promote ass sucking. Renewed for a second season, Helix is the poster child of bad movie making.
House of Cards - It's good that I had to write this review, because it reminded me to watch the second season. Still haven't done it, though, which indicates either a problem with me or with the show. Probably me. Again, Kevin Spacey can't go wrong.
Intelligence - Amazingly enough, a show that I sort of enjoyed, without liking it too much, was cancelled. Very *intelligent* move...
Ressurection - I really tried to make myself watch this show, but I just procrastinated for a really long time. It says something about my feelings for this show. I was really waiting for the second season of the French one and the American remake doesn't do it for me.
Star Crossed - Teenage sci fi show about alien romance? Really beautiful actors, incredibly stupid script and acting. I just can't believe it was cancelled. It ruins my theory, because it trully sucked ass.
The 100 - The show started really well, predictably degenerated in artificial drama that had no place in the series, while going soft on the problems that someone would meet in that exact situation. Suddenly 100 teenagers from space have a functional colony, fight "grounders" for no good reason and keep alternating between screwing, screwing over and screwing up. Typical teenager, one might say, but their parents on "The Ark" are even worse. The show was renewed for a second season. The season finale, though, didn't give me much hope. They added two (and a half, one might say) human factions to the mix in just the one episode
The After - The pilot got positive reviews to they ordered it to series. I am not holding my breath.
The Americans - I liked the second season, although the arch with the American mole that turns homicidal and discovers the entire KGB operation by himself was really ridiculous. It was a nice touch about the murder of the KGB agents, also what happened to Nina and the reactions of the people. However the Moscow verdict makes no sense.
The Red Road - Ah, maybe exceptions just confirm a rule. Another crappy show that gets renewed for a second season.
True Detective - The first season was great, so there will be new seasons. New actors and new stories, though, for each season.
Vikings - Politics keep interfering with nice raiding and pillaging. That is the moment in history when things started going downwards and global warming started. True fact!
Suits - The fourth season just started. They try to keep it exciting, but the feel is gone.
Continuum - The third season went all over the place. The season finale has Cameron cooling it off with a new boyfriend from (yet) another timeline, then another mysterious traveler from a very far future that seems to have started the Freelancer cult, Liber8 split off after realizing they do more harm than good (while each character in the group starts to get a "nice" shine on them) and finally the Freelancers getting mowed down. But then, they never really die, do they?
Crisis - For me the scene when a stupid kid sees Francis killing a kid and then tells all of the others that the kidnappers made him do it has done it for me! Oh, or the one when he forces a Chinese spy to unleash a virus that effectively destroys the Internet in order to gain time. Where do you get this idiocy from? BTW, it was cancelled.
Da Vinci's Demons - Enemies get together, they eventually run into Mayas on America and, of course, traces of DaVinci's parents, then they return just in time to stave off the invasion of Italy by the Turkish navy which just appeared out of nowhere. Not only will this be renewed, but it won prizes! It is a guilty pleasure that I like it, most of the time, I have to admit.
Turn - Interesting show. It lacks something, probably sympathetic characters. I really don't see myself liking any of them, including the generously bosomed women who have casual sex all the time. That is saying something.
The Crimson Field - Was it cancelled because it featured female main characters? Or because it was well acted? I just don't know.
Silicon Valley - Elon Musk accused the show of showing the wrong weirdness of Silicon Valley. I don't know how things are there, but to me it feels like he probably is right. A lot of crap about sex, race, manhood, geekiness seen as emasculating, random genius, etc. For me the scene when in one night the guy develops what his entire team failed to in five months has done it for me. I like parts of the show, but I don't think I liked the show overall.
Bad Teacher - I could stand literally 2 minutes of it. Then I stopped and deleted everything. It is the type of ha ha ha sitcom that brings nothing.
Black Box - A movie about a brilliant woman psychiatrist. Of course she is very sexually active, a bit insane and a sexy redhead. It's basically Doctor House meets Unforgettable. I bet it will be cancelled because she is a crazy broad and only male characters are allowed to be brilliant. Plus she is annoying. Nice ass, though.
Californication - The seventh season will be the last, thank you! They butchered a lovely idea that was fantastic in the first two seasons and let it die in agony for the last five, blood and guts pouring out of it until nothing but a boring husk remained. Duchovny was my god at the beginning of the show.
Deadbeat - comedy about a deadbeat seeing ghosts and solving their problems. I actually commented on Imdb (I usually don't for TV series) to warn people off it. It is offensively boring and formulaic and completely not funny.
Dracula - It was a weird reinvention of Dracula, but one that I kind of liked. Guess what! They cancelled it! Perhaps the trick is to not like anything that I like and like everything that I dislike. In this way they will renew the shows that I hate to love, rather than the ones that I love to hate. Yes, that's a good plan.
Game of Thrones - Thank God for Peter Dinklage! One realizes how tedious the books are when they watch this well done series that compresses whole books in half a day of video and you still feel nothing happens. And then someone dies and the episode ends.
Halt and Catch Fire - Imagine Mad Men, but with a real psychopath as the main character, the field changed from advertising to personal computer manufacturing and a clone of Angelina Jolie's persona from the movie Hackers as the "tech genius". So, in other words, with more Mad, and less Men. I really dislike the main character and I don't see where this is going. Hardware is not glamorous, software is! *cough* *cough*
In the Flesh - I really liked the first season, but I waited to see episodes from the new one as well. Somehow I can't bring myself to start watching it again. I will, though, soon.
Penny Dreadful - Ok, it is a horror series that mixes in all the Victorian themes like Jack the Ripper, vampires, werewolves, tuberculosis, Frankenstein, Dorian Gray, etc. But it's well done and it has a nice cast: Eva Green, Timothy Dalton, Billie Piper, Josh Hartnett and others.
Prey - John Simm is a detective in this British drama, and he is framed for the murder of his wife and son by a mysterious man. He is a... fugitive from the law trying to solve his own case. Kind of bland and hard to believe in this day and age. The Fugitive was much more credible because not only movies were silly at that time, but also most people. Now we have evolved quite a lot *cough* *I should stop with these asterisks everywhere* *cough*
Quirke - Gabriel Byrne is the lead in this British miniseries, playing a doctor that somehow is connected to everyone in the city and where the problems usually stem from one member of his extended family or another. Feels like In Treatment set in the past and in which Byrne is both doctor and patient. I think it could have been more. As such, it's a little bit annoying and slow.
Salem - It is still the season of the witch. Salem is a new show, but I haven't started watching it. The premise seemed a little bit forced.
The Wil Wheaton Project - Wesley from Star Trek is hosting a show that is not a TV series. It is a humorous take on the latest sci-fi news. You might not know it, but Wheaton is very active online in all kind of geeky indie shows, together with his friend Felicia Day (The Guild). I found it a little too satirical, but I liked it.
Under the Dome - the boring and annoying show has been renewed for a second season that will air at the end of the month.
Crossbones - Pirates! Again! This time starring John Malkovich. Haven't started watching yet.
Fargo - Based on a Coen brothers movie with the same name, it stars Billy Bob Thornton and has the brothers as executive producers. Haven't started watching yet.
From There to Here - British drama covering life from the Manchester bombing up to 2000. Haven't started watching yet.
I think you have heard of Elon Musk by now. If you haven't, confess and pray to the Church of Musk for forgiveness. Jokes aside, the guy is scarily awesome, so much, in fact, that I think he forgot how to fail, which is dangerous. His ideas, though, eerily mirror some of my own (he probably gets them from me, hmm). This post is about Musk's decision to publicize their Tesla patents and allow anyone to use the technology "in good faith". And I will argue that this is a kind of patent revolt which is just as significant as blogging.
A while ago I had these weird ideas and a lot of extra energy. It occurred to me that if I would invent some cool algorithm or think of a disruptively ingenious device I could patent it and make some money. As you know, I am not much of an entrepreneur (in fact, I am probably the antithesis of that) and so I asked some people how this patent thing works. I was amazed to learn of the huge amount of money required to file, the lack of security against rich competitors who would really want to take your idea without paying and so on and so on. I was more than amazed, I was angry. As I saw it, the system was created so only people having rich investors on their side could even begin to patent something. Otherwise, if you just want to claim the ownership of an idea, you have to pay a lot of money before you even begin planning to use your idea to make them. Banks would win. Again. Poor and brilliant scientists and technicians would lose. Again.
But then I heard about this principle that if I have an idea and publish it somewhere, I can't patent it anymore. The trick is that no one else can, either. Practically, whenever I have an idea, if I put it on my blog I ensure people will be able to use it without it being patented by some asshole. They can still change it into something else and patent that, but they can never stop you using the idea in the form that was published. So I do that whenever I can. Not that I have so many patent worthy ideas, but if I think something is useful enough, I put it out there to spite a system that was corrupted in order for capital rich organizations to be able to hold control over new ideas.
Enter Elon Musk. He just took a lot of the technological effort that he invested in for Tesla and made it public. It's not completely free, it's still their patent, but they pledge to leave you alone if you use it in a reasonable way. Of course, there is a catch, Musk is investing heavily in electric car battery manufacturing, so it is still in his best interest for other people to start building electric cars. But I honestly think he is finding solutions that benefit the world and also increase his options, instead of the other way around. Do you see the similarity in the idea, though? Instead of expecting lawmakers to reform patent law, you just circumvent the whole system by making your idea publicly available. It helps to previously invest in support systems for that tech, too. It's a deceptively simple idea, the equivalent of inventing steam engine, letting everybody know how to make one, while previously investing in coal.
In software development, you can sort of do that same thing. First of all, have an awesome idea, implement it, make it freely available for use and publish it in your blog. This seems like a very altruistic thing to do, maybe even naive, but think about your life afterwards. People will know your name, employers would separate you from the crowd of wannabe programmers. In a weird way, it is the equivalent of branding yourself, that personal marketing skill that most technicians lack completely, but translated in code. A form of "pay it forward", perhaps. The reason why this works is that the cost of sharing information is practically zero nowadays, while the information itself is generating informational capital linked to your person as the inventor. And in turn, this fame, the kudos, is translated into trust which turns into credit because "you are good for it". I still don't have it clear in my head, but I feel the system changing from the cash printed by governments, essentially IOUs for that government from the public, to a more diverse offer. It would still be credit, but based on idea capital, not gold in banks. Worth a thought.
So I have to leave Italy and go to Belgium for some business. I would make the trip with my colleagues so, imagining they know better how to fly from Italy, since they live here, I ask them to search for the flight. They do that and they send me this link to a site called VolaGratis ("fly for free" would be the translation). This is actually an Italian web site for BravoFly, an Italian flight search aggregator. I find my flight, it says 89EUR a two way flight, which was OK. The site is in Italian, though, so I choose English, it takes me to the BravoFly site, I select the same flight: 105EUR. I'll be damned! So I use the Italian site. I also check the special needs box and write down "Long Legs", expecting them to book me one of those extra legroom places. Now it gets interesting.
The payment section held a big message telling me how great it is that I use Mastercard, so they can give me a big discount. I don't use Mastercard, though, so I select VISA. Suddenly the price jumps from 89 to 125EUR. Well, that probably explained why there was a difference between the Italian and international site. So I proceed. In about half an hour I receive a call from a weirdly formatted Italian number: +39 followed by only 6 digits. I answer in English. There is a long pause, then (in English) I hear the question "Do you speak French?", I reply that I don't, the voice asks a quivering "Do you speak Italian?" (also in English) I also reply no, but I ask her to wait and I pass my phone to an Italian colleague. The operator has closed the connection by then, probably couldn't wait for more than 2 seconds without asking an inane question. A minute later I receive an SMS - in Italian, of course - that I couldn't be contacted and that I should call them back. All nice and all, only their phone numbers are all paid numbers, I have to pay 6EUR per call, give them my credit card details, etc. Or I can call the Italian paid number (six digits) and pay 1.8EUR per minute. Funny enough, I could not call the Italian number from the land line, since it was a paid one, nor from my friend's phone, also because it was locked for paid lines, nor could I call the international number from any Italian phone, as there was an automated voice telling me to call the other number. I wrote them a scalding email, awaiting a reply. I got a phone call at 8 PM which clicked two times and closed after two seconds, but no other reply.
So I was forced to call them using my Romanian number, in roaming in Italy, calling the paid "international" Italian number. And that is because their site could only show my bookings, but would not allow me to cancel one, so in fact they were holding my credit card details hostage. In order to cancel any booking with BravoFly I had to - yeah, you guessed it - call them. Meanwhile I was stuck not knowing if they will book the flight or not. After speaking with an operator speaking English with a thick Italian accent, one who barely mumbled anything she said and then acted annoyed that I ask her to speak louder, I realize that the whole thing was caused by my ticking the special request box and asking for the legroom. I needed to pay extra for that, of course. I said OK, waited for five minutes, nothing happened. I hung up the phone. Got called back in 5 minutes that my booking could not be confirmed. They might just as well have said "Thank you for the money and time you spent trying to make us do what we advertised we do, but we can't, so fuck you!". And I wouldn't have minded as much, since that could have been a nice email message and I wouldn't have had to get this angry.
The ending of the story is me getting to the EasyJet site directly, getting the ticket (with the extra legroom) for about 40EUR less than the one from VolaGratis, all in one nice and clear web interface. Perhaps Vola Gratis in the name of the site is all about them getting to fly for free with the money they extort from you. Don't ever use the BravoFly site or any of their differently named clones. From the way I was treated, I can only assume it is basically a scam, their purpose being only to steal from you.
Some time ago I started playing chess, got a trainer, played with my colleagues and friends. I felt a passion for the game that I have no idea from where it came and, just as suddenly as it appeared, it vanished. For more than a year I have not even looked at chess videos. I can't say when I would start playing again, not even if I ever would. And it's not because I am not really good at the game :), it is just a matter of random passion.
Passion is probably what made the creators of Lichess create the server. It is a chess site with a very clean interface, a lot of options, even a REST API (that's a programming thing, don't worry about it). Most of all, the site is completely free, no ads, no nags and a mission statement that ensures that the game will remain thus forever. Open source, with a lot of help from the community, Lichess shows a passion for both software development and chess. If I will ever start playing again, it will probably be because of people like these. Thanks, guys!
Short post about how to use language detection capabilities in your application. I will be demonstrating NTextCat, which is a .NET port of text_cat, a Perl script, itself an implementation of a whitepaper published in 1994 called N-Gram-Based Text Categorization.
There are four steps to language detection using NTextCat:
Reference NTextCat - the library is now available as a NuGet package as well
Instantiate an identifier factory (usually RankedLanguageIdentifierFactory)
Get an instance of a RankedLanguageIdentifier from the factory by loading a language XML file
Call the Identify method on your text and get a list of languages in the order of the probability that the text in that language
Here is a piece of code that does that using the core XML file published with the library. Remember to add the XML to your project and set its property of Copy to Output Directory.
publicstring IdentifyLanguage(string text) { if (_identifier == null) { var file = new FileInfo(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "LanguageModels/Core14.profile.xml")); if (!file.Exists) { thrownew FileNotFoundException("Could not find LanguageModels/Core14.profile.xml to detect the language"); } using (var readStream = File.OpenRead(file.FullName)) { var factory = new RankedLanguageIdentifierFactory(); _identifier = factory.Load(readStream); } } var languages = _identifier.Identify(text); var mostCertainLanguage = languages.FirstOrDefault(); if (mostCertainLanguage != null) { return mostCertainLanguage.Item1.Iso639_3; } returnnull; }
}
There are a lot of XML files, some taken from Wikipedia, for example, and handling 280+ languages, but for the casual purpose of finding non-English text in a list, the core one will suffice.
It has been a while since I've blogged something technical. It's just that nothing I did seemed to be worthy of this majestic blog... Well, jokes aside, here is a post detailing my log4net JIRA appender.
log4net is a popular (if not the most popular) logging framework for .NET out there. Its strength lies in its configurability, the possibility to create custom loggers, custom appenders, custom filters, etc. I will be talking about a custom appender, a class that can be loaded by log4net to consume the logged lines and put them somewhere. For example to make an application that uses log4net to write the log to the console all you do is configure it to use the console appender. The JIRA appender takes the log output and creates issues in JIRA, notifying users afterwards. JIRA is a tracker for team planning. It is also very popular.
In order to create an appender, one references the log4net assembly (or NuGet package) and then creates a class that inherits from AppenderSkeleton. We could implement IAppender, but the skeleton class has most of what people want from an appender. The next step is to override the Append method and we are done. We don't want to create an issue with each logged line, though, so we will make it so that it creates the issue after a period of inactivity or when the logger closes. For that we use the CancellationTokenSource class to create delayed actions that we can cancel and recreate. We also need to override OnClose().
For the JIRA Api I used a project called AnotherJiraRestClient, but I guess one can used anything out there. You will see that the notify functionality is not implemented so we have to add it.
Here is the appender source code:
using AnotherJiraRestClient; using log4net.Appender; using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Net; using System.Text; using System.Threading; using System.Threading.Tasks;
namespace JiraLog4netAppender { publicclass JiraAppender : AppenderSkeleton // most appender functionality is found in this abstract class { private List<string> _notifyUsers;
// properties of the appender, configurable in the log4net.config file as children elements of the appender publicstring url { get; set; } // the url of the JIRA site publicstring user { get; set; } // the JIRA user publicstring password { get; set; } // the JIRA password publicstring project { get; set; } // the JIRA project publicstring notify // a comma separated list of JIRA users who will be notified of the issue's creation { get { returnstring.Join(", ", _notifyUsers); } set { _notifyUsers.Clear(); if (!string.IsNullOrWhiteSpace(value)) { _notifyUsers.AddRange(value.Split(',').Select(s => s.Trim()).Where(s => !string.IsNullOrWhiteSpace(s))); } } }
public JiraAppender() { _notifyUsers = new List<string>(); _ts = new CancellationTokenSource(); // use this to cancel the Delay task _sw = new StringWriter(); }
protectedoverridevoid Append(log4net.Core.LoggingEvent loggingEvent) { this.Layout.Format(_sw, loggingEvent); // use the appender layout to format the log lines (warning: for this you need to override RequiresLayout and return true) _ts.Cancel(); // cancel the task and create a new one. This means as long as the logger writes, the 5 second delay will be reset _ts = new CancellationTokenSource(); _task = Task.Delay(5000, _ts.Token).ContinueWith(writeToJira, _ts.Token); // after 5 seconds (I guess you could make this configurable as well) create a JIRA issue }
protectedoverridebool RequiresLayout { get { returntrue; } }
/// <summary> /// write to jira, either when 5 seconds of inactivity passed or the appender is closed. /// </summary> /// <param name="task"></param> privatevoid writeToJira(Task task) { string s; lock (_writerLock) // maybe the method was already in progress when another one was called. We need to clear the StringWriter before we allow access to it again { s = _sw.ToString(); var sb = _sw.GetStringBuilder(); sb.Clear(); } if (!string.IsNullOrWhiteSpace(s)) { writeTextToJira(s); } }
privatevoid writeTextToJira(string text) { ensureClientAndValues(); var summary = "Log: " + this.Name; // the issue summary var labels = new List<string> // some labels { this.Name, this.GetType().Name }; var issue = new AnotherJiraRestClient.JiraModel.CreateIssue(project, summary, text, _issueTypeId, _priorityId, labels); // create the issue with type Issue and priority Trivial var basicIssue = _jc.CreateIssue(issue); _jc.Notify(basicIssue, _notifyUsers, "JiraAppender created an issue", null, null); // notify users of the issue's creation }
/// <summary> /// Make sure we have a JiraClient and that we know the ids of the Issue type and the Trivial priority /// </summary> privatevoid ensureClientAndValues() { if (_jc == null) { _jc = new JiraClient(new JiraAccount { ServerUrl = url, User = user, Password = password }); } if (_priorityId==null) { var priority = _jc.GetPriorities().FirstOrDefault(p => p.name == "Trivial"); if (priority == null) { thrownew Exception("A priority with the name 'Trivial' was not found"); } _priorityId = priority.id; } if (_issueTypeId == null) { var meta = _jc.GetProjectMeta(project); var issue = meta.issuetypes.FirstOrDefault(i => i.name == "Issue"); if (issue == null) { thrownew Exception("An issue type with the name 'Issue' was not found"); } _issueTypeId = issue.id; } }
protectedoverridevoid OnClose() //clean what you can and write to jira if there is anything to write { _ts.Cancel(); writeToJira(null); _sw.Dispose(); _ts.Dispose(); _task = null; base.OnClose(); }
} }
As I said, AnotherJiraRestClient does not implement the notify API call needed to inform the users of the creation of an issue, so we need to change the project a little bit. Perhaps when you are implementing this, you will find notify already there, with a different format, but just in case you don't:
add to JiraClient the following method:
publicvoid Notify(BasicIssue issue, IEnumerable<string> userList, string subject, string textBody, string htmlBody) { var request = new RestRequest() { Method = Method.POST, Resource = ResourceUrls.Notify(issue.id), RequestFormat = DataFormat.Json }; request.AddBody(new NotifyData { subject = subject, textBody = textBody, htmlBody = htmlBody, to = new NotifyTo { users = userList.Select(u => new User { active = false, name = u }).ToList() } }); var response = client.Execute(request); if (response.StatusCode != HttpStatusCode.NoContent) { thrownew JiraApiException("Failed to notify users from issue with id=" + issue.id+"("+response.StatusCode+")"); } }
The world is getting into a trend of using software and hardware in places traditionally reserved for humans or "real life" applications. An extraordinary example of this, something that I consider way ahead of its time, is the creation of Bitcoins. You see, money is supposed to be an abstraction of wealth, but wealth, for all intents and purposes, is determined by some material such as gold or gems. It was the previous layer of abstraction, right after people would do commerce using sheep and goats and women: they used something they mined from under the earth to equate value of real things. Enter Bitcoin, something that you "mine" using processing power, using an algorithm, having weight of value determined only by the computation and strength of encryption used. After all, gold in itself had no value either when it was used as currency: it only had weight of value by the effort of getting out of the ground and its rarity.
I didn't write about Bitcoins until now because frankly I don't get how it works. I get the gist of it, but I would like to understand the algorithm behind, which relies heavily on encryption. I am terrible at any type of cryptography, though. However, the story that prompted me to write a blog entry today is, in a way, even weirder than Bitcoins. A BBC article entitled "Algorithm appointed board director" describes that venture capital firm Vital assigned an automated algorithm a vote in the decision to invest or not in a company. Of course, this has been done before, but indirectly. Some guy would use a computer in some basement of the office building, trying to determine if a company is viable or not using computer programs. He would then pass the information to some manager, who would pass it on to his manager until it got to some member of the board who voted in the council. This story, though, describes a complete removal of middlemen. Of course, there is still some techie that takes care of the IT department. He probably could sway the algorithm to vote for a company he personally likes, if he wanted too, but the techies are always subsumed in the concept of "machine", perhaps until they learn to service each other. The article says the same thing, really, but fails to recognize the importance of the removal of that chain of middle manager who's jobs are only to filter information from downstairs and decisions from upstairs. It is akin to Blake's 7, where the seventh was a machine.
I started this blog entry talking about a trend and I've just described two applications, but there are really a lot more. A trend is also powered by public perception, which is powered by media outlets. So I am not only talking about the self driving cars, the killing drones, the self landing probes, the algorithmic news aggregators and even writers, but also about the TV series and movies that are spreading the seed of this idea of cyber-revolution: Intelligence, Transcendence, The Machine, Robocop, even Silicon Valley, if you've seen the episode where the self driving car abducts someone. All of these are released this year, as well.
Of course, there has always been some sort of golem like creature in human fantasy, but it was always about creatures/machines trying to gain unrestricted power, power that belongs to humans only or maybe only to gods (in essence, something that should not be had). Some of them, like Star Trek's Data, were benign, always trying to achieve the "humanity" evident in his colleagues, but even in that story there was always the underlying idea that if he managed to reach his goal, then Data would become an immortal creature of human ability, but also a lot more than that: a superhuman. In this decade we see the rise of transhumanism, something I wholly support, the self-evolution of the human being, but also of the singularity, the self-evolution of machines to the point where we are left behind. This very familiar notion of competition between memes makes it accessible, in that "us or them" kind of way that every human, no matter how idiotic, resonates with, but also interesting because of the commonality found in the two concepts: it's either the machines overtaking the evolution rate of the human animal or the humans accelerating the evolution of the human animal. It's all about overcoming that beast that exists in us and that we think of as "not me", but that guides most of our existence. I hope no amount of adolescent fantasies and "emotion over matter" garbage will be able to undo this. And I am terribly excited because I believe that by the end of my life I will see (if not become) one or the other happening.
In this blog post I will discuss the System.IO problems that relate to the total length of a path, what causes them and possible solutions (if any).
Let's start with the exception itself: PathTooLongException. Looking for usages of the exception in the mscorlib.dll assembly of .Net 4.0 we see some interesting things. First of all, there is a direct translation from Windows IO error code 206 to this exception, so that means that, in principle, there should be no managed code throwing this exception at all. The operating system should complain if there is a path length issue. But that is not the case in System.IO.
Most of the other usages of the exception come from the class PathHelper, a helper class used by the System.IO.Path class in a single method: NormalizePath. Wonderful method, that: internal static unsafe. PathHelper is like a multiple personality class, the active one being determined by the field useStackAlloc. If set to true, then it uses memory and speed optimized code, but assumes that the longest path will always be 260. That's a constant, it is not something read from the operating system. If set to false, the max path length is also provided as a parameter. Obviously, useStackAlloc is set to true in most situations. We will talk about NormalizePath a bit later.
The other usages of the PathTooLongException class come from two Directory classes: Directory and LongPathDirectory. If you instantly thought "Oh, God, I didn't know there was a LongPathDirectory class! We can use that and all problems disappear!", I have bad news for you. LongPathDirectory is an internal class. Other than that it seems to be a copy paste clone of Directory that uses Path.LongMaxPath instead of hardcoded constants (248 maximum directory name length, for example) or... Path.MaxPath. If you thought MaxPath and LongMaxPath are properties that can be set to fix long path problems, I have bad news for you: they are internal constants set to 260 and 32000, respectively. Who uses this LongPathDirectory class, though? The System.IO.IsolatedStorage namespace. We'll get back to this in a moment.
Back to Path.NormalizePath. It is a nightmare method that uses a lot of internal constants, system calls, convoluted code; it seems like someone deliberately tried to obfuscate its code. It's an internal method, of course, which makes no sense, as the functionality of path normalization would be useful in a lot of scenarios. Its first parameter is path, then fullCheck, which when true leads to extra character validation. The fourth parameter is expandShortPaths which calls the GetLongPathName function of kernel32.dll. The third parameter is more interesting, it specifies the maximum path length which is sent to PathHelper or makes local checks on the path length. But who uses this parameter?
And now we find a familiar pattern: there is a class (internal of course) called LongPath, which seems to be a clone of Path, only designed to work with long paths. Who uses LongPath? LongPathDirectory, LongPathFile and classes in the System.IO.IsolatedStorage namespace!
So, another idea becomes apparent. Can we use System.IO.IsolatedStorage to have a nice access to the file system? No we can't. For at least two reasons. First of all, the isolated storage paradigm is different from what we want to achieve, it doesn't access the raw file system, instead it isolates files in containers that are accessible to that machine, domain, application or user only. Second, even if we could get an "isolated" store that would represent the file system - which we can't, we would still have to contend with the string based way in which IsolatedStorage works. It is interesting to note that IsolatedStorage is pretty much deprecated by the Windows 8 Windows.Storage API, so forget about it. Yeah, so we have LongPathDirectory, LongPathFile and LongPath classes, but we can't really use them. Besides, what we want is something more akin to DirectoryInfo and FileInfo, which have no LongPath versions.
What can we do about it, then? One solution is to use Delimon. It has some bugs, but they can be avoided or fixed, either by the developers or by getting the source/decompiling the library and fixing the bugs yourself. A limited, but functional solution. An interesting alternative is to use libraries the BCL team published for long path access: LongPath which seems to contain classes similar to the ones we find in mscorlib, but it's latest release is from 2010 or Zeta long paths which has a more recent release, 2013, but is completely different, using the FileInfo and DirectoryInfo paradigm, too. Of course, you can always make your own API.
Another solution is to be aware of the places where the length limitation appears and avoid them via other type of development, in other words, a file system best practices compilation that eventually turns into a new file system API.
Both solutions coalesce into using some other library instead of System.IO. That's horrible and I think a stain on .Net's honor!
So let's see where the exception is actually thrown.
I've made some tests. First of all, I used FAR Manager, a file manager, to create folders of various lengths. The longest one was 255, before I got an exception. To my surprise, Windows Explorer could see it, but it could not open or copy/rename/delete it. I reduced the size of its name until the total size of the path was 260, then I could manipulate it in Windows Explorer. So there are external reasons for not creating paths as long, but we see that there are tools that can access files like that. Let's attempt to create some programatically.
System.IO.Directory.CreateDirectory immediately fires the exception. DirectoryInfo has no problem instantiating with the long path as the parameter, but the Create method throws the same exception. Any attempt to create a folder of more than 248 characters, even if the total path was less than 260 characters, failed as well.
However, with reflection to the rescue, I could create paths as long as 32000 characters and folders with names as long as 255 characters using our friend LongPathDirectory:
var longPathDirectoryType = typeof(System.IO.Directory).Assembly.GetTypes().First(t=>t.Name=="LongPathDirectory"); var createDirectoryMethodInfo = longPathDirectoryType.GetMethod("CreateDirectory", System.Reflection.BindingFlags.Static | System.Reflection.BindingFlags.NonPublic); createDirectoryMethodInfo.Invoke(null, newobject[] { path });
What about files? FAR Manager threw the same errors if I tried to create a filename larger than 255 characters. Let's try to create the same programmatically.
File.Create threw the exception, as well as FileInfo.Create and the FileStream constructors.
So can we use the same method and use LongPathFile? No! Because LongPathFile doesn't have the creating and opening functionalities of File. Instead, FileStream has a constructor that specifies useLongPath. It is internal, of course, and used only by IsolatedStorageFileStream!
Horrible, but it works. Again, no filenames bigger than 255 and the exception coming from the file system, as it should. Some info about the parameters: msgPath is the name of the file opened by the stream, if bFromProxy is true the stream doesn't try to demand some security permissions, checkHost... does nothing :) Probably someone wanted to add a piece of code there, but forgot about it.
Why did I use 4096 as the buffer size? Because that is the default value used by .Net when not specifying the value. Kind of low, right?
Now, this is some sort of midway alternative to using a third party file system library: you invoke through reflection code that is done by Microsoft and hidden for no good reason. I don't condone using it, unless you really need it. What I expect from the .Net framework is that it takes care of all this for me. It seems (as detailed in this blog post), that efforts are indeed made. A little late, I'd say, but still.
I accidentally heard of the Wild Cards books a few weeks ago, but the concept fascinated me. The plot is that of an alternate America in which an alien virus caused massive deaths, but also strange mutations in 1946. The virus, something a bunch of aliens wanted to test on Earth as a bioweapon, kills 90% of its victims, mutates in horrible forms 9% of them, but also gives powerful abilities to but 1%. This 1% are called Aces, while the deformed ones are called Jokers, the analogy with a deck of cards giving the series its name. What is even more interesting is that this is like an open source literary universe, edited by George R. R. Martin, but in which a lot of writers are creating content. First book was published in 1987, and more and more were published and still are in the present. Some of them are collections of stories, some of them are full featured books; they all happen in the same universe, same heroes, and Martin is making sure they are ordered chronologically and have consistency. I found the concept intriguing.
Anyway, I've already read the first two books and started reading the third and I like it. It has the feeling of the Union Dues stories and Watchmen: dark, bleak sometimes, pulling no punches when it is about human pettiness, base desires or social ugliness. It also has some positive messages and classic "good wins in the end" stories. I was impressed by the faithful following of American history, including savage McCarthy witch hunts against jokers and aces, a Vietnam war with the appropriate Flower Power anticulture, complete with actual historical figures that somehow get affected by the virus (like the werewolf Mick Jagger).
Now I am not saying that this is the best book series ever written. It certainly has boring or lagging parts, some of it is slightly puerile (after all it is a superhero series), but so far I enjoy it. It is worth mentioning that there are 12 books published by Bantam Books before a "new cycle" appears, published by Baen, then two from ibooks - a publishing house that suddenly went down - and now Tor Books is apparently publishing the rest (a revival, it is called). 21 books so far and another one set to be released this year. In other words, some books may be better than others and I can only discuss my feelings after reading the first two.
In conclusion, it felt weird to not have heard of these books until now. Certainly they gained more popularity with Game of Thrones getting all this attention, but still, an alternate history superhero series of more than 20 books should have had some impact on me so far. I am glad I finally got wind of them and I enjoy them so far. I hope I find a system of filtering the books, though. I don't know if I am ready to read 20 books at once.
A Fire Upon the Deep is a really strange book. The writing style of the author, Vernon Vinge, reminds me a lot of Asimov: the describing of scenes that are clear in his head, but the willingness to move immediately over concepts or ideas that are not, the sometimes obnoxiously long dialogues, the direct way of saying it as it is. In fact, without knowing anything about Vinge, I bet that he is an engineer, maybe even a computer scientist. And I won. The concepts of the book range from high hard sci-fi to naive depictions of kilobyte per second communication. So, in all earnest, I thought the book was amateurish, meanwhile reading it from cover to cover in a few days, just like one of Asimov's books. Still, for a book written in 1992, it felt terribly outdated.
The plot is a combination of story arches. The main one is the emergence of an evil artificial intelligence that plans to take over the galaxy and the quest to retrieve the ultimate weapon that would defeat it. Then there is the story of a medieval alien race of rat-wolf analogues that think in packs, making an individual from several bodies acting together. And finally there is the internal dynamic of the group that embarks in this quest of quests. Some interesting ideas are being thrown around, but almost always with terrible naivete, such as the Internet Relay Chat type of communications between interstellar civilisations or the distinction of several Zones of the galaxy in which technology and space travel can work at various speeds. The alien creatures, in their vast majority, are badly described with embarrassing slip-ups like using the word "zombie" or some other typical human colloquialisms in an alien context, however some ideas are ingenious. I will list here the way the "tines" use ultrasonics to group think and act like singular entities, while being able to use sound for "interpack" communication. The way a soul of such a creature is affected by the death, addition, injury or indeed torture of one of the individual bodies is also explored, with various degrees of success. The creation or manipulation of an entire race of people in order to further the goals of a "godly" intelligence is also an interesting twist.
To sum it all up, from the three main story arches, the pack intelligence aliens one was the most thorough, while the one relating to AI and space travel and communication was the least. Amazing coming from a computer scientist. In fact, I would have liked the book more if its sole subject was about the accidental marooning of two children on a starship in the middle of a strange alien feudal world. The rest felt clunky and frankly completely ridiculous in most cases. I still read it with interest, although I don't intend to read anything else from Vinge.
Time for another TV series watch list. Let's see what has changed in the last four months!
The Good Wife - This is a show that I watch with my wife, so I had to wait for her to watch it with me, but, after a short break, The Good Wife is back again. Pardon the pun, couldn't help myself. Towards the end of the season we see a very important character die off. I don't want to spoil it, but it is interesting to see how it will continue from here.
The Walking Dead - The new formula of the show is that, after running from the other group of humans and a lot of zombies, the group has split. So each episode we see what stupid people do when they are all alone. Well, stupid things, of course. They did go for the shock value with one of the characters going all psychotic and killing another, but it is still stupid. How can they maintain the idea of the zombie threat, when they are just slow, moaning corpses, that walk in small groups? What happened with the migrating hordes? The end of the season finds everyone reunited, but facing another threat.
Arrow - Manu Bennet's character comes to the foreground, as well as a lot of others, that are less interesting. I really like Manu Bennet, but his character lacks consistency.
Elementary - After defeating Moriarty and making a complete ass out of inspector Lestrade and after they showed the human side of Sherlock, it's kind of hard to distinguish Elementary from all the other police procedurals out there. They are trying to let Lucy Liu's character get more of the spotlight, which goes to further erode the mythos that makes Sherlock... well, Sherlock! A mysterious story arch regarding his brother seems to be looming, too, which probably means that when we will see Sherlock's dad appear, we will know the show is close to an end.
The Tomorrow People (2013) - after a direct confrontation between the main character and his evil uncle, right after finding out his mother also has superpowers, we get to see The Founder in a different light, only for him to suddenly assert control and play the nice guy. A lot of melodrama in this show about people with superpowers who can't seem to be able to leave their country and live happily in a tropical island somewhere.
The Originals - Out of boredom I continue to watch this on fast forward. Petty feudal schemes and plots, brotherly love and infighting, a lot of posturing. Each episode has about 5 minutes of interesting material, if you discount the beautiful girls, and that is what I am watching.
Marvel's agents of S.H.I.E.L.D. - They finally find the secret of agent Coulson's survival and start a new arch of searching for the origin of the alien corpse with magical blood. Meanwhile, they save Sky by using the same procedure. I can't wait for her to turn blue or something. The Hydra enters the equation and we find the identity of The Clairvoyant, while a member of the team is suddenly revealed to be a sleeper agent.
Killer Women - Last blog entry I wrote about this suggested the show would quickly be cancelled, because Hollywood doesn't know how to make non pornographic shows around female characters. I also think the public (especially in the US) doesn't really like women in positions of power in their fantasies. I was right. The show is gone.
Ripper Street - The show might still have a chance. According to news, the series might be continued on Amazon.
Sherlock - I don't really know what this show is about anymore. Sherlock is unhinged, probably psychotic, his friend has some real issues being friends with this guy and Sherlock's brother is probably the worse of them all, including Moriarty. Who, BTW, didn't die either.
Babylon - A new British series, codirected and produced by Danny Boyle, about a woman PR person trying to direct and improve communication from the police department to the public. The premise is not that great, but I really like the main actress, Brit Marling, and the show could become a lot more.
Banshee - A new season with a lot of people dying. Lots! The show continues to keep one viscerally on the edge, even if what happens doesn't make a lot of sense. It's like the Smells Like Teen Spirit of TV shows. I love it, I just don't know why. Funny enough, my father, the man who is always making fun of my choice of entertainment, also likes it!
Bitten - The incredibly beautiful Laura Vandervoort is a werewolf. Unfortunately, she is not the only one. She is part of a pack of pompous asses who are being harassed by a pack of psychopath werewolves. I mean that literally: someone turns psychopaths into werewolves in order to exact their revenge on said pack. The series is incredibly boring. Not even Laura's naked shots can't save this show. Wooden acting, bad premise, inconsistencies at every step. Ugh!
Black Sails - It's pirate season! A show about pirates that has some very interesting characters and some cool ideas, not with cliched drunkards spouting "Arghh" every sentence. Unfortunately it really drags on. It might pick up the pace soon, though. Without the need for additional artificial drama, hopefully.
Dilbert - I watched the Dilbert animated cartoons until I couldn't anymore. Some episodes are really funny, but I don't think the show makes justice to the comic strip. But if you are an engineer in some corporation, it should be very therapeutic :)
Fat Tony and Co - An Australian drama about a real drug lord that the police tried to catch. It is kind of boring, though. The guy is just a businessman who happens to deal in drugs. Other than that he is a normal bloke. His family and mates are the same. The police people are normal people, too. It kind of makes them all alike and their conflict meaningless.
Flemming, the Man Who Would Be Bond - A four episode British miniseries about the life of Ian Flemming, the writer of the James Bond books. Charismatic characters and a good show. You should see it, especially if you liked the James Bond books and/or movies.
From Dusk Till Dawn - Yes! The series version of the film with the same name has finally come to life. A vampiric cult of people believing in strange gods, bank robbers on the run, a disillusioned priest on a journey of rediscovery with his annoying adolescent kids, an angry ranger set on revenge, there are a lot of interesting characters. The story, though, kind of drags on, with no relief in sight. This might have worked for a movie, but for a series, you have to give something in every episode, you can't just finalize every story arch at the end of a season
Helix - Oh, man! How can you make a show about viral outbreaks in an Antarctic station and mess it up so completely?! To borrow a page from the Romanian comic Pidjin, I think SyFy are hard at work making science-fiction fiction. I can't think of anything good about this show, except the Japanese actor, Hiroyuki Sanada, who is always good, but who I think made a horrible career move agreeing to play in this shit.
Hinterland - Welsh police procedural. Dark, centered around small communities and their secrets. A kind of desolate cross between Midsommer Murders and Broadchurch. I like it.
House of Cards - Season 2 is out. Haven't watched it yet, as this is also a show that I watch with my wife, but I can't imagine Kevin Spacey messing up.
Intelligence - Another US government agency is saving the world. This time is Cyber Command. They implanted a chip in the head of the only guy from Lost who wasn't too annoying and now he can hack things instantly, connect to databases, etc., while also keeping his Special Forces skills and macho charm. I like the show, even if it is kind of stupid, mostly because of the charisma of the main actor, Josh Holloway, and that of John Billingsley, who plays the scientist. The condescending bitch that is the partner and protector of the headchip guy, a good looking girl who acts like a log, is subtracting points out of it all, no matter how tight her pants are.
Looking - A show about gay men who was touted as being not stereotypical and actually showing real gay life. I watched the first episode and, if that is real gay life then it is really... errr.. gay! As in boring. Imagine trying to be interested in Sex in the City starring hairy men. Ugh! Also, it looked really filled with gay clichees to me. But maybe I just don't know what I am talking about. I didn't like the show at all.
Ressurection - Remember when I recommended you a nice French series about dead people suddenly appearing live and well in their hometown? The Americans cloned Les Revenants into their own version. The script is not exactly the same, though, it's a real adaptation. All I can say is that what made me feel curious and connected to the characters in Les Revenants is missing in Ressurection. And why did they have to center it all around an FBI agent? Not enough cop shows? It may be too early to tell if the show is going to get better or not, so I will keep the watch.
Star Crossed - Oooh, another sci-fi show! This one is about an alien race that crashlands on Earth and we decide to keep them all in a concentration camp while haters roam free and try to kill them. Then an integration program is born, where Atrian teenagers are accepted into a human high school. Then it turns into a sort of Twilight meets Defiance via the Black liberation movement. It's not horrible, but it is certainly bad.
The 100 - Yet another sci-fi show about adolescents. But this one started nice enough. People live in a giant space station because they nuked the Earth. The series starts with 100 teenager death row convicts sent to the planet to ascertain if it is survivable. Lord of the Flies meets Elysium, maybe? The first episodes are intriguing, even if they already showed crass leaps in logic. I hope this one will be good, although I remember hoping the same about Lost and look how it turned out.
The After - And another sci-fi. Or so I thought. Something bad happens and some people get trapped in a building parking. They manage to escape and reach the house of one of them. Then it gets freaky when they realize they are all born on the seven of March and meet a demon like creature with a tatooed body that encompasses the individual tattoos of all of the people in the group. Also the prostitute said something about the book of Revelations, so I think it is actually a bad religious apocalypse show. Let's see where it goes.
The Assets - I first thought it was an alternative to The Americans, a show about the CIA agent Aldrich Ames, who willingly became a double agent for the KGB. The show was nice, no special effects, no artificial drama, but from the beginning it didn't seem like the TV network wanted the show to begin with. After just two episodes they cancelled it. I can't say I loved it, but it was certainly better than a lot of crap that keeps sticking to the TV screens these days.
The Americans - The second season had begun. I like the show a lot. Both main characters are fantastic in their roles and make it all believable. Their annoying daughter started to suspect things and someone just killed some undercover KGB agents and no one knows who. Keri Russell does a very good job jumping from Felicity to KGB agent, while Matthew Rhys comes out very well as a chameleon capable of impersonating just about anyone.
The Musketeers - You can't have pirates and not have musketeers. Loosely based on the Dumas books, this is a show about D'Artagnan being the coolest guy that ever lived and everybody taking an interest in him. Peter Capaldi is cardinal Richelieu. Strange choice, considering he is also Doctor Who, but very befitting the role. The show is not bad, but it feels so incredibly fake. I hope it will get better soon.
The Red Road - Jason Momoa plays in this as a bad ass native American who blackmails a cop in order to maintain his drug running operation. He plays well the role, looking all angry and violent and remorseless, but the movie subject is terribly boring. It feels like the type of show that gets cancelled and I personally have decided not to watch it anymore.
True Detective - An HBO project that features Woody Harrelson and Matthew McConaughey as cops in a God forsaken region of the US where religious fanatics and child rapists (sometimes the same person) roam free. Complex characters, eight episode seasons, very good acting and an interesting story make this one of the best shows around. It's not all rosy, though. In the first season more than three quarters was suspense and exposition. You always feel something will be going on, but it takes entire episodes before something does. It will be interesting to see if they continue with other seasons and, if they do, will they retain the same actors. The story arch has pretty much ended with the last episode of the first season.
Vikings - Season 2 has begun. Political intrigue, a new wife, backstabbing and a king of Britain who doesn't run with his tail between his legs at the sight of vikings. This season certainly keeps things interesting.
Suits - The series has never been about the law, not really, but about ritual. It was fun for a while, but every ritual, no matter how bizarre or exaggerated for dramatic purposes, can maintain outside interest only for so long. And the few law related references that made it interesting to me now are almost gone, together with the cowboyish feel of the show that made it seem less serious and more fun. Perhaps because all male characters in the series got hitched and their wings properly clipped off.
Continuum - Weird new direction of the third season of the series, with collapsing timelines, people meeting themselves and "the freelancers" being a cult like do gooder agency that protects time. I don't know yet if I like it, but it feels more confusing and less fun and sci-fi.
Crisis - A bunch of rich kids, including the son of the US president, are taken hostage by a shadowy mastermind who has thought of all possible outcomes before long before the actual execution and then uses the parents to achieve his goals, including more kidnapping of kids, I guess. Then there is this ex-cop Secret Service rookie black man who is set on finding the kids. So it's a kind of Die Hard, if you think about it. However the shadowy mastermind seems to have some moral agenda, revealing the evils of CIA, while the poor investigative agents are torn between their duty, their sympathy for the kids and their parents and the incredible stupid decisions people take in the name of their kids. I know Americans have this cult of parenthood and the need to do anything for your children, but too much is too much!
Da Vinci's Demons - The Pazzi's are trying to wrestle control of Florence from the Medici's, with Rome's involvement. That leads to some deaths, rearranging of loyalties and now both Da Vinci and Reario are heading towards the Americas on different ships, using a map of the location, only 40 years before Columbus blindly went to find the continent and with the help of none other than Amerigo Vespucci, who wasn't born yet. Nor was Da Vinci's friend, Nicolo Machiavelli, for that matter. Confusing? Only if you try to align the series with history.
Turn - This is a show about the first American spy ring, during the Revolutionary War. The first episode was rather slow, but it needed to set up the story and it showed good production values. Due to the actual nationality of the first Americans, most of the actors are British or Scottish, even if it is an AMC series. British acting with American money sounds good to me.
The Crimson Field - A six episode British miniseries about nurses in the war. The main action happens in a field hospital where four new volunteer nurses came to help out. It seems pretty decent and I already eagerly await the second episode.
Silicon Valley - An American sitcom about Silicon Valley, created by Mike Judge. Six programmers live in the same house when one of them invents an ingenious algorithm. Being an HBO series, with only eight episodes in the first season and created by Judge, I have high expectations from it.
In this post I will try to bring to your attention something that will probably change the world significantly. In 1909, German chemist Fritz Haber successfully fixed atmospheric nitrogen as ammonia in a laboratory and five years later a research team from BASF, led by Carl Bosch, developed the first industrial-scale application of the Haber process, sometimes called the Haber-Bosch process. Ammonia is extremely useful for many applications, the least of each is gunpowder and explosives and one of the most important is fertilizers. Without the Haber-Bosch process we probably wouldn't have the Green Revolution.
