Whenever you want to test a REST API, Postman is a great tool. It allows configuring all aspects of a request: Method (GET, POST, etc), Headers, keeps previous attempts in history, manages collections of requests and saves them and it is installed as a Chrome extension, bringing it only two clicks away. It does everything! ... or does it? Short story long: no!

Reported as a problem here: Referer header is not sent when set in Postman, the issue appears to be that some headers are "protected" by Chrome, therefore unusable. Well, it is a bug in the sense that Postman should tell you that when you write something there it is completely ignored! There is a solution, that can be found as a link in the bug report, but it involves installing other crap and running Python scripts. Ugh!

Here is a list of the Chrome protected headers:

• Accept-Charset
• Accept-Encoding
• Access-Control-Request-Headers
• Access-Control-Request-Method
• Connection
• Content-Length
• Cookie
• Cookie 2
• Content-Transfer-Encoding
• Date
• Expect
• Host
• Keep-Alive
• Origin
• Referer
• TE
• Trailer
• Transfer-Encoding
• Upgrade
• User-Agent
• Via

So whenever you believe that some web site has used a magical solution to detect your sneaky attempts to access their web API or site and you are wondering what, just remember that it is most likely a Referer header that Postman (via Chrome) silently ignored.

The Prince of Nothing trilogy was a total masterpiece, full of harrowing experiences of flawed men and women and characters so deep and original that they defied belief. You get the godlike Kellhus, not only freezingly rational and intelligent, but also mastering the Gnosis, the art of magic, while being a textbook example of a charismatic psychopath. You get Akkamian, a worldly sorcerer and spy, a teacher and a hopeless romantic. You get Cnaiür, a monstrous barbarian driven by revenge, but trapped by love. Esmenet, the whore empress mother, being heartbreakingly and treacherously a woman. And all these complex characters get to live in an epic world of different cultures, with politics, and military campaigns and evil creatures serving the No-god, shape shifting assassins and magic schools. In fact, it was so great that I found myself feeling dread of reading more, so terrifying the deep and personal pain of the characters that it was becoming mine.

Now I have finished The Judging Eye, the first book of the Aspect-Emperor trilogy, itself no more than a direct continuation of Prince of Nothing. R. Scott Bakker kind of cheats by using the same basic scaffolding for this story: a military campaign where an innocent and sympathetic character is being eroded by Kellhus' influence, a dark character driven by revenge on a lonely and dangerous quest, against himself slowly warming himself to the presence of a woman, court machinations driven by self serving creatures and the general backdrop of a clash between religions. He does it masterfully, though, switching the characters around and adding new ones to fill the roles left empty. In The Judging Eye one gets something as similar to The Darkness That Comes Before as needed to please the crowds that enjoyed Prince of Nothing, but as different to make it a completely different story. You get more of the same, so to speak, with an emphasis on 'more'. But also a little bit less.

Not everything is perfect. I feel that the inner dialogues of the characters have become more opaque, more strained. The metaphors flow just as in the previous books, but they communicate less, one makes more of an effort to get them and feel what the author meant, giving it a slight air of pompousness. I mean even a little kid philosophizes more than all the adults I know. The book has slightly more action scenes than I remember in Prince of Nothing, but they also feel more confusing. For all of Bakker's talent, I think he doesn't really understand combat and physical violence. He more than compensates with emotional and visceral violence, I agree, but I can't stop myself thinking of all the tactical applications that were never explored in battles purportedly fought by hardened veteran mercenaries.

His biggest sin, I believe, is that he doesn't follow through with the revelations that he awards the reader. I remember he was doing something similar in Prince of Nothing as well. He takes one through the labyrinthine mental processes of a character that marches uneventfully, but he fails to explain what exactly is happening - or at least what the characters are thinking - when something extraordinary happens. For example, in one scene, they discover a crazed individual with an eye on his heart, driven mad by the darkness that his heart is plunged into. A very powerful image. Yet Bakker doesn't explain anything. How was that possible? How did Akka know to look at his heart? What the hell were hordes of Sranc doing in a sterile dead mountain tunnel? No, instead we get to read about every single internal thought that the characters have about themselves and their lives. To quote from the book: "the work of a race that has gone insane for staring inward".

Bottom line: there is so much introspection in this book that barely anything happens. Falling into his own trap, Bakker gets seduced by his characters to the point of ignoring the universe in which they live. The book remains incredibly good and so I will venture to read the next one in the series. I already lament the fact that the third book of the Aspect-Emperor is not yet published and dread the moment when I will finish the second.

There is a nice web site called Chess Pastebin that allows one to publish a chess game as a PGN, comment on it using a Disqus form and share it with whomever you wish. However when I tried to use it, it refused to accept my PGN (extracted from Chess Arena). After many trials I realized that it considered invalid a move written as 10. .. some move instead accepting only 10... some move (no spaces between the dot of the move number and the dots representing a black move after a variation). So just replace ". .." with "..." to get a PGN acceptable by Chess Pastebin.

I've always heard this when I was young, as a Latin proverb - and therefore old and wise: in vino veritas. I've always interpreted it as "alcohol loosens the tongue", but today I had a revelation. Yeah, alcohol does remove some inhibitions, but usually the things we say to other people while inebriated are really dumb things that only a drunken man would say. They are not truth, they are wishful thinking, fears, pains. Instead, I propose that the real truth of the wine is you cannot so easily lie to yourself!

Indeed, I've noticed this in several situations, highly emotional ones or normal ones - it doesn't matter, when I have drunk alcohol and I am thinking to myself I always reach the conclusion that lying to myself, no matter how comforting, is not worth it, and I often expose and dispel things like hypocrisy, pettiness, delusion and so on. My best psychotherapy was always alone, drunk or comfortably inebriated, having the opportunity and courage to confront myself.

Now, that might seem boisterous or even something a drunken man would write. And that is true. However, it doesn't invalidate the argument. I have recently counseled a good friend who just lost his mother (he didn't lose her, he knows where she is, but she just died) to drink - alone - and speak to himself. I only wanted to help using my own experience, but that prompted me to think a little about it and that materialized into this blog entry. Drink a little with your friends, relax, chill, do whatever social thing you want to do, perform whatever ritual your tribe is comfortable with, but that only removes the stress. It does little else. At worst, it makes a fool out of you. True drunkenness is lonely and revealing and bitter. It is not pleasant, it is, at best - when done right, or when lucky - therapeutic.

That's my two cents about the subject, but I feel I need to explain a passage above: "comfortably inebriated". Sometimes, especially if confronted with strong emotions (or even boredom or gluttony, why not?), we drink too much. We don't consider the "alcoholemy", the amount of alcohol in the blood, the rates of absorption and so on. If there is a "sweet spot" a place where the quantity of alcohol in our blood is good for us, the only way to maintain it is to compute the ingested quantity compared to the quantity of blood one has and maybe some empirical factors like tiredness, personal resistance to intoxication, body mass, what you ate and so on. More simply: find the number of minutes that you can afford to drink a beer and then continue to drink beers every such interval so that you not get completely wasted. Of course, the equations are slightly more complicated, but you get the gist of it. I submit that you probably don't need to get completely drunk to reach that sweet spot, instead just research and find the perfect combination for you. More than a few times I got wasted after I had stopped drinking, as the alcohol in my guts was getting absorbed.

I may be wrong. There is always the dark specter of acquired resistance to any intoxicant, so that while the experiment may be perfectly scientific and true, one would need increasing quantities of the drug to get to the same result. However, empirical evidence of people who started drinking a little bit, then more and more, shows that there is a point where they stop and get the same result with similar quantities of alcohol. There is the sad case of alcoholics, but I believe that to be a small percentage of people experimenting with alcohol.

Anyway, the thing to remember: a few (more) beers could be as good as a year of therapy, if you are willing to drop the veil and be honest... to yourself. Anyone else wouldn't understand anyway.

Another really nice book I am glad I readlistened to. In 2001 Neil Gaiman published this book, American Gods, and then,ten years later, a special tenth anniversary edition, with the "author's preferred text" and including an additional 12,000 words, was published by William Morrow in June 2011. Both versions of the book were adapted to audiobooks, the first just narrated, the second with a full (and talented) cast. This second version I listened to, on YouTube no less. Long story short, I liked the book a lot, even if it was a bit too long winding.

The story is not only interesting in concept - gods being created and fed by people's beliefs, then being forgotten and dying as belief wanes, but also in its many details borrowed from the world's superstitions and religions. We start with a man in prison, a calm, cerebral man, but also big and muscular. He is about to be freed from prison and return to his wife and his best friend to start his life anew. From then he is just thrown into this crazy world of supernatural beings, but not like all these lazy TV shows where there are vampires and werewolves at war, but slowly and subtly. Gods are at the same time what we remember them to be: big spiders, thunderbirds, gods of war, but also men and women that can be killed. They can be pitiful beings, old creatures that resent the newness of the world and of its people. I also liked that there was not a lot of the major religions, just a few hints at the end. No Christian or Muslim stuff, I mean. At the end of the audio book there is an appendix, Shadow meeting Jesus, that the author wrote but decided to keep out of the book because it didn't feel like it was part of it. I agree with him.

There is also a kind of Lord of the Rings ending to the book. Even if the main story arches ended, there are still several chapters after that. I could have lived without them, I guess. This verbosity is also the problem with the book. Laden with details and side stories and keeping the pace slow can be slightly boring. I never read the version published in 2001, but it was shorter, which was probably for the better. I liked that Gaiman left a lot of hints in his writing and even I was able to see through them. That was maybe another reason why the pace felt too slow, since I've glimpsed a little of the whole story and then just waited for it to reach that point. However, that is the exact reaction of the protagonist, when he figures stuff out. He feels like he already knew, but refused to actually bring it into consciousness.

The adaptation of the book was top notch, the protagonist, but more than this the man called Wednesday, were interpreted by talented actors and I found myself drawn into the story like a man slowly being pulled down in quicksand. Give it a try.

It was announced by Starz in July 2014 that they are preparing an adaptation of the book into a TV series. Also, while there are other Gaiman books and stories in the same universe (like Anansi Boys, from 2005, which some call American Gods #2), the author announced he is working on a direct sequel of the book only in 2011.

Shinsekai Yori, translated as From the New World in English, is the anime adaptation of the homonymous book from 2008, written by Yusuke Kishi. It shows, too, as the subjects touched are deep, the characters are complex and the story is wonderful. It is a true sci-fi, not only set into the far future, but also using serious concepts such as what it means to be human, what is the price of peace and questioning if we can ever change as a species and as a culture.

It is a complete plot told in 25 episodes, well animated, but I wouldn't call the animation special, yet the story is certainly worth it. If you want to compare it with something, try a combination between The Village and some fantasy kid school movie. While it begins like a post apocalyptic version of Harry Potter, it quickly turns into a discussion about the sacrifices required to preserve peace. It doesn't just stop at the young adult audience, but continues with new and new twists until it feels like you have a collection of stories that just happen to follow one another, yet they are very connected. The film is filled with Japanese ways of seeing the world, from the absolute obedience towards authority to the horror they instinctively feel when talking about mass destruction, but also random cruelty based on a class system, or that sense of duty that permeates everything everybody does, or girls always stumbling or being interrupted by men when they talk and told what to do. However, it doesn't stop there and it explains, in a way, why things are like that and what are their consequences.

In the end, you feel like humanity has been deconstructed and its ways of functioning laid bare and put to trial. I liked the characters and the emotional rollercoaster the anime has put me through. Really nice, Hollywood should take heed on how to do a good story and put it into motion. I highly recommend it.

Update: This post discusses shrinking the data file of a Microsoft SQL database, caused in this case by misconfiguring the initial size of the database. For shrinking the log file one must at least use type 1, not 0, in the query. Also, a very pertinent comment from NULLable warns of the performance issues related to shrinking database files resulting from the fragmentation of the file.

I had this situation when the available space on the SQL database disk was less than the size of the database, in this case the temp database. Someone had wrongly configured the database to have an initial size of 64GB. Changing the size of the file in Microsoft SQL Management Studio doesn't work because it tries to create a different file, fill it with the data and then replace the file. No space for that. Also, it is damn slow, even if you have the space (I have no idea why). Shrink doesn't work either, because the database will not go smaller than the configured initial size. Time to do it command line style. Well, with sql queries, but you know what I mean.

The code for it goes like this:

USE [master];
GO

CHECKPOINT;
DBCC DROPCLEANBUFFERS;
DBCC FREEPROCCACHE;
DBCC FREESYSTEMCACHE ('ALL');
DBCC FREESESSIONCACHE;
GO

USE [tempdb]
GO

DBCC SHRINKFILE (tempdev, 3000);   --- New file size in MB

As you can see, you need to know not only the name of the database, but also the logical name of the database file that you want to shrink. It is not even a string, it is like a keyword in the DBCC SHRINKFILE command. Even if it does work, one would benefit from encapsulating it into a stored procedure. Here is the final code:

CREATE PROC ShrinkDatabase(@DbName NVARCHAR(100),@SizeMB INT)
AS
BEGIN


DECLARE @filename NVARCHAR(255)

DECLARE @sql NVARCHAR(Max) = 'SELECT @filename = dbf.name FROM ['+REPLACE(@DbName,'''','''''')+'].sys.database_files dbf WHERE dbf.[type]=0'
EXEC sp_executesql @sql,N'@filename NVARCHAR(255) OUTPUT',@filename OUTPUT

SET @sql='USE [master];
CHECKPOINT;
DBCC DROPCLEANBUFFERS;
DBCC FREEPROCCACHE;
DBCC FREESYSTEMCACHE (''ALL'');
DBCC FREESESSIONCACHE;'
EXEC sp_executesql @sql

SET @sql='USE ['+REPLACE(@DbName,'''','''''')+'];
DBCC SHRINKFILE ('+REPLACE(@filename,'''','''''')+', '+CONVERT(NVARCHAR(100),@SizeMb)+');'
EXEC sp_executesql @sql

END

Create it in the master database and use it like this:

EXEC master.dbo.ShrinkDatabase 'tempdb',3000

Take note that you cannot use this to "shrink up" the database. If the value you set is larger than the current size, the file will remain the same size as well as the setting for the initial size. Also take note of the fact that this stored procedure only shrinks the data file, not the log file (dbf.[type]=0).

Just a quick reminder that there are a lot of Star Trek productions out there, some really terrible, some really good. I am going to talk about three(ish) of them today.


First one is Star Trek: Federation One. It is with a lot of actors you have seen in other Hidden Frontier and Studio Areakt productions and focuses on the chief security of the presidential office of the Federation. While the plot is kind of a mess, the acting was decent and I was actually very sad to see that after the first "season" of two episodes they continued the story in an audio format. Even if the main character looked like he was on speed all the time, I had fun with it.


Another one is Star Trek: The Infinite Chain. Even if it is split into "episodes" it is actually a feature film. The acting is a bit amateurish, but decent. No obese teens in this one. It uses the now tired plot of a Federation starship thrown into unknown space by "an anomaly". The worst part of it was the "doctor" that was assigned to the mission. His character was believable: the guy in charge of the project creating the sensors used to study the anomaly, but in truth not a good manager or having any merit in the team. He just explodes randomly in childish emotional outbursts, threatening everybody with his artificial authority, and everybody hates him. I totally know people like that. However the actor was just goddamn awful and his script lines ridiculous! The makeup was ingenious, but really cheesy.


The last, but certainly not least, is something made by the Hidden Frontier team, but not set in the Star Trek universe. I am talking about Frontier Guard. The casting is made with the better actors of previous productions and, while the universe seems very similar to the Star Trek one, it is actually a different thing altogether. They use another type of space propulsion, it involves a FederationFrontier Guard academy and it concerns The Arc, a huge alien artifact that may have seeded the galaxy with intelligent life. Unfortunately, after some great episodes, they completely lost it. They began (some) episodes with "My name is..." evoking feelings of remembrance towards all the bad DC superhero series and movies, they started pointless subplots like the gay relationship thing - which always pisses me off, not because of the gayness, but because of the relationship that doesn't further the plot in any meaningful way - and threw in the towel after the 12th episode (which was an elevator show, basically).

From these three Frontier Guard was the best. My guess is that they wanted to do a Star Trek Academy series - which makes total sense and it should be the direction of the next Star Trek series - but they didn't get the approval. Since this sounds like a really good idea and one that might attract a lot of fans, I believe studios are keeping the story in reserve. However, not being Star Trek allowed them to evolve both stories and characters and I was really excited to see this done. In a world where everybody is remaking remakes, we need and deserve original content. Just look at David Feintuch's Seafort Saga series of books. That would make a fantastic TV series and has enough material to allow for several companion feature films.

For people just considering starting to watch these, do not expect Star Trek Next Generation. The actors are amateurs, I mean real amateurs not professionals who just started acting, most of them are American Trekkies and - even if I don't want to insult - some of them are just humongous pieces of fat with big glasses, which kind of explains the whole geek high school thing that is kind of difficult to understand in Europe. The stories, the lines, the makeup, the uniforms, the special effects, they are all done by enthusiasts. Sometimes they feel cheesy, sometimes they look completely and horribly fake. But once you realize that they are people like you, enjoying the wonderful universe of Star Trek, you can begin enjoying these productions.

I've watched other things as well, including Machinima style animations like Borg War - which I wouldn't recommend, but wasn't bad, and I will continue to look for these things. Just by looking at people like Hidden Frontier, who start doing something for the fun of it and end up doing original stuff with veteran amateur actors, I get filled with hope for the whole fan made universe, not just the Star Trek one. Keep up the good work guys (and if you can't do it good, keep working and it will get there eventually :) )

The best thing about it is that most of these series are online, on YouTube, free to watch. Here are some links to the shows above and others:
Star Trek Federation One - 1.01 Unity
Star Trek Federation One - 1.02 Institutions

Star Trek: The Infinite Chain

Frontier Guard

Star Trek: Digital Ghost

Special mention: Star Trip - a humorous parody of Star Trek.

I had this situation where I had to match a string by several regular expressions and choose the most specific match. Let me explain it a little bit further: you have a string and several regular expressions that match that string. The requirement is to choose which one matches better. This would be useful if, for example, you want to match mobile manufacturers and so you define one configurations for Apple and another for Android and another for the rest. The regular expressions would be "Apple", "Android" and ".*" (which matches everything). So you would want to choose only "Apple" for Apple devices and not ".*".

Now, I've studied regular expressions a bit, but I don't want to implement a regular expression parser in order to compute the complexity of the expression tree. What I want is to have various Regex objects and, somehow, compare their complexity. I am not going to pretend that I understand what goes on within the Regex object, instead I will say that I noticed the complexity of a regular expression is directly proportional to the length of the pattern and, in the case of the Regex object, with the number of items in the _codes int32 array found in the code RegexCode object found as a private field of the Regex object.

So, the simplest code for that is this:

private double regexComplexity(string pattern)
{
    var reg = new Regex(pattern, RegexOptions.Singleline | RegexOptions.IgnoreCase);
    var codeField = reg.GetType().GetField("code", BindingFlags.Instance | BindingFlags.NonPublic);
    var code = codeField.GetValue(reg);
    var _codesField = code.GetType().GetField("_codes", BindingFlags.Instance | BindingFlags.NonPublic);
    var codes = (int[])_codesField.GetValue(code);
    return codes.Length;
}

However we must take into consideration several cases:

• The regular expression pattern might be wrong
• Reflection is slow

So, let's first catch all exceptions and use the length of the pattern as the complexity and then find a way to cache the complexity of a string, in case we use it more than once. Also, cache the FieldInfo objects for use at every call.

Here is the final code:

public static class RegularExpressionExtensions
{
    private static ConcurrentDictionary<string, int> _dict;
    private static FieldInfo _codeField;
    private static FieldInfo _codesField;

    static RegularExpressionExtensions()
    {
        _dict = new ConcurrentDictionary<string, int>();
        _codeField = typeof(Regex).GetField("code", BindingFlags.Instance | BindingFlags.NonPublic);
        _codesField = _codeField.FieldType.GetField("_codes", BindingFlags.Instance | BindingFlags.NonPublic);
    }

    public static int Complexity(this Regex reg)
    {
        if (reg == null) return 0;
        var pattern = reg.ToString();
        var complexity = _dict.GetOrAdd(pattern, p => getComplexity(reg));
        return complexity;
    }

    private static int getComplexity(Regex reg)
    {
        var code = _codeField.GetValue(reg);
        var codes = (int[])_codesField.GetValue(code);
        return codes.Length;
    }
}

Basically we encapsulate the Regex complexity in an static extension class that we can use on any Regex instance as reg.Complexity().

Just wanted to post these really funny Star Trek fan made animations.

South Trek - a South Park animated Star Trek:


Star Trekkin' song:


Stone Park - a Flinstones animated Star Trek:



Some other funny stuff:

Today I published a raw version of a program that solves Pixelo puzzles, a Flash version of the game generally known as Nonogram, Picross, Hanjie, etc.. I was absolutely fascinated by the game, not only the concept, but also the attention to details that Tamaii lent to the game. The program is called Pixelo Solver and you can find it at Github, complete with source code.


I liked working on this because it covers several concepts that I found interesting:

• Responding to global key combinations
• Getting a snapshot of the screen and finding an object in it
• Parsing a visual image for digits in a certain format
• The algorithm for solving the puzzle in a reasonable amount of time and memory (it was not trivial)

Quick how-to: get all the files from here and copy them in a folder, then run PixeloTools.GameSolver.exe and press Ctrl-Shift-P on your puzzle in the browser.

Usage:

1. Start PixeloTools.GameSolver.exe
2. Open the browser to the Pixelo game
3. Start a puzzle
4. Press Ctrl-Shift-P

However, if the list of numbers per row or column is too long, the automated parsing will not work. In that case, you may use it offline, with a parameter that defines a file in the format:

5 1 10, 3 3 7, 2 5 4, 2 5 2, 2 1 1 1, 1 1 1 1 1, 1 1 3 1 1, 1 1 1 1 1, 1 1 1, 1 1, 1 1, 2 2 2 4, 2 1 1 1 2 4, 2 1 2 1, 7 4, 2 2 1 2 2 2, 2 1 1 1 1 1 1, 2 1 1 4 2, 1 3 4 2 1, 1 1
8 1 1, 5 2 2 1, 2 1 3, 1 1, 1 7 4 1, 3 5 1, 3 1 1 3, 4 3 4, 3 1 1 3, 3 5 1, 1 7 4 1, 1 2, 1 1 1, 2 2 2, 2 1 1 2, 2 1 2 2, 3 2 1, 3 1 1, 4 1 2 2, 9 2 3

where the first line is the horizontal lines and the second line the vertical lines. When the parsing fails, you still get something like this in the output of the program. You can just copy paste the two lines in a file, edit it so that it matches the puzzle, then run:

start "" PixeloTools.GameSolver.exe myFile.txt

The file can also be in XML format, same as in the file the game loads. That's for Tamaii's benefit, mostly.

Let me know what you think in the comments below.

As some of you know, I am watching a lot of TV series, including cop and lawyer stuff, childish fantasy and sci-fi series, pirate and musketeer crap, etc. So when I say that I couldn't watch something, you should heed me. There are also some shows that I actually can stand watching, but barely, usually using the fast forward to watch an episode in ten minutes or less.
Here is a list of just such shows:

• Lost - I think it all started with Lost. After a brilliant pilot episode, it got into 1001 Arabian Nights territory and became ridiculously unwatchable. While it wasn't a truly bad show (some people actually enjoyed all 6 seasons) I couldn't watch it more than 2 seasons. Calm down, the rest of the shows will be more recent.
• The Messengers - every once in a while the American Bible thumpers convince producers to create religion based TV series or movies. While every possible superstition has and will be used in films, the Christian ones are standalone - in their own cinematic universe, if you will - and take themselves seriously every time. Messengers is one of these, and I couldn't watch more than 2 episodes before I gave up. It's (again) about the upcoming apocalypse and end of days and these 8 idiots are given superpowers to convince God that we are still worth it, while Satan goes around doing silly things. It was not only about the most tabloid part of (a part of) the bible, but it was acted and scripted badly. Ugh!
• Dig - This relates more to Hebrew religion and feels like a TV series take on the Da Vinci Code, with clones of Jesus (I think), FBI policemen running around in Israel doing whatever they feel like, shady cabals and shady opposing cabals. It could have been interesting and I admit that I was drawn into it by the acting and production values. Unfortunately the script was a convoluted mess that had no way of improving. They never learn to ease into it, present a little bit of the story, add another element later on, make people feel rewarded for continuing to watch the series. Anyway, four or five episodes in, I gave up.
• iZombie - Oh, you have to love this: young blonde accidentally gets turned into a zombie and gets a job at the morgue in order to have access to a fresh supply of brains. While consuming said brains, she gets visions of what the brain's owner had lived and thus helps a police detective to find out who murdered them. It combines police procedurals with zombies, comedy, romance and drama. Talking of messes, this is as messy as they come. I watched a few episodes before I slapped myself "what the hell are you doing watching this swill?"
• Eye Candy - A darker show about a girl using her hacker tech skills to try and catch the serial abductor and killer of young women, including her sister. Yet the killer is just as able and a game of cat and mouse ensues. It started well, then went into pop culture territory. Besides, she continues on only because the killer refuses to murder her and get on with his job. Something about the way the film was made or the main actress or the ridiculous plot made it impossible for me to get past the second episode.
• Allegiance - It is strange to put this series here, as I watched all the episodes so far. However, it was hope that kept me going, not the quality of the show. While it is about Russian spies in the US, living as a family, while their unknowing son is recruited into the FBI, while the antagonists are interesting, the story goes so much over the top that it becomes overcheesy. That means it starts as cheesy, then it starts to smell. Just do yourself a favor and do not watch it.
• Suits - Yet another flashy show that I kept watching for no good reason. It is about lawyers. Young, cocky, caught in personal intrigues and competing with one archnemesis after another, while occasionally engaging in some silly romance. It makes no sense, although I have to admit that I liked the feel of the show. It just goes nowhere. They get the mood right, the actors, the characters, then they make a joke of the storyline and script.
• Helix - Helix is the new Lost! It starts with a subject close to my heart: killer viruses, and then it crashes into la-la-land. Viruses that make people immortal and strong so they form a world controlling organization, CDC teams that act like they are terrorist splinter cells with no support, no rules, no communication, strains of the virus that turns people into The Strain like vampires, religious cults that are also genetic scientists living on their own island, etc. It starts as something, goes into something else, switches back. It's like someone was undecided what to write about and they wrote about everything then they just drew lines between the unrelated subjects and wrote a script. It is a complete and utter waster of time.
• Elementary - I watched two seasons of this until I admitted to myself that there is no point in continuing. I like both Johnny Lee Miller and Lucy Liu, I love Arthur Conan Doyle's Sherlock Holmes character and so a modern Sherlock should have been something I would at least enjoy. Unfortunately, it shortly turns into another "gifted person helps the New York police" show and one can feel when the writers just completely stopped giving a damn, if they ever did before beyond pushing the pilot.
• Scorpion - Talking about gifted persons helping the police, this had a promising concept: a genius young man creates a sort of a thinktank of gifted people then starts helping his FBI parent surrogate. While the characters were brilliant, the people writing their script obviously were not. It took a few episodes to completely fail and another ten for me to just give up hope.
• The Librarians - God, what a complete and utter mess. A TV show spinoff of the ridiculous Librarian movies, it was completely stupid. I mean, the movies were masterpieces compared to the writing for this show. Stay away!
• The Newsroom - This is a show that has ended and I watched all of its episodes. However, it was under duress. My wife liked it, but then I could see that she couldn't watch it anymore. I watched the last mini-season just in order to tell her it was not worth watching. It's about a news organisation that fights for its moral integrity. It would have been fun if not for the hyperactive writers that thought people could normally carry conversations like machineguns shoot bullets and the ridiculous romantic interludes that made no sense for anyone but a New Yorker raised on Woody Allen movies.
• State of Affairs - I already ranted enough about this piece of propagandistic crap. The plot was stupid, the script was stupid, the actors couldn't give a damn. It's just bad!
• The 100 - In theory I am still watching this young adult sci-fi TV show. In reality, I can't make myself start watching the second season. Why? Because the sci, as usual, gets completely negated by the fi. Add to this hormone induced behaviour of teenage kids and savage barbarians and you get a load of messy crap. It started really well, though. Too bad.
• Extant - Another sci-fi show in which they dumped randomly whatever was hot at the moment: aliens, artificial intelligence, the Japanese guy from Helix, etc. Add to this the excruciating acting skills of Halle Berry and you get something unwatchable.
• Stalker - This is one of the more interesting police procedurals. It involves a team that is tasked to handle stalkers. The new hire of the agency may be a stalker himself. The lead actors are people that I respect. However the story was so banal and lacking anything that showed any concern for the lives of the characters or victims, it was so formulaic that it ended up being something I regret having watched.
• Arrow - OK, I am still watching The Arrow, but I do it on fast forward. It started as a fun comic book TV show with a vigilante dressed in green killing assholes with the use of arrows. It then started growing and expanding until it reached a full team of heroes, with impossible situations, history, skill learning curves, ridiculous villains. It culminated with John Barrowman having a line in the show in which he tells three other characters he is the better man for a job because he is the better at acting. Which is true, but it just shows that even the scriptwriters are tired of the mediocre talents in this mediocre show. I keep watching it because it is set in the same universe as The Flash, which is the better TV show, even if not by much.
• Gracepoint - When Americans are redoing a show with the same script as the original show from a different country I snicker. Stoopeed Americaens, they can't read subtitles. But Gracepoint upped the ante: they redid Broadchurch, a British TV show. Not only that, they cast the same actor as the main character. I love David Tennant, but I can't possibly condone watching this. Anyway, it was cancelled after a season, and then Broadchurh continued for a second season, also starring Tennant.
• Sleepy Hollow - An entire TV series based on the Legend of the Sleepy Hollow. Starring a colonial Mason that fought for and knew the forefathers of the United States, resurrected in our times and fighting along a black female police officer to stop the End of Times. What can possibly go wrong here? Yet it wasn't that clear cut. The comedy was fun, the original drama had impact, the romance was believable and John Noble, like in Fringe, was awesome. However the storyline, like in Fringe, descended into a chaotic mess that makes no sense and no one could ever care for. Too bad, since it wasn't a complete waste of time, especially at the beginning.
• Madam Secretary - Just as with State of Affairs, I've ranted enough about it. It's a pure propaganda piece, with no artistic value whatsoever.
• Haven - It started as a nice supernatural police procedural with a sexy FBI agent coming in a town where people get dangerous "troubles" which act like superpowers out of control. I kept waiting for it to get anywhere, instead it just went more and more overdramatic and adding storylines that made no sense and in the end turned into a supernatural 90120. Too bad, since it started well and has some actors that I really like.
• Forever - Forever boring, this show is about a British doctor who never ages and, whenever gets killed, he finds himself alive and well in a nearby body of water, completely naked. And what does he do with that? He helps the police, as a forensic medic. Just as a general rule: avoid the gifted people helping the police catch bad guys shows. They never end well and they are as well propaganda pieces.
• The Lottery - Badly made show. Bad casting, bad acting, bad scripting. The plot: people stop having children, embryos stop getting fertilized and a woman doctor finds a way. Obviously, the powers that be want to control this in a market of dwindling supplies. Could have been nice, probably, although shows about children and parently feelings never do well in the long run, but it was pure shit from the beginning.
• Taxy Brooklyn - Someone tried to combine the French Taxi storyline with a police procedural. Sexy ass police detective, a Black French taxi driver, some intrigue and of course the obligatory connection with an old case involving a relative. But it was bad. The actors did what they could with lazy writing, but they couldn't save it.
• The Witches of East End - Some of the most beautiful women in acting get together with ridiculously good looking men in order to act a really stupid Dynasty-like plot. Just like similarly afflicted The Originals, it is a complete waste of time, unless you are a hormonal teenager and you need something to jack off to.
• The Leftovers - Trying to express the horrible anguish of being among the ones left on Earth after 2% of the population randomly disappears. Of course, it hints of another Bible concept: the Rapture. It felt to me overly dramatic, with people just doing stuff for no good reason and overly depressing as well for equally no good reason. It seemed to be well acted, but I just couldn't watch it.
• Resurrection - an American remake after Les Revenants, it missed the point completely. Ended up being this drawn out mystery that led nowhere, with really unsympathetic characters and without any of the real tension from the French series.
• Da Vinci's Demons - I know, I am still watching it, but not because it is worth watching. It is about a young Da Vinci who does everything from saving Florence to saving Italy and then going to the Americas (way before it was discovered) to look for clues about his mother, who was there previously. It is a fun to watch show, but it really goes nowhere. I liked the acting and the production values, though.
• Crisis - An intriguing concept: some very resourceful people kidnap a lot of high profile adolescent kids, including the son of the president of the United States, then they use the parents to achieve their goals. While the show was well done, the story was just a lazy fantasy trying to surf the edge of suspension of disbelief. Again, it goes nowhere.
• The Tomorrow People - An American take of an old British children's TV show that wasn't good to begin with, it only served to promote certain actors to better TV shows. It used the common tropes: child with mysterious father who dies doing something awesome, mother unknowing, uncle being involved somehow, evil organizations, rebellious teenagers fighting them, etc. It just collapses under the ridiculous ideas that they introduce to appease a boring viewership.
• The Originals - Some spin-off from a vampire show that I never watched, it is about pompous asses acting even more pompous because they are the original vampires. Werewolves and witches are thrown in the mix, but the only worth mentioning attribute of the show is the pompousness. Which is annoying. So don't watch it.
• Killer Women - Trying to get on the diversity bandwagon and get some easy money, it was supposed to be a police procedural with tough women. Didn't pan out. Was cancelled.
• Bitten - One of the most beautiful actresses around is a werewolf. Instead of being about her biting more than she can chew (heh heh), it is about another clan/royalty obnoxiousness, while the world is blissfully unaware and rivals are trying to take power or kill them or whatever. Do you know why people don't officially acknowledge vampires and werewolves? Because they act all annoying and overbearing while being completely idiotic. Nobody wants to know these people!

OK, as an interesting experiment on my preferences vs the public preferences, I will tell you how many of the shows above were cancelled and how many were not. Ready?

Completed (meaning not cancelled, but finishing their story): 2
Cancelled: 16
Still Running or undecided: 15

So there you go, studios think 50% of the shows that I can't stand should be running and they are basing their decisions on public choice. What do you guys think?

Relational databases have the great advantage of being able to use indexes. These are constructs that trade space and the speed of inserts and updates for query speed. This introduces a problem: what do you do when you have a high input and high output application? How can you make your database fast for both changes and queries?

The traditional solution is a write-database and a read-database. There are background services that ensure that the read-database is updated as fast as necessary with the new data, most of the time also doing extra computation and caching for the information one needs extracted. OLAP systems are a common solution, where the aggregated data required by various reports is computed and stored, ready to be read by your applications. I am not saying this is a bad idea, in fact I am saying this is the best idea for this scenario. The problem that I have with it is that you can hardly automate the process. You need to know what you want to read, you need to write the software to create the data and aggregate it into what you need.

So I decided to try to build a system that obeys the following rules:

1. The speed of insert and update operations needs to be unhindered by indexes. Indeed, changes to the original data should be avoided.
2. The select operation need to benefit from the indexing system.
3. The system must be able to determine by itself the data structures needed to cover the first two rules.
4. Optionally, there should be a way to encapsulate this new behaviour into a separate data structure from the original database.

In order to insert and update as fast as possible, I needed tables that only had primary keys, identity integers rather than uniqueidentifiers, as they lead to fragmentation of clustered indexes. In order to not only index the columns that are used in where and join conditions, but also encapsulate the behaviour in some other data structure, I decided to create shadow tables with the columns that I needed for querying. These tables I would then index in order to improve selects. The connection between the original insert table and the select table would be made via primary keys and the synchronization between the two types of tables would be done in the background, whenever needed. Best of all, analysis on execution plans could automatically determine the columns needed for this system and create the tables and indexes required, then suggest improvements on the observed queries.

In order to test my theory I created the following tables:

• OriginalTable - with a primary key identity ingeniously called Id and two other columns called Latitude and Longitude, containing random decimal(18,6) values
• LocationIndexTable - with a refId integer primary key column that links to the OriginalTable Id - without being a foreign key - and two Latitude and Longitude columns that are indexed by the same non clustered index
• LocationIndexTable2 - with a refId integer primary key column that links to the OriginalTable Id - without being a foreign key - and a locId integer column that links to another table called LocationTable and has its own non clustered index
• LocationTable - with a primary key identity column and Latitude and Longitude columns. I know that this looks identical to LocationIndexTable, but this is the more complex case where there are multiple records with the same location. LocationTable holds the distinct Location and Latitude values
• LocationIndexTable3 - with a refId integer column that links to the OriginalTable Id and is indexed by its own nonclustered index - without being a foreign key - and two Latitude and Longitude columns that are indexed by a clustered index

With a number of 77179072 original table rows, I attempted the queries for each case, careful to clean the cache and memory buffers before that. Here are the results:

• SELECT count(1) FROM OriginalTable WHERE Latitude BETWEEN 45.5 AND 46 AND Longitude BETWEEN 8.5 AND 9 - no indexes whatsoever. Result: 30 seconds
• SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable lit ON lit.RefId=ot.Id WHERE lit.Latitude BETWEEN 45.5 AND 46 AND lit.Longitude BETWEEN 8.5 AND 9. Result: 17 seconds
• SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable2 lit2 ON lit2.RefId=ot.Id INNER JOIN LocationTable lt ON lit2.LocId=lt.Id WHERE lt.Latitude BETWEEN 45.5 AND 46 AND lt.Longitude BETWEEN 8.5 AND 9. Result: 41 seconds
• SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable3 lit ON lit.RefId=ot.Id WHERE lit.Latitude BETWEEN 45.5 AND 46 AND lit.Longitude BETWEEN 8.5 AND 9. Result: 22 seconds

Unexpectedly for me, the most comfortable solution also seems the faster. Indeed, one issue is that I didn't have duplicate location data in the database, so there was no advantage in adding a new table to link locations to the original table. That means that in cases where the indexed data has many duplicates, it might be advantageous to experiment with a "distinct" table, although indexing should take this case into account, as well. The clustered index is slower than the unclustered one, that was a surprise. Also, the indexing just made the query twice as fast - I had expected more. Of course, all this benchmarking was done after deleting the cache and buffers with the commands DBCC FREEPROCCACHE; DBCC DROPCLEANBUFFERS. It is interesting to see how fast they queries go without this clearing. The first unindexed query takes 3 or 4 seconds, while all the others take less than 2.

There is one more thing that needs to be addressed: moving these tables into another database. Are the indexes just as fast? They should be, but we must test interdatabase communication. This would allow to move the entire system outside a target database, truly encapsulated, really adding value to a database that remains unaffected. My tests show the following results: 28, 18, 29 and 18 seconds, respectively. Yes, you saw that right, the speed of the joins when the indexed databases are in another database on the same server seems faster! Just to make sure I reran the original tests on the same database and I got approximately the same results: 29,19,43 and 24, respectively. The only thing I didn't try (and I don't intend to) is to create primary-foreign key associations, as this means modifying the original tables, something I wish to avoid.

So, after all this I believe my idea has been validated. A lot more work has to be done in order to automate this system, but at least a no-effort manual solution is possible. Let's recap:

1. The speed of row inserts and updates remains unchanged: the only index is the original primary key identity integer key that should always exist in a table anyway.
2. The speed of selects is improved by creating tables that have an integer primary key that links to the original table, and only the columns used in queries, over which indexes are created.
3. Queries can be used to determine the columns needed to index. Even better, computed columns can be used instead of the original columns, which adds a little more performance.
4. Encapsulation is achieved by not only creating other tables for reading, but also moving these tables into another database, after which, unexpectedly, the performance is even better.

The end result is a system similar to indexing, but which takes space in another database and which is updated on demand, not when the system deems it necessary. Any SQL statements that worked before will continue to work unaffected, but faster alternatives for selects will become available. Overall, I am very satisfied, although I had expected better performance improvements than observed.

There are these time periods in which movies and TV shows start to follow a common theme. I don't know exactly why that happens, but I suspect it's about Hollywood corporate managers stealing everything that seems to work and getting to copy each other. We seem to be living under the sign of the Spy right about now and in this blog post I will be discussing the following TV series (listed in alphabetical order):

• Allegiance - a show about a CIA analyst who is the unknowing son of a pair of Russian KGB spies. He is also a kind of idiot savant which can remember a lot of things and make connections that no one can. It gets really hard to believe soon as it mashes up family drama with evil SVR (former KGB) machinations, CIA and FBI operations, traitors and double agents, hired assassins, plots to bring down the United States, and so on. It's one of the lesser spy shows and I don't recommend it.
• The Americans - I believe this is the gem of the lot. The protagonists are two fake Americans, in reality KGB agents. They live like an American couple, with the mandatory two children and their neighbor is an FBI agent in the very taskforce that is supposed to catch people like them. The acting is very good and the show is a serious one, going only occasionally over the top. Also the lovely Keri Russell makes it very appealing as well :)
• The Assets - The Assets has ended after a single season, with the successful completion of the main plot. It was a strange mix of really good acting and rather weak characterisation. It followed the story of Aldrich Ames, a famous disgruntled CIA operative who sold American secrets to the Russian KGB. As it is a true story, the entire plot was not terribly captivating, but the problem for me was with the characters.
• Turn - About the spy ring of the American rebels against the British during the Washington era, it is an intricate and interesting story. I like most of the characters, even the villains, which is rare in films these days. I particularly like Burn Gorman in this, even if he plays this British buffoon that understands nothing of what is going on around him.
• X Company - Set during World Word II, it is about a company of spies working in the German occupied territories. Their newest recruit is a guy who can remember everything he sees, but is a little weird. Sound familiar? Heh

From the list above, The Americans shines bright. Good acting, good directing, but also a dedication to show the political and social issues of the time. From the ones here, it is the only one who I feel can teach one something about history and about human nature in general. Turn is also rather good, but it is too reliant on a number of characters that keep ending up crossing paths in implausible situations, and many times goes a little over the top. The actors are really good in this one, though, and it is worth watching for sure. X Company is more like the typical WWII movie, with the evil Germans and the brave resistance fighters. They do attempt occasionally to enter the grey areas, like good German officers, conflicted collaborators or remorseful secret agents who have to compromise between human values and the mission, but it is not enough. When the beautiful team leader played by Evelyne Brochu reveals that she is half Jewish and the whole story goes into Holocaust territory you can feel the show going south. The Assets was brave in not employing only beautiful actors. The main character was ugly and weak and constantly fearing discovery, hard to empathize with. But so were the people who were trying to catch him, led by an obsessive woman that acted like a blood hound the entire time. Not being able to make an emotional connection to any of the characters hurt the success of this otherwise interesting story. This leaves Allegiance, which feels a lot like a fairy tale combined with a police "special" procedural (you know the type: special person helps police solve crimes) which incidentally features spies. While I cannot really recommend X Company, Allegiance is the only one that I would recommend against.

Worth mentioning is that three of these shows are based on real events. The Assets follows the exploits of Aldrich Ames, X Company is loosely based on Camp X and Turn links to the story of the Culper Ring.

That's it for pure spy shows. I am not putting in this list shows like Homeland or Legends, which also show undercover operatives, but are not specifically aimed at spying as the main subject of the plot. Tell me what you think.

Another fantasy audio book, The Black Company is the first of ten of the ongoing series with the same name. Glen Cook probably thought about it for a while before writing it, because you can see that the first three books in the saga were published mere months apart (*cough*R.R.Martin*cough*). I was attracted by the story, because it seemed similar to my favorite Malazan Book of the Fallen series: a mercenary company fighting under the flag of powerful god-like magical beings that want to conquer the world. And in part the plot and characters were satisfactory. Alas, the many details and focus on small things like what games they played to pass time or what each character was like or maybe the narration style... I don't know; it all made it hard to focus on the story. I felt like the book could (and should) have been half in size, and better for it.

The main character is the doctor and analyst of the company, Croaker. Before your filthy mind thinks of other things, analyst in this context means he is a keeper of records, a chronicler of the company's history and, occasionally, the person who draws parallels between past experiences and current events. Croaker is an inquisitive person, often risking his personal safety to unlock a riddle or reveal a secret. Unlike Malazan, he is the lead character, through and through, and the company itself with its many soldiers and camaraderie just the backdrop.

Plot follows the Black Company in employment of The Lady, a powerful magician who wants to conquer the world, fighting the Rebel, a group of lesser magicians who have gathered the people of the land in response. Croaker has romantic fantasies about her, but throughout the book he discovers that his affections are misdirected, even if a strange relationship develops between them. Most of the story is about various battles, painstakingly (and painfully) described, only to be followed by the occasional interesting bit of character and plot development. In fact, I was kind of annoyed when I read the book summary on Wikipedia and I couldn't think of much that was left out. I mean, it's a big book.

The bottom line is that I am unsure if I want to listen to the rest of the series. While I can't say it was a bad book, or that is had weak plot or characters, I am reluctant to go through all that again for nine times. Basically I feel that the characters were never made empathetic enough, at least for me, and in the end all I cared about was what was going to happen next and how it would all end. In that case I would be better off reading synopses rather than listening to the entire thing. The overall structure of the story, though, has a lot of potential and I don't know if the series would not become really cool afterwards. To make the final parallel with Malazan, the first book in that series was not really making people eager to read the next, but it turned out to be great overall. I guess time will tell.