Shinsekai Yori, translated as From the New World in English, is the anime adaptation of the homonymous book from 2008, written by Yusuke Kishi. It shows, too, as the subjects touched are deep, the characters are complex and the story is wonderful. It is a true sci-fi, not only set into the far future, but also using serious concepts such as what it means to be human, what is the price of peace and questioning if we can ever change as a species and as a culture.

It is a complete plot told in 25 episodes, well animated, but I wouldn't call the animation special, yet the story is certainly worth it. If you want to compare it with something, try a combination between The Village and some fantasy kid school movie. While it begins like a post apocalyptic version of Harry Potter, it quickly turns into a discussion about the sacrifices required to preserve peace. It doesn't just stop at the young adult audience, but continues with new and new twists until it feels like you have a collection of stories that just happen to follow one another, yet they are very connected. The film is filled with Japanese ways of seeing the world, from the absolute obedience towards authority to the horror they instinctively feel when talking about mass destruction, but also random cruelty based on a class system, or that sense of duty that permeates everything everybody does, or girls always stumbling or being interrupted by men when they talk and told what to do. However, it doesn't stop there and it explains, in a way, why things are like that and what are their consequences.

In the end, you feel like humanity has been deconstructed and its ways of functioning laid bare and put to trial. I liked the characters and the emotional rollercoaster the anime has put me through. Really nice, Hollywood should take heed on how to do a good story and put it into motion. I highly recommend it.

Update: This post discusses shrinking the data file of a Microsoft SQL database, caused in this case by misconfiguring the initial size of the database. For shrinking the log file one must at least use type 1, not 0, in the query. Also, a very pertinent comment from NULLable warns of the performance issues related to shrinking database files resulting from the fragmentation of the file.

I had this situation when the available space on the SQL database disk was less than the size of the database, in this case the temp database. Someone had wrongly configured the database to have an initial size of 64GB. Changing the size of the file in Microsoft SQL Management Studio doesn't work because it tries to create a different file, fill it with the data and then replace the file. No space for that. Also, it is damn slow, even if you have the space (I have no idea why). Shrink doesn't work either, because the database will not go smaller than the configured initial size. Time to do it command line style. Well, with sql queries, but you know what I mean.

The code for it goes like this:

USE [master];
GO

CHECKPOINT;
DBCC DROPCLEANBUFFERS;
DBCC FREEPROCCACHE;
DBCC FREESYSTEMCACHE ('ALL');
DBCC FREESESSIONCACHE;
GO

USE [tempdb]
GO

DBCC SHRINKFILE (tempdev, 3000);   --- New file size in MB

As you can see, you need to know not only the name of the database, but also the logical name of the database file that you want to shrink. It is not even a string, it is like a keyword in the DBCC SHRINKFILE command. Even if it does work, one would benefit from encapsulating it into a stored procedure. Here is the final code:

CREATE PROC ShrinkDatabase(@DbName NVARCHAR(100),@SizeMB INT)
AS
BEGIN


DECLARE @filename NVARCHAR(255)

DECLARE @sql NVARCHAR(Max) = 'SELECT @filename = dbf.name FROM ['+REPLACE(@DbName,'''','''''')+'].sys.database_files dbf WHERE dbf.[type]=0'
EXEC sp_executesql @sql,N'@filename NVARCHAR(255) OUTPUT',@filename OUTPUT

SET @sql='USE [master];
CHECKPOINT;
DBCC DROPCLEANBUFFERS;
DBCC FREEPROCCACHE;
DBCC FREESYSTEMCACHE (''ALL'');
DBCC FREESESSIONCACHE;'
EXEC sp_executesql @sql

SET @sql='USE ['+REPLACE(@DbName,'''','''''')+'];
DBCC SHRINKFILE ('+REPLACE(@filename,'''','''''')+', '+CONVERT(NVARCHAR(100),@SizeMb)+');'
EXEC sp_executesql @sql

END

Create it in the master database and use it like this:

EXEC master.dbo.ShrinkDatabase 'tempdb',3000

Take note that you cannot use this to "shrink up" the database. If the value you set is larger than the current size, the file will remain the same size as well as the setting for the initial size. Also take note of the fact that this stored procedure only shrinks the data file, not the log file (dbf.[type]=0).

Just a quick reminder that there are a lot of Star Trek productions out there, some really terrible, some really good. I am going to talk about three(ish) of them today.


First one is Star Trek: Federation One. It is with a lot of actors you have seen in other Hidden Frontier and Studio Areakt productions and focuses on the chief security of the presidential office of the Federation. While the plot is kind of a mess, the acting was decent and I was actually very sad to see that after the first "season" of two episodes they continued the story in an audio format. Even if the main character looked like he was on speed all the time, I had fun with it.


Another one is Star Trek: The Infinite Chain. Even if it is split into "episodes" it is actually a feature film. The acting is a bit amateurish, but decent. No obese teens in this one. It uses the now tired plot of a Federation starship thrown into unknown space by "an anomaly". The worst part of it was the "doctor" that was assigned to the mission. His character was believable: the guy in charge of the project creating the sensors used to study the anomaly, but in truth not a good manager or having any merit in the team. He just explodes randomly in childish emotional outbursts, threatening everybody with his artificial authority, and everybody hates him. I totally know people like that. However the actor was just goddamn awful and his script lines ridiculous! The makeup was ingenious, but really cheesy.


The last, but certainly not least, is something made by the Hidden Frontier team, but not set in the Star Trek universe. I am talking about Frontier Guard. The casting is made with the better actors of previous productions and, while the universe seems very similar to the Star Trek one, it is actually a different thing altogether. They use another type of space propulsion, it involves a FederationFrontier Guard academy and it concerns The Arc, a huge alien artifact that may have seeded the galaxy with intelligent life. Unfortunately, after some great episodes, they completely lost it. They began (some) episodes with "My name is..." evoking feelings of remembrance towards all the bad DC superhero series and movies, they started pointless subplots like the gay relationship thing - which always pisses me off, not because of the gayness, but because of the relationship that doesn't further the plot in any meaningful way - and threw in the towel after the 12th episode (which was an elevator show, basically).

From these three Frontier Guard was the best. My guess is that they wanted to do a Star Trek Academy series - which makes total sense and it should be the direction of the next Star Trek series - but they didn't get the approval. Since this sounds like a really good idea and one that might attract a lot of fans, I believe studios are keeping the story in reserve. However, not being Star Trek allowed them to evolve both stories and characters and I was really excited to see this done. In a world where everybody is remaking remakes, we need and deserve original content. Just look at David Feintuch's Seafort Saga series of books. That would make a fantastic TV series and has enough material to allow for several companion feature films.

For people just considering starting to watch these, do not expect Star Trek Next Generation. The actors are amateurs, I mean real amateurs not professionals who just started acting, most of them are American Trekkies and - even if I don't want to insult - some of them are just humongous pieces of fat with big glasses, which kind of explains the whole geek high school thing that is kind of difficult to understand in Europe. The stories, the lines, the makeup, the uniforms, the special effects, they are all done by enthusiasts. Sometimes they feel cheesy, sometimes they look completely and horribly fake. But once you realize that they are people like you, enjoying the wonderful universe of Star Trek, you can begin enjoying these productions.

I've watched other things as well, including Machinima style animations like Borg War - which I wouldn't recommend, but wasn't bad, and I will continue to look for these things. Just by looking at people like Hidden Frontier, who start doing something for the fun of it and end up doing original stuff with veteran amateur actors, I get filled with hope for the whole fan made universe, not just the Star Trek one. Keep up the good work guys (and if you can't do it good, keep working and it will get there eventually :) )

The best thing about it is that most of these series are online, on YouTube, free to watch. Here are some links to the shows above and others:
Star Trek Federation One - 1.01 Unity
Star Trek Federation One - 1.02 Institutions

Star Trek: The Infinite Chain

Frontier Guard

Star Trek: Digital Ghost

Special mention: Star Trip - a humorous parody of Star Trek.

I had this situation where I had to match a string by several regular expressions and choose the most specific match. Let me explain it a little bit further: you have a string and several regular expressions that match that string. The requirement is to choose which one matches better. This would be useful if, for example, you want to match mobile manufacturers and so you define one configurations for Apple and another for Android and another for the rest. The regular expressions would be "Apple", "Android" and ".*" (which matches everything). So you would want to choose only "Apple" for Apple devices and not ".*".

Now, I've studied regular expressions a bit, but I don't want to implement a regular expression parser in order to compute the complexity of the expression tree. What I want is to have various Regex objects and, somehow, compare their complexity. I am not going to pretend that I understand what goes on within the Regex object, instead I will say that I noticed the complexity of a regular expression is directly proportional to the length of the pattern and, in the case of the Regex object, with the number of items in the _codes int32 array found in the code RegexCode object found as a private field of the Regex object.

So, the simplest code for that is this:

private double regexComplexity(string pattern)
{
    var reg = new Regex(pattern, RegexOptions.Singleline | RegexOptions.IgnoreCase);
    var codeField = reg.GetType().GetField("code", BindingFlags.Instance | BindingFlags.NonPublic);
    var code = codeField.GetValue(reg);
    var _codesField = code.GetType().GetField("_codes", BindingFlags.Instance | BindingFlags.NonPublic);
    var codes = (int[])_codesField.GetValue(code);
    return codes.Length;
}

However we must take into consideration several cases:

• The regular expression pattern might be wrong
• Reflection is slow

So, let's first catch all exceptions and use the length of the pattern as the complexity and then find a way to cache the complexity of a string, in case we use it more than once. Also, cache the FieldInfo objects for use at every call.

Here is the final code:

public static class RegularExpressionExtensions
{
    private static ConcurrentDictionary<string, int> _dict;
    private static FieldInfo _codeField;
    private static FieldInfo _codesField;

    static RegularExpressionExtensions()
    {
        _dict = new ConcurrentDictionary<string, int>();
        _codeField = typeof(Regex).GetField("code", BindingFlags.Instance | BindingFlags.NonPublic);
        _codesField = _codeField.FieldType.GetField("_codes", BindingFlags.Instance | BindingFlags.NonPublic);
    }

    public static int Complexity(this Regex reg)
    {
        if (reg == null) return 0;
        var pattern = reg.ToString();
        var complexity = _dict.GetOrAdd(pattern, p => getComplexity(reg));
        return complexity;
    }

    private static int getComplexity(Regex reg)
    {
        var code = _codeField.GetValue(reg);
        var codes = (int[])_codesField.GetValue(code);
        return codes.Length;
    }
}

Basically we encapsulate the Regex complexity in an static extension class that we can use on any Regex instance as reg.Complexity().

Just wanted to post these really funny Star Trek fan made animations.

South Trek - a South Park animated Star Trek:


Star Trekkin' song:


Stone Park - a Flinstones animated Star Trek:



Some other funny stuff:

Today I published a raw version of a program that solves Pixelo puzzles, a Flash version of the game generally known as Nonogram, Picross, Hanjie, etc.. I was absolutely fascinated by the game, not only the concept, but also the attention to details that Tamaii lent to the game. The program is called Pixelo Solver and you can find it at Github, complete with source code.


I liked working on this because it covers several concepts that I found interesting:

• Responding to global key combinations
• Getting a snapshot of the screen and finding an object in it
• Parsing a visual image for digits in a certain format
• The algorithm for solving the puzzle in a reasonable amount of time and memory (it was not trivial)

Quick how-to: get all the files from here and copy them in a folder, then run PixeloTools.GameSolver.exe and press Ctrl-Shift-P on your puzzle in the browser.

Usage:

1. Start PixeloTools.GameSolver.exe
2. Open the browser to the Pixelo game
3. Start a puzzle
4. Press Ctrl-Shift-P

However, if the list of numbers per row or column is too long, the automated parsing will not work. In that case, you may use it offline, with a parameter that defines a file in the format:

5 1 10, 3 3 7, 2 5 4, 2 5 2, 2 1 1 1, 1 1 1 1 1, 1 1 3 1 1, 1 1 1 1 1, 1 1 1, 1 1, 1 1, 2 2 2 4, 2 1 1 1 2 4, 2 1 2 1, 7 4, 2 2 1 2 2 2, 2 1 1 1 1 1 1, 2 1 1 4 2, 1 3 4 2 1, 1 1
8 1 1, 5 2 2 1, 2 1 3, 1 1, 1 7 4 1, 3 5 1, 3 1 1 3, 4 3 4, 3 1 1 3, 3 5 1, 1 7 4 1, 1 2, 1 1 1, 2 2 2, 2 1 1 2, 2 1 2 2, 3 2 1, 3 1 1, 4 1 2 2, 9 2 3

where the first line is the horizontal lines and the second line the vertical lines. When the parsing fails, you still get something like this in the output of the program. You can just copy paste the two lines in a file, edit it so that it matches the puzzle, then run:

start "" PixeloTools.GameSolver.exe myFile.txt

The file can also be in XML format, same as in the file the game loads. That's for Tamaii's benefit, mostly.

Let me know what you think in the comments below.

As some of you know, I am watching a lot of TV series, including cop and lawyer stuff, childish fantasy and sci-fi series, pirate and musketeer crap, etc. So when I say that I couldn't watch something, you should heed me. There are also some shows that I actually can stand watching, but barely, usually using the fast forward to watch an episode in ten minutes or less.
Here is a list of just such shows:

• Lost - I think it all started with Lost. After a brilliant pilot episode, it got into 1001 Arabian Nights territory and became ridiculously unwatchable. While it wasn't a truly bad show (some people actually enjoyed all 6 seasons) I couldn't watch it more than 2 seasons. Calm down, the rest of the shows will be more recent.
• The Messengers - every once in a while the American Bible thumpers convince producers to create religion based TV series or movies. While every possible superstition has and will be used in films, the Christian ones are standalone - in their own cinematic universe, if you will - and take themselves seriously every time. Messengers is one of these, and I couldn't watch more than 2 episodes before I gave up. It's (again) about the upcoming apocalypse and end of days and these 8 idiots are given superpowers to convince God that we are still worth it, while Satan goes around doing silly things. It was not only about the most tabloid part of (a part of) the bible, but it was acted and scripted badly. Ugh!
• Dig - This relates more to Hebrew religion and feels like a TV series take on the Da Vinci Code, with clones of Jesus (I think), FBI policemen running around in Israel doing whatever they feel like, shady cabals and shady opposing cabals. It could have been interesting and I admit that I was drawn into it by the acting and production values. Unfortunately the script was a convoluted mess that had no way of improving. They never learn to ease into it, present a little bit of the story, add another element later on, make people feel rewarded for continuing to watch the series. Anyway, four or five episodes in, I gave up.
• iZombie - Oh, you have to love this: young blonde accidentally gets turned into a zombie and gets a job at the morgue in order to have access to a fresh supply of brains. While consuming said brains, she gets visions of what the brain's owner had lived and thus helps a police detective to find out who murdered them. It combines police procedurals with zombies, comedy, romance and drama. Talking of messes, this is as messy as they come. I watched a few episodes before I slapped myself "what the hell are you doing watching this swill?"
• Eye Candy - A darker show about a girl using her hacker tech skills to try and catch the serial abductor and killer of young women, including her sister. Yet the killer is just as able and a game of cat and mouse ensues. It started well, then went into pop culture territory. Besides, she continues on only because the killer refuses to murder her and get on with his job. Something about the way the film was made or the main actress or the ridiculous plot made it impossible for me to get past the second episode.
• Allegiance - It is strange to put this series here, as I watched all the episodes so far. However, it was hope that kept me going, not the quality of the show. While it is about Russian spies in the US, living as a family, while their unknowing son is recruited into the FBI, while the antagonists are interesting, the story goes so much over the top that it becomes overcheesy. That means it starts as cheesy, then it starts to smell. Just do yourself a favor and do not watch it.
• Suits - Yet another flashy show that I kept watching for no good reason. It is about lawyers. Young, cocky, caught in personal intrigues and competing with one archnemesis after another, while occasionally engaging in some silly romance. It makes no sense, although I have to admit that I liked the feel of the show. It just goes nowhere. They get the mood right, the actors, the characters, then they make a joke of the storyline and script.
• Helix - Helix is the new Lost! It starts with a subject close to my heart: killer viruses, and then it crashes into la-la-land. Viruses that make people immortal and strong so they form a world controlling organization, CDC teams that act like they are terrorist splinter cells with no support, no rules, no communication, strains of the virus that turns people into The Strain like vampires, religious cults that are also genetic scientists living on their own island, etc. It starts as something, goes into something else, switches back. It's like someone was undecided what to write about and they wrote about everything then they just drew lines between the unrelated subjects and wrote a script. It is a complete and utter waster of time.
• Elementary - I watched two seasons of this until I admitted to myself that there is no point in continuing. I like both Johnny Lee Miller and Lucy Liu, I love Arthur Conan Doyle's Sherlock Holmes character and so a modern Sherlock should have been something I would at least enjoy. Unfortunately, it shortly turns into another "gifted person helps the New York police" show and one can feel when the writers just completely stopped giving a damn, if they ever did before beyond pushing the pilot.
• Scorpion - Talking about gifted persons helping the police, this had a promising concept: a genius young man creates a sort of a thinktank of gifted people then starts helping his FBI parent surrogate. While the characters were brilliant, the people writing their script obviously were not. It took a few episodes to completely fail and another ten for me to just give up hope.
• The Librarians - God, what a complete and utter mess. A TV show spinoff of the ridiculous Librarian movies, it was completely stupid. I mean, the movies were masterpieces compared to the writing for this show. Stay away!
• The Newsroom - This is a show that has ended and I watched all of its episodes. However, it was under duress. My wife liked it, but then I could see that she couldn't watch it anymore. I watched the last mini-season just in order to tell her it was not worth watching. It's about a news organisation that fights for its moral integrity. It would have been fun if not for the hyperactive writers that thought people could normally carry conversations like machineguns shoot bullets and the ridiculous romantic interludes that made no sense for anyone but a New Yorker raised on Woody Allen movies.
• State of Affairs - I already ranted enough about this piece of propagandistic crap. The plot was stupid, the script was stupid, the actors couldn't give a damn. It's just bad!
• The 100 - In theory I am still watching this young adult sci-fi TV show. In reality, I can't make myself start watching the second season. Why? Because the sci, as usual, gets completely negated by the fi. Add to this hormone induced behaviour of teenage kids and savage barbarians and you get a load of messy crap. It started really well, though. Too bad.
• Extant - Another sci-fi show in which they dumped randomly whatever was hot at the moment: aliens, artificial intelligence, the Japanese guy from Helix, etc. Add to this the excruciating acting skills of Halle Berry and you get something unwatchable.
• Stalker - This is one of the more interesting police procedurals. It involves a team that is tasked to handle stalkers. The new hire of the agency may be a stalker himself. The lead actors are people that I respect. However the story was so banal and lacking anything that showed any concern for the lives of the characters or victims, it was so formulaic that it ended up being something I regret having watched.
• Arrow - OK, I am still watching The Arrow, but I do it on fast forward. It started as a fun comic book TV show with a vigilante dressed in green killing assholes with the use of arrows. It then started growing and expanding until it reached a full team of heroes, with impossible situations, history, skill learning curves, ridiculous villains. It culminated with John Barrowman having a line in the show in which he tells three other characters he is the better man for a job because he is the better at acting. Which is true, but it just shows that even the scriptwriters are tired of the mediocre talents in this mediocre show. I keep watching it because it is set in the same universe as The Flash, which is the better TV show, even if not by much.
• Gracepoint - When Americans are redoing a show with the same script as the original show from a different country I snicker. Stoopeed Americaens, they can't read subtitles. But Gracepoint upped the ante: they redid Broadchurch, a British TV show. Not only that, they cast the same actor as the main character. I love David Tennant, but I can't possibly condone watching this. Anyway, it was cancelled after a season, and then Broadchurh continued for a second season, also starring Tennant.
• Sleepy Hollow - An entire TV series based on the Legend of the Sleepy Hollow. Starring a colonial Mason that fought for and knew the forefathers of the United States, resurrected in our times and fighting along a black female police officer to stop the End of Times. What can possibly go wrong here? Yet it wasn't that clear cut. The comedy was fun, the original drama had impact, the romance was believable and John Noble, like in Fringe, was awesome. However the storyline, like in Fringe, descended into a chaotic mess that makes no sense and no one could ever care for. Too bad, since it wasn't a complete waste of time, especially at the beginning.
• Madam Secretary - Just as with State of Affairs, I've ranted enough about it. It's a pure propaganda piece, with no artistic value whatsoever.
• Haven - It started as a nice supernatural police procedural with a sexy FBI agent coming in a town where people get dangerous "troubles" which act like superpowers out of control. I kept waiting for it to get anywhere, instead it just went more and more overdramatic and adding storylines that made no sense and in the end turned into a supernatural 90120. Too bad, since it started well and has some actors that I really like.
• Forever - Forever boring, this show is about a British doctor who never ages and, whenever gets killed, he finds himself alive and well in a nearby body of water, completely naked. And what does he do with that? He helps the police, as a forensic medic. Just as a general rule: avoid the gifted people helping the police catch bad guys shows. They never end well and they are as well propaganda pieces.
• The Lottery - Badly made show. Bad casting, bad acting, bad scripting. The plot: people stop having children, embryos stop getting fertilized and a woman doctor finds a way. Obviously, the powers that be want to control this in a market of dwindling supplies. Could have been nice, probably, although shows about children and parently feelings never do well in the long run, but it was pure shit from the beginning.
• Taxy Brooklyn - Someone tried to combine the French Taxi storyline with a police procedural. Sexy ass police detective, a Black French taxi driver, some intrigue and of course the obligatory connection with an old case involving a relative. But it was bad. The actors did what they could with lazy writing, but they couldn't save it.
• The Witches of East End - Some of the most beautiful women in acting get together with ridiculously good looking men in order to act a really stupid Dynasty-like plot. Just like similarly afflicted The Originals, it is a complete waste of time, unless you are a hormonal teenager and you need something to jack off to.
• The Leftovers - Trying to express the horrible anguish of being among the ones left on Earth after 2% of the population randomly disappears. Of course, it hints of another Bible concept: the Rapture. It felt to me overly dramatic, with people just doing stuff for no good reason and overly depressing as well for equally no good reason. It seemed to be well acted, but I just couldn't watch it.
• Resurrection - an American remake after Les Revenants, it missed the point completely. Ended up being this drawn out mystery that led nowhere, with really unsympathetic characters and without any of the real tension from the French series.
• Da Vinci's Demons - I know, I am still watching it, but not because it is worth watching. It is about a young Da Vinci who does everything from saving Florence to saving Italy and then going to the Americas (way before it was discovered) to look for clues about his mother, who was there previously. It is a fun to watch show, but it really goes nowhere. I liked the acting and the production values, though.
• Crisis - An intriguing concept: some very resourceful people kidnap a lot of high profile adolescent kids, including the son of the president of the United States, then they use the parents to achieve their goals. While the show was well done, the story was just a lazy fantasy trying to surf the edge of suspension of disbelief. Again, it goes nowhere.
• The Tomorrow People - An American take of an old British children's TV show that wasn't good to begin with, it only served to promote certain actors to better TV shows. It used the common tropes: child with mysterious father who dies doing something awesome, mother unknowing, uncle being involved somehow, evil organizations, rebellious teenagers fighting them, etc. It just collapses under the ridiculous ideas that they introduce to appease a boring viewership.
• The Originals - Some spin-off from a vampire show that I never watched, it is about pompous asses acting even more pompous because they are the original vampires. Werewolves and witches are thrown in the mix, but the only worth mentioning attribute of the show is the pompousness. Which is annoying. So don't watch it.
• Killer Women - Trying to get on the diversity bandwagon and get some easy money, it was supposed to be a police procedural with tough women. Didn't pan out. Was cancelled.
• Bitten - One of the most beautiful actresses around is a werewolf. Instead of being about her biting more than she can chew (heh heh), it is about another clan/royalty obnoxiousness, while the world is blissfully unaware and rivals are trying to take power or kill them or whatever. Do you know why people don't officially acknowledge vampires and werewolves? Because they act all annoying and overbearing while being completely idiotic. Nobody wants to know these people!

OK, as an interesting experiment on my preferences vs the public preferences, I will tell you how many of the shows above were cancelled and how many were not. Ready?

Completed (meaning not cancelled, but finishing their story): 2
Cancelled: 16
Still Running or undecided: 15

So there you go, studios think 50% of the shows that I can't stand should be running and they are basing their decisions on public choice. What do you guys think?

Relational databases have the great advantage of being able to use indexes. These are constructs that trade space and the speed of inserts and updates for query speed. This introduces a problem: what do you do when you have a high input and high output application? How can you make your database fast for both changes and queries?

The traditional solution is a write-database and a read-database. There are background services that ensure that the read-database is updated as fast as necessary with the new data, most of the time also doing extra computation and caching for the information one needs extracted. OLAP systems are a common solution, where the aggregated data required by various reports is computed and stored, ready to be read by your applications. I am not saying this is a bad idea, in fact I am saying this is the best idea for this scenario. The problem that I have with it is that you can hardly automate the process. You need to know what you want to read, you need to write the software to create the data and aggregate it into what you need.

So I decided to try to build a system that obeys the following rules:

1. The speed of insert and update operations needs to be unhindered by indexes. Indeed, changes to the original data should be avoided.
2. The select operation need to benefit from the indexing system.
3. The system must be able to determine by itself the data structures needed to cover the first two rules.
4. Optionally, there should be a way to encapsulate this new behaviour into a separate data structure from the original database.

In order to insert and update as fast as possible, I needed tables that only had primary keys, identity integers rather than uniqueidentifiers, as they lead to fragmentation of clustered indexes. In order to not only index the columns that are used in where and join conditions, but also encapsulate the behaviour in some other data structure, I decided to create shadow tables with the columns that I needed for querying. These tables I would then index in order to improve selects. The connection between the original insert table and the select table would be made via primary keys and the synchronization between the two types of tables would be done in the background, whenever needed. Best of all, analysis on execution plans could automatically determine the columns needed for this system and create the tables and indexes required, then suggest improvements on the observed queries.

In order to test my theory I created the following tables:

• OriginalTable - with a primary key identity ingeniously called Id and two other columns called Latitude and Longitude, containing random decimal(18,6) values
• LocationIndexTable - with a refId integer primary key column that links to the OriginalTable Id - without being a foreign key - and two Latitude and Longitude columns that are indexed by the same non clustered index
• LocationIndexTable2 - with a refId integer primary key column that links to the OriginalTable Id - without being a foreign key - and a locId integer column that links to another table called LocationTable and has its own non clustered index
• LocationTable - with a primary key identity column and Latitude and Longitude columns. I know that this looks identical to LocationIndexTable, but this is the more complex case where there are multiple records with the same location. LocationTable holds the distinct Location and Latitude values
• LocationIndexTable3 - with a refId integer column that links to the OriginalTable Id and is indexed by its own nonclustered index - without being a foreign key - and two Latitude and Longitude columns that are indexed by a clustered index

With a number of 77179072 original table rows, I attempted the queries for each case, careful to clean the cache and memory buffers before that. Here are the results:

• SELECT count(1) FROM OriginalTable WHERE Latitude BETWEEN 45.5 AND 46 AND Longitude BETWEEN 8.5 AND 9 - no indexes whatsoever. Result: 30 seconds
• SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable lit ON lit.RefId=ot.Id WHERE lit.Latitude BETWEEN 45.5 AND 46 AND lit.Longitude BETWEEN 8.5 AND 9. Result: 17 seconds
• SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable2 lit2 ON lit2.RefId=ot.Id INNER JOIN LocationTable lt ON lit2.LocId=lt.Id WHERE lt.Latitude BETWEEN 45.5 AND 46 AND lt.Longitude BETWEEN 8.5 AND 9. Result: 41 seconds
• SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable3 lit ON lit.RefId=ot.Id WHERE lit.Latitude BETWEEN 45.5 AND 46 AND lit.Longitude BETWEEN 8.5 AND 9. Result: 22 seconds

Unexpectedly for me, the most comfortable solution also seems the faster. Indeed, one issue is that I didn't have duplicate location data in the database, so there was no advantage in adding a new table to link locations to the original table. That means that in cases where the indexed data has many duplicates, it might be advantageous to experiment with a "distinct" table, although indexing should take this case into account, as well. The clustered index is slower than the unclustered one, that was a surprise. Also, the indexing just made the query twice as fast - I had expected more. Of course, all this benchmarking was done after deleting the cache and buffers with the commands DBCC FREEPROCCACHE; DBCC DROPCLEANBUFFERS. It is interesting to see how fast they queries go without this clearing. The first unindexed query takes 3 or 4 seconds, while all the others take less than 2.

There is one more thing that needs to be addressed: moving these tables into another database. Are the indexes just as fast? They should be, but we must test interdatabase communication. This would allow to move the entire system outside a target database, truly encapsulated, really adding value to a database that remains unaffected. My tests show the following results: 28, 18, 29 and 18 seconds, respectively. Yes, you saw that right, the speed of the joins when the indexed databases are in another database on the same server seems faster! Just to make sure I reran the original tests on the same database and I got approximately the same results: 29,19,43 and 24, respectively. The only thing I didn't try (and I don't intend to) is to create primary-foreign key associations, as this means modifying the original tables, something I wish to avoid.

So, after all this I believe my idea has been validated. A lot more work has to be done in order to automate this system, but at least a no-effort manual solution is possible. Let's recap:

1. The speed of row inserts and updates remains unchanged: the only index is the original primary key identity integer key that should always exist in a table anyway.
2. The speed of selects is improved by creating tables that have an integer primary key that links to the original table, and only the columns used in queries, over which indexes are created.
3. Queries can be used to determine the columns needed to index. Even better, computed columns can be used instead of the original columns, which adds a little more performance.
4. Encapsulation is achieved by not only creating other tables for reading, but also moving these tables into another database, after which, unexpectedly, the performance is even better.

The end result is a system similar to indexing, but which takes space in another database and which is updated on demand, not when the system deems it necessary. Any SQL statements that worked before will continue to work unaffected, but faster alternatives for selects will become available. Overall, I am very satisfied, although I had expected better performance improvements than observed.

There are these time periods in which movies and TV shows start to follow a common theme. I don't know exactly why that happens, but I suspect it's about Hollywood corporate managers stealing everything that seems to work and getting to copy each other. We seem to be living under the sign of the Spy right about now and in this blog post I will be discussing the following TV series (listed in alphabetical order):

• Allegiance - a show about a CIA analyst who is the unknowing son of a pair of Russian KGB spies. He is also a kind of idiot savant which can remember a lot of things and make connections that no one can. It gets really hard to believe soon as it mashes up family drama with evil SVR (former KGB) machinations, CIA and FBI operations, traitors and double agents, hired assassins, plots to bring down the United States, and so on. It's one of the lesser spy shows and I don't recommend it.
• The Americans - I believe this is the gem of the lot. The protagonists are two fake Americans, in reality KGB agents. They live like an American couple, with the mandatory two children and their neighbor is an FBI agent in the very taskforce that is supposed to catch people like them. The acting is very good and the show is a serious one, going only occasionally over the top. Also the lovely Keri Russell makes it very appealing as well :)
• The Assets - The Assets has ended after a single season, with the successful completion of the main plot. It was a strange mix of really good acting and rather weak characterisation. It followed the story of Aldrich Ames, a famous disgruntled CIA operative who sold American secrets to the Russian KGB. As it is a true story, the entire plot was not terribly captivating, but the problem for me was with the characters.
• Turn - About the spy ring of the American rebels against the British during the Washington era, it is an intricate and interesting story. I like most of the characters, even the villains, which is rare in films these days. I particularly like Burn Gorman in this, even if he plays this British buffoon that understands nothing of what is going on around him.
• X Company - Set during World Word II, it is about a company of spies working in the German occupied territories. Their newest recruit is a guy who can remember everything he sees, but is a little weird. Sound familiar? Heh

From the list above, The Americans shines bright. Good acting, good directing, but also a dedication to show the political and social issues of the time. From the ones here, it is the only one who I feel can teach one something about history and about human nature in general. Turn is also rather good, but it is too reliant on a number of characters that keep ending up crossing paths in implausible situations, and many times goes a little over the top. The actors are really good in this one, though, and it is worth watching for sure. X Company is more like the typical WWII movie, with the evil Germans and the brave resistance fighters. They do attempt occasionally to enter the grey areas, like good German officers, conflicted collaborators or remorseful secret agents who have to compromise between human values and the mission, but it is not enough. When the beautiful team leader played by Evelyne Brochu reveals that she is half Jewish and the whole story goes into Holocaust territory you can feel the show going south. The Assets was brave in not employing only beautiful actors. The main character was ugly and weak and constantly fearing discovery, hard to empathize with. But so were the people who were trying to catch him, led by an obsessive woman that acted like a blood hound the entire time. Not being able to make an emotional connection to any of the characters hurt the success of this otherwise interesting story. This leaves Allegiance, which feels a lot like a fairy tale combined with a police "special" procedural (you know the type: special person helps police solve crimes) which incidentally features spies. While I cannot really recommend X Company, Allegiance is the only one that I would recommend against.

Worth mentioning is that three of these shows are based on real events. The Assets follows the exploits of Aldrich Ames, X Company is loosely based on Camp X and Turn links to the story of the Culper Ring.

That's it for pure spy shows. I am not putting in this list shows like Homeland or Legends, which also show undercover operatives, but are not specifically aimed at spying as the main subject of the plot. Tell me what you think.

Another fantasy audio book, The Black Company is the first of ten of the ongoing series with the same name. Glen Cook probably thought about it for a while before writing it, because you can see that the first three books in the saga were published mere months apart (*cough*R.R.Martin*cough*). I was attracted by the story, because it seemed similar to my favorite Malazan Book of the Fallen series: a mercenary company fighting under the flag of powerful god-like magical beings that want to conquer the world. And in part the plot and characters were satisfactory. Alas, the many details and focus on small things like what games they played to pass time or what each character was like or maybe the narration style... I don't know; it all made it hard to focus on the story. I felt like the book could (and should) have been half in size, and better for it.

The main character is the doctor and analyst of the company, Croaker. Before your filthy mind thinks of other things, analyst in this context means he is a keeper of records, a chronicler of the company's history and, occasionally, the person who draws parallels between past experiences and current events. Croaker is an inquisitive person, often risking his personal safety to unlock a riddle or reveal a secret. Unlike Malazan, he is the lead character, through and through, and the company itself with its many soldiers and camaraderie just the backdrop.

Plot follows the Black Company in employment of The Lady, a powerful magician who wants to conquer the world, fighting the Rebel, a group of lesser magicians who have gathered the people of the land in response. Croaker has romantic fantasies about her, but throughout the book he discovers that his affections are misdirected, even if a strange relationship develops between them. Most of the story is about various battles, painstakingly (and painfully) described, only to be followed by the occasional interesting bit of character and plot development. In fact, I was kind of annoyed when I read the book summary on Wikipedia and I couldn't think of much that was left out. I mean, it's a big book.

The bottom line is that I am unsure if I want to listen to the rest of the series. While I can't say it was a bad book, or that is had weak plot or characters, I am reluctant to go through all that again for nine times. Basically I feel that the characters were never made empathetic enough, at least for me, and in the end all I cared about was what was going to happen next and how it would all end. In that case I would be better off reading synopses rather than listening to the entire thing. The overall structure of the story, though, has a lot of potential and I don't know if the series would not become really cool afterwards. To make the final parallel with Malazan, the first book in that series was not really making people eager to read the next, but it turned out to be great overall. I guess time will tell.

... and I don't mean something like injecting ads; I mean they modify the images you download and the pages that you read. They do that without telling you, under the umbrella of "improving your browsing experience". Let me give you some examples.

Today I copied two image files on a server: a JPEG and a PNG file. When downloaded via a normal network connection, I was getting the original file, about 50KB in size. When downloaded via 3G the image was different! In the jpeg case the file was smaller by a few hundred bytes and in the case of the PNG the file was actually bigger than the original. What was worse, the metadata information in those photos, like the software used to compress it, for example, was completely lost.

I couldn't believe my eyes. I strongly believe that what you ask for from the Internet you should get. This may not have been obvious for someone downloading the images in order to see them, but I was actually conducting a test that depended on the exact size of the file I was downloading. This practice seems to be widespread, but when Googling for it very few links pop up, showing that people mostly have no idea that it happens. In this operator's case, they seem to only change images, but people on the Internet tell stories of bundling CSS and Javascript files inside HTML files, or removing comments from either of them.

The morality of this is dubious at best, in my view it should be illegal, however things are worse than that: this behavior breaks functionality in existing sites. How can you possibly guarantee that your application works as expected when mobile operators (and I guess any ISP) can change your content arbitrarily and without the possibility to opt out? It's like that joke with the boyscout explaining at camp that little old ladies are hardier than one might think, as they squirm and shout and hit you when you try to cross them the street and the instructor soon finds out that it never occurred to the boy that he should first make sure they want to get to the other side.

Here are some links regarding this, just to make sure people can find them more easily:

Mobile operators altering (and breaking) web content
Should mobile operators be free to modify content they deliver?
Mobile Proxy Cache content modification by O2
O2 UK mobile users - your operator is breaking this site for you.
Prevent mobile website image compression over 3G
Get rid of image compression on O2′s network
ByteMobile Adaptive Traffic Management

From the last link you can see that they are caching and modifying even movies, through practices like giving you a lower rate movie or caching a version with a lower resolution. They do this in the name of delivering you video content compressed with a format and codec that your device can safely open.

A solution for this? Encryption. Using HTTPS prevents access to the content from a third party. HTTPS is becoming more and more used, as the hardware requirements for its implementation become less restrictive and with the many revelations about government scrutiny of Internet communications. However, you might be interested to know that mobile operators are feeling threatened by it. Articles from their point of view decry the "threat" of encryption and the solutions against it! The terrible impact of encryption is seen as an impediment in their rightful "content and delivery optimization techiques".

Things are getting even worse. Remember the concept of network neutrality? It is a very hot topic today and a very important political and economic fight is being fought right now to protect the transparency of the Internet. However, if you look further, you see that nobody considers the practice of "optimization" as something net neutrality should protect. In December 2010, the US Federal Communications Commission set three basic rules for net neutrality:

• Transparency. Fixed and mobile broadband providers must disclose the network management practices, performance characteristics, and terms and conditions of their broadband services;
• No blocking. Fixed broadband providers may not block lawful content, applications, services, or non-harmful devices; mobile broadband providers may not block lawful websites, or block applications that compete with their voice or video telephony services; and
• No unreasonable discrimination. Fixed broadband providers may not unreasonably discriminate in transmitting lawful network traffic.

So they are primarily focusing on blocking and throttling, but not on preserving the integrity of the transmitted data!

EZTV.it was a site that listed in a very simple interface the latest TV shows in the English language, together with links to their torrent or magnet links. It was very easy to use and, as opposed to other similar sites, was selective in bringing you the best link for a TV show. You didn't get a billion different torrents for an episode of a TV show, you just got the low resolution and high resolution versions and sometimes the repacked or fixed versions when the first upload was defective. It was almost ad free, as well.

Well, lately some strange things have happened and it seems to me that there is more than meets the eye in the entire story. It started with the eztv.it domain being abandoned after the intervention of the Italian authorities. EZTV were planning to change the domain to a more amenable country, like Switzerland. That happened in January.

It didn't take long until someone bid for the domain name in auction and in March the eztv.it site was back, but under a different management. For a while they were content to proxy the new EZTV site (eztv.ch), in a site that looked exactly as the old one. As you can see from the link above, the new guys actually wanted to "manage" EZTV, getting money from the torrent link business, while doing nothing than own the domain. Novaking, the guy who fronts the EZTV group, refused any deal and so, after a while, torrent links started redirecting to ad sites.

It gets even more interesting, as somehow - and I admit domain name auctioning is not something I am an expert in - the eztv.ch domain was also captured by the same people who got the .it domain. As of now, there are no official EZTV sites anymore, both eztv.it and eztv.ch are impostor sites and with a horrible quality of service.

We have to consider several factors here. First of all, Novaking is the nickname of a person that, as far as I know, desired to remain anonymous. Probably that is why it is easier for someone to seize domains like that if they are willing to pay money and even name themselves. Then there is the actual intervention of the Italian authorities. They seemed to be angry about a site that provided torrent links, but don't seem to have anything against an exact copy of that site bought money down and that intents to make a lot of money out of ads. What happened in Switzerland is less clear. Did Novaking just abandon the fight and ditched that domain as well because he thought the entire thing was too big a hassle or was there some kind of hostile takeover, just like with the Italian domain? And finally, you have the new registrar of the two domains listing a Dominguez Emmanuel Hernandez, with an address in the United Kingdom. As several cases in the past showed, UK is really against file sharing sites, even if they are just linking to illegal content.

As far as I can gather, it should have been more difficult, not easier, for an entity as Hernandez to do the kind of work that Novaking did. It could all be foul play from the authorities and/or the legal and commercial entities dedicated to fight piracy. Certainly they would not be shy to use the vague laws that govern the Internet to their advantage to hurt their adversaries who do the same thing. However, if that would be true, they would lose the moral ground for their supposed reason to fight against file sharing.

Of course, it may also be some sort of internal split and subsequent power struggle inside the EZTV distribution group. Maybe this Hernandez guy was an inside man all along and he decided he wanted more money from his work. However that doesn't sound all true either, as the quality of the work on the fake eztv sites is terrible compared to what it was before and the functionality of the site is not all there, like someone has copied the old EZTV site from the web and not having access to the source code. There are some tweets that mention the site having reverted to "ancient passwords", though, so maybe there was access to the database of the site, somewhere in the past.

The last speculation I have is that after losing the .it domain EZTV voluntarily sold the .ch domain. An IRC message on the EfNet #eztv channel announced the end of the Novaking EZTV era:

[2015.04.25 20:24 PDT] <@NovaKing_> it was fun running eztv. Hope you guys had enjoyed it. site has new owners now. farewell.

As you can see he uses the term "owners", which might suggest a sell.

Well, I am curious of what actually happened in this very interesting story, but as things are right now, I am not sure I will ever find out. TorrentFreak has posted a more detailed explanation of what happened: it was a dubious hack, involving snatching one's domain from under them, stealing their email and impersonating them with inpunity. One thing is certain: you should not use the eztv.it and eztv.ch sites anymore, as well as search the net using the eztv brand name. In the best case scenario, you would be supporting an immoral act of web site theft and impersonation, and in the worst you would be using a web site handled by antipiracy organizations.

The Gift of Fear has popped up repeatedly in my field of view, recommended by multiple sources. I started reading it and at the beginning I thought it had a nice concept: the systematic study of violence perpetrated by people, written for reasons of protecting ourselves. However, Gavin de Becker has a writing style that got to me really fast. He sounds like he is the lecturer in a police conference, and half of everything he says is just marketing for the bits that are going to come next. You know the type: "I will reveal to you the secrets of the universe, but before I do that in one of the next chapters, let me tell you a little story". I mean, what he is saying makes sense, but he oversells it so brutally that I could not continue reading past the half of the book. You know, he sounds a lot like Walter O'Brien, the guy who's life is supposedly the basis of the TV show Scorpion. He doesn't sound like the O'Brien on TV, but like the actual guy, always overselling and overstating everything he allegedly did. Also the little anecdotes are useful in the book, but his explanations are so over the top. Man!

Anyway, the things that I chose to take from the book is the JACA system for assessing threats and the fact that when your intuition is telling you something, it either means it has access to some data that you are not conscious of or that it malfunctions and in either case you need to pay attention. The JACA system is about someone being more of a credible threat if they pass four tests. J: they feel Justified to harm you. A: they feel that they have no Alternative to violence. C: they believe the Consequences of violent action will be manageable. A: they believe they are Able to do you harm. Of course, that immediately makes someone believe that the first step of counteracting such a person is to convince them they are not justified, which fails on so many levels, especially with an antagonist.

The book covers all kind of violence: rape, murder, stalking, assassination, road rage, office vengeance, domestic violence, even violent children (I haven't gotten to that part). I can imagine how this book would be very useful to young people, scared women, maybe even children, but with the language being so pretentious and the guy making it all sound like a marketing pitch, I doubt it would be accessible to any of them. Let me reiterate: I believe the subject of the book is a good one and it should be addressed. I also don't criticize the conclusions that Becker reaches or doubt his professional experience. What I am saying is that the way the book is written stylistically made it unreadable for me. So instead of reading a few pages every week, I've decided to stop reading it. Sorry, Gavin! I only wish someone would make a short summary of it, since a lot of the stuff there is at least interesting, if not downright useful.

The same people that did Star Trek: Hidden Frontier created the three season Star Trek: Odyssey. The first season was pretty good, beginning a story that was a combination of Voyager, Deep Space Nine and even Enterprise, the second season had fewer resources and the third season was decent, but the story was a complete mess that was very hard to grasp. Luckily enough, the superfluous homosexual stories in Hidden Frontier were almost not existent in Odyssey. Instead we were treated to intergalactic invasions, religious empires, slipstream drives, artificial wormholes, omega particles, anti-omega particles, Romulan first officer played by lovely Michelle Laurent and some decent (not good) screenplay. The technical effects, acting and directing varied from OK to really bad. They clearly learned a lot from Hidden Frontier, but the green screen continues to be their greatest enemy.

Even better, it is a crowdfunded show and you can watch it all online, for free, on YouTube. Here are the links for all three seasons:

Season 1
Season 2
Season 3

Enjoy!

I was trying to do a simple thing: configure a daily rolling log for log4net, meaning that I wanted that log files would be created daily and the name of the files would contain the date. The log was already configured and working with a normal FileAppender, so all I had to do was find the correct configuration. There are several answers on the Internet regarding this. I immediately went to the trusty StackOverflow and read the first answers, copy pasted lazily, and it seemed to work. But it did not. So warning, the first answer on StackOverflow about this is wrong.

So, my logs had to be named something like Application_20150420.log. That means several things:

• The name of the appender class has to be set to log4net.Appender.RollingFileAppender
• The name of the log files need to start with Application_ and end in .log
• The name of the log files need to contain the date in the correct format
• The logger needs to know that the files need to be created daily

This is the working configuration:

<appender name="FileAppender" type="log4net.Appender.RollingFileAppender">
     <filter type="log4net.Filter.LevelRangeFilter">
         <acceptOnMatch value="true" />
         <levelMin value="DEBUG" />
     </filter>

  <file type="log4net.Util.PatternString" value="c:\logfiles\Application_.log" />
  <appendToFile value="true" />
  <rollingStyle value="Date" />
  <datePattern value="yyyyMMdd" />
  <preserveLogFileNameExtension value="true"/>
  <staticLogFileName value="false" />

  <lockingModel type="log4net.Appender.FileAppender+MinimalLock"/>
  <layout type="log4net.Layout.PatternLayout">
    <conversionPattern value="%date %-5level %logger - %message%newline"/>
  </layout>
</appender>

As you can see, it is a little counter-intuitive. You do not need to specify the date in the file name, it will be added automatically, and you absolutely need to use preserveLogFileNameExtension, otherwise your files would look like Application_.log20140420