and has 0 comments

Intro


I was thinking about what to discuss about computers next and I realized that I hardly ever read anything about the flow of an application. I mean, sure, when you learn to code the first thing you hear is that a program is like a set of instructions, not unlike a cooking recipe or instructions from a wife to her hapless husband when she's sending him to the market. You know, stuff like "go to the market and buy 10 eggs. No, make it 20. If they don't have eggs get salami." Of course, her software developer husband returns with 20 salamis "They didn't have eggs", he reasons. Yet a program is increasingly not a simple set of instructions neatly following each other.

So I wrote like a 5 page treatise on program control flow, mentioning Turing and the Benedict Cumberbatch movie, child labor and farm work, asking if it is possible to turn any program, with its parallelism and entire event driven complexity to a Turing machine for better debugging. It was boring, so I removed it all. Instead, I will talk about conditional statements and how to refactor them, when it is needed.

A conditional statement is one of the basic statements of programming, it is a decision that affects what the program will execute next. And we love telling computers what to do, right? Anyway, here are some ways if/then/else or switch statements are used, some bad, some good, and how to fix whatever problems we find.

Team Arrow


First of all, the arrow antipattern. It's when you have if blocks in if blocks until your code looks like an arrow pointing right:
if (data.isValid()) {
if (data.items&&data.items.length) {
var item=data.items[0];
if (item) {
if (item.isActive()) {
console.log('Oh, great. An active item. hurray.');
} else {
throw "Item not active! Fatal terror!";
}
}
}
}
This can simply be avoided by putting all the code in a method and inverting the if branches, like this:
if (!data.isValid()) return;
if (!data.items||!data.items.length) return;
var item=data.items[0];
if (!item) return;
if (!item.isActive()) {
throw "Item not active! Fatal terror!";
}
console.log('Oh, great. An active item. hurray.');
See? No more arrow. And the debugging is so much easier.

There is a sister pattern of The Arrow called Speedy. OK, that's a Green Arrow joke, I have no idea how it is called, but basically, since a bunch of imbricated if blocks can be translated into a single if with a lot of conditions, the same code might have looked like this:
if (data.isValid()&&data.items&&data.items.length&&data.items[0]) {
var item=data.items[0];
if (!item.isActive()) {
throw "Item not active! Fatal terror!";
}
console.log('Oh, great. An active item. hurray.');
}
While this doesn't look like an arrow, it is in no way a better code. In fact it is worse, since the person debugging this will have to manually check each condition to see which one failed when a bug occurred. Just remember that if it doesn't look like an arrow, just its shaft, that's worse. OK, so now I named it: The Shaft antipattern. You first heard it here!

There is also a cousin of these two pesky antipatterns, let's call it Black Shaft! OK, no more naming. Just take a look at this:
if (person&&person.department&&person.department.manager&&person.department.manager.phoneNumber) {
call(person.department.manager.phoneNumber);
}
I can already hear a purist shouting at their monitor something like "That's because of the irresponsible use of null values in all programming languages!". Well, null is here to stay, so deal with it. The other problem is that you often don't see a better solution to something like this. You have a hierarchy of objects and any of them might be null and you are not in a position where you would cede control to another piece of code based on which object you refer to. I mean, one could refactor things like this:
if (person) {
person.callDepartmentManager();
}
...
function callDepartmentManager() {
if (this.department) {
this.department.callManager();
}
}
, which would certainly solve things, but adds a lot of extra code. In C# 6 you can do this:
var phoneNumber = person?.department?.manager?.phoneNumber;
if (phoneNumber) {
call(phoneNumber);
}
This is great for .NET developers, but it also shows that rather than convince people to use better code practices, Microsoft decided this is a common enough problem it needed to be addressed through features in the programming language itself.

To be fair, I don't have a generic solution for this. Just be careful to use this only when you actually need it and handle any null values with grace, rather than just ignore them. Perhaps it is not the best place to call the manager in a piece of code that only has a reference to a person. Perhaps a person that doesn't seem to be in any department is a bigger problem than the fact you can't find their manager's phone number.

The Omnipresent Switch


Another smelly code example is the omnipresent switch. You see code like this:
switch(type) {
case Types.person:
walk();
break;
case Types.car:
run();
break;
case Types.plane:
fly();
break;
}
This isn't so bad, unless it appears in a lot of places in your code. If that type variable is checked again and again and again to see which way the program should behave, then you probably can apply the Replace Conditional with Polymorphism refactoring method.



Or, in simple English, group all the code per type, then only decide in one place which of them you want to execute. Polymorphism might work, but also some careful rearranging of your code. If you think of your code like you would a story, then this is the equivalent of the annoying "meanwhile, at the Bat Cave" switch. No, I want to see what happens at the Beaver's Bend, don't fucking jump to another unrelated segment! Just try to mentally filter all switch statements and replace them with a comic book bubble written in violently zigzagging font: "Meanwhile...".

A similar thing is when you have a bool or enum parameter in a method, completely changing the behavior of that method. Maybe you should use two different methods. I mean, stuff like:
function doWork(iFeelLikeIt) {
if (iFeelLikeIt) {
work();
} else {
fuckIt();
}
}
happens every day in life, no need to see it in code.

Optimizing in the wrong place


Let's take a more serious example:
function stats(arr,method) {
if (!arr||!arr.length) return;
arr.sort();
switch (method) {
case Methods.min:
return arr[0];
case Methods.max:
return arr[arr.length-1];
case Methods.median:
if (arr.length%2==0) {
return (arr[arr.length/2-1]+arr[arr.length/2])/2;
} else {
return arr[Math.ceiling(arr.length/2)];
}
case Methods.mode:
var counts={};
var max=-1;
var result=-1;
arr.forEach(function(v) {
var count=(counts[v]||0)+1;
if (count>max) {
result=v;
max=count;
}
counts[v]=count;
});
return result;
case Methods.average:
var sum=0;
arr.forEach(function(v) { sum+=v; });
return sum/arr.length;
}
}

OK, it's still a silly example, but relatively less silly. It computes various statistical formulas from an array of values. At first, it seems like a good idea. You sort the array that works for three out of five methods, then you write the code for each, which is greatly simplified by working with a sorted array. Yet for the last two, being sorted does nothing and both of them have loops through the array. Sorting the array would definitely loop through the array as well. So, let's move the decision earlier:
function min(arr) {
if (!arr||!arr.length) return;
return Math.min.apply(null,arr);
}

function max(arr) {
if (!arr||!arr.length) return;
return Math.max.apply(null,arr);
}

function median(arr) {
if (!arr||!arr.length) return;
arr.sort();
var half=Math.ceiling(arr.length/2);
if (arr.length%2==0) {
return (arr[half-1]+arr[half])/2;
} else {
return arr[half];
}
}

function mode(arr) {
if (!arr||!arr.length) return;
var counts={};
var max=-1;
var result=-1;
arr.forEach(function(v) {
var count=(counts[v]||0)+1;
if (count>max) {
result=v;
max=count;
}
counts[v]=count;
});
return result;
}

function average(arr) {
if (!arr||!arr.length) return;
return arr.reduce(function (p, c) {
return p + c;
}) / arr.length;
}

As you can see, I only use sorting in the median function - and it can be argued that I could do it better without sorting. The names of the functions now reflect their functionalities. The min and max functions take advantage of the native min/max functions of Javascript and other than the check for a valid array, they are one liners. More than this, it was natural to use various ways to organize my code for each method; it would have felt weird, at least for me, to use forEach and reduce and sort and for loops in the same method, even if each was in its own switch case block. Moreover, now I can find the min, max, mode or median of an array of strings, for example, while an average would make no sense, or I can refactor each function as I see fit, without caring about the functionality of the others.

Yet, you smugly point out, each method uses the same code to check for the validity of the array. Didn't you preach about DRY a blog post ago? True. One might turn that into a function, so that there is only one point of change. That's fair. I concede the point. However don't make the mistake of confusing repeating a need with repeating code. In each of the functions there is a need to check for the validity of the input data. Repeating the code for it is not only good, it's required. But good catch, reader! I wouldn't have thought about it myself.

But, you might argue, the original function was called stats. What if a manager comes and says he wants a function that calculates all statistical values for an array? Then the initial sort might make sense, but the switch doesn't. Instead, this might lead to another antipattern: using a complex function only for a small part of its execution. Something like this:
var stats=getStats(arr);
var middle=(stats.min+stats.max)/2;
In this case, we only need the minimum and maximum of an array in order to get the "middle" value, and the code looks very elegant, yet in the background it computes all the five values, a waste of resources. Is this more readable? Yes. And in some cases it is preferred to do it like that when you don't care about performance. So this is both a pattern and an antipattern, depending on what is more important to your application. It is possible (and even often encountered) to optimize too much.

The X-ifs


A mutant form of the if statement is the ternary operator. My personal preference is to use it whenever a single condition determines one value or another. I prefer if/then/else statements to actual code execution. So I like this:
function boolToNumber(b) {
return b?1:0;
}

function exec(arr) {
if (arr.length%2==0) {
split(arr,arr.length/2);
} else {
arr.push(newValue());
}
}
but I don't approve of this:
function exec(arr) {
arr.length%2
? arr.push(newValue())
: split(arr,arr.length/2);
}

var a;
if (x==1) {
a=2;
} else {
a=6;
}

var a=x==1
? 2
: (y==2?5:6);
The idea is that the code needs to be readable, so I prefer to read it like this. It is not a "principle" to write code as above - as I said it's a personal preference, but do think of the other people trying to make heads and tails of what you wrote.

We are many, you are but one


There is a class of multiple decision flow that is hard to immediately refactor. I've talked about if statements that do the entire work in one of their blocks and of switch statements that can be easily split into methods. However there is the case where you want to do things based on the values of multiple variables, something like this:
if (x==1) {
if (y==1) {
console.log('bottom-right');
} else {
console.log('top-right');
}
} else {
if (y==1) {
console.log('bottom-left');
} else {
console.log('top-left');
}
}
There are several ways of handling this. One is to, again, try to move the decision on a higher level. Example:
if (x==1) {
logRight(y);
} else {
logLeft(y);
}
Of course, this particular case can be fixed through computation, like this:
var h=y==1?'right':'left';
var v=x==1?'bottom':'top';
console.log(v+'-'+h);
Assuming it was not so simple, though, we can choose to reduce the choice to a single decision:
switch(x+','+y) {
case '0,0': console.log('top-left'); break;
case '0,1': console.log('bottom-left'); break;
case '1,0': console.log('top-right'); break;
case '1,1': console.log('bottom-right'); break;
}



The Lazy Event


Another more complicated issue regarding conditional statements is when they are not actually encoding a decision, but testing for a change. Something like:
if (current!=prev) {
clearData();
var data=computeData(current);
setData(data);
prev=current;
}
This is a perfectly valid piece of code and in many situations is what is required. However, one must pay attention to the place where the decision gets taken as compared with the place the value changed. Isn't that more like an event handler, something that should be designed differently, architecture wise? Why keep a previous value and react to the change only when I get into this piece of code and not react to the change of the value immediately? Fire an event that the value is changed and subscribe to the event via a piece of code that refreshes the data. One giveaway for this is that in the code above there is no actual use of the prev value other than to compare it and set it.

Generalizations


As a general rule, try to take the decisions that are codified by if and switch statements as early as possible. The code must be readable to humans, sometimes in detriment of performance, if it is not essential to the functionality of your program. Avoid decision statements within other decision statements (arrow ifs, ternary operator in a ternary operator, imbricated switch and if statements). Split large pieces of code into small, easy to understand and properly named, methods (essentially creating a lower level than your conditional statement, thus relatively taking it higher in the code hierarchy).

What's next


I know this is a lower level programming blog post, but not everyone reading my blog is a senior dev - I mean, I hope so, I don't want to sound completely stupid. I am planning some new stuff, related to my new work project, but it might take some time until I understand it myself. Meanwhile, I am running out of ideas for my 100 days of writing about code challenge, so suggestions are welcome. And thank you for reading so far :)

and has 6 comments

Intro


I want to talk today about principles of software engineering. Just like design patterns, they range from useful to YAA (Yet Another Acronym). Usually, there is some guy or group of people who decide that a set of simple ideas might help software developers write better code. This is great! Unfortunately, they immediately feel the need to assign them to mnemonic acronyms that make you wonder if they didn't miss some principles from their sets because they were bad at anagrams.

Some are very simple and not worth exploring too much. DRY comes from Don't Repeat Yourself, which basically means don't write the same stuff in multiple places, or you will have to keep them synchronized at every change. Simply don't repeat yourself. Don't repeat yourself. See, it's at least annoying. KISS comes from Keep It Simple, Silly - yeah, let's be civil about it - anyway, the last letter is there just so that the acronym is actually a word. The principle states that avoiding unnecessary complexity will make your system more robust. A similar principle is YAGNI (You Aren't Gonna Need It - very New Yorkish sounding), which also frowns upon complexity, in particular the kind you introduce in order to solve a possible future problem that you don't have.

If you really want to fill your head with principles for software engineering, take a look at this huge list: List of software development philosophies.

But what I wanted to talk about was SOLID, which is so cool that not only does it sound like something you might want your software project to be, but it's a meta acronym, each letter coming from another acronym:
  • S - SRP
  • O - OCP
  • L - LSP
  • I - ISP
  • D - DIP

OK, I was just making it look harder than it actually is. Each of the (sub)acronyms stands for a principle (hence the last P in each) and even if they have suspiciously sounding names that hint on how much someone wanted to call their principles SOLID, they are really... err... solid. They refer specifically to object oriented programming, but I am sure they apply to all types. Let's take a look:

Single Responsibility


The idea is that each of your classes (or modules, units of code, functions, etc) should strive towards only one functionality. Why? Because if you want to change your code you should first have a good reason, then you should know where is that single (DRY) point where that reason applies. One responsibility equals one and only one reason to change.

Short example:
function getTeam() {
var result=[];
var strongest=null;
this.fighters.forEach(function(fighter) {
result.push(fighter.name);
if (!strongest||strongest.power<fighter.power) {
strongest=fighter;
}
});
return {
team:result,
strongest:strongest;
};
}

This code iterates through the list of fighters and returns a list of their names. It also finds the strongest fighter and returns that as well. Obviously, it does two different things and you might want to change the code to have two functions that each do only one. But, you will say, this is more efficient! You iterate once and you get two things for the price of one! Fair enough, but let's see what the disadvantages are:
  • You need to know the exact format of the return object - that's not a big deal, but wouldn't you expect to have a team object returned by a getTeam function?
  • Sometimes you might want just the list of fighters, so computing the strongest is superfluous. Similarly, you might only want the strongest player.
  • In order to add stuff in the iteration loop, the code has become more complex - at least when reading it - than it has to be.

Here is how the code could - and should - have looked. First we split it into two functions:
function getTeam() {
var result=[];
this.fighters.forEach(function(fighter) {
result.push(fighter.name);
});
return result;
}

function getStrongestFighter() {
var strongest=null;
this.fighters.forEach(function(fighter) {
if (!strongest||strongest.power<fighter.power) {
strongest=fighter;
}
});
return strongest;
}
Then we refactor it to something simple and readable:
function getTeam() {
return this.fighters
.map(function(fighter) { return fighter.name; });
}

function getStrongestFighter() {
return this.fighters
.reduce(function(strongest,val) {
return !strongest||strongest.power<val.power?val:strongest;
});
}

Open/Closed


When you write your code the last thing you want to do is go back to it and change it again and again whenever you implement a new functionality. You want the old code to work, be tested to work, and allow new functionality to be built on top of it. In object oriented programming that simply means you can extend classes, but practically it also means the entire plumbing necessary to make this work seamlessly. Simple example:
function Fighter() {
this.fight();
}
Fighter.prototype={
fight : function() {
console.log('slap!');
}
};

function ProfessionalFighter() {
Fighter.apply(this);
}
ProfessionalFighter.prototype={
fight : function() {
console.log('punch! kick!');
}
};

function factory(type) {
switch(type) {
case 'f': return new Fighter();
case 'pf': return new ProfessionalFighter();
}
}

OK, this is a silly example, since I am lazily emulating inheritance in a prototype based language such as Javascript. But the result is the same: both constructors will .fight by default, but each of them will have different implementations. Note that while I cannibalized bits of Fighter to build ProfessionalFighter, I didn't have to change it at all. I also built a factory method that returns different objects based on the input. Both possible returns will have a .fight method.

The open/closed principle also applies to non inheritance based languages and even in classic OOP languages. I believe it leads to a natural outcome: preferring composition over inheritance. Write your code so that the various modules just seamlessly connect to each other, like Lego blocks, to build your end product.

Liskov substitution principle


No, L is not coming from any word that they could think of, so they dragged poor Liskov into it. Forget the definition, it's really stupid and complicated. Not KISSy at all! Assume you have the base class Fighter and the more specialized subclass ProfessionalFighter. If you had a piece of code that uses a Fighter object, then this principle says you should be able to replace it with a ProfessionalFighter and the piece of code would still be correct. It may not be what you want, but it would work.

But when does a subclass break this principle? One very ugly antipattern that I have seen is when general types know about their subtypes. Code like "if I am a professional fighter, then I do this" breaks almost all SOLID principles and it is stupid. Another case is when the general class has everything they can think of, either abstract or implemented as empty or throwing an error, then the base classes implement one or the other of the functions. Dude! If your fighter doesn't implement .fight, then it is not a fighter!

I don't want to add code to this, since the principle is simple enough. However, even if it is an idea that primarily applies to OOP it doesn't mean it doesn't have its uses in other types of programming languages. One might say it is not even related to programming languages, but to concepts. It basically says that an orange should behave like an orange if it's blue, otherwise don't call it a blue orange!

Interface segregation principle


This one is simple. It basically is the reverse of the Liskov: split your interfaces - meaning the desired functionality of your code - into pieces that are fully used. If you have a piece of code that uses a ProfessionalFighter, but all it does is use the .fight method, then use a Fighter, instead. Silly example:
public class Fighter {
public virtual void Fight() {
Console.WriteLine("slap!");
}
}

public class EnglishFighter {
public override void Fight() {
Console.WriteLine("box!");
}
public override void Talk() {
Console.WriteLine("Oy!");
}
}

class Program {
public static void Main() {
EnglishFighter f=getMeAFighter();
f.Fight();
}
}

I don't even know if it's valid code, but anyway, the idea here is that there is no reason for me to declare and use the variable f as an EnglishFighter, if all it does is fight. Use a Fighter type. And a YAGNI on you if you thought "wait, but what if I want him to talk later on?".

Dependency inversion principle


Oh, this is a nice one! But it doesn't live in a vacuum. It is related to both SRP and OCP as it states that high level modules should not depend on low level modules, only on their abstractions. In other words, use interfaces instead of implementations wherever possible.

I wrote an entire other post about Inversion of Control, the technique that allows you to properly use and enjoy interfaces, while keeping your modules independent, so I am not going to repeat things here. As a hint, simply replacing Fighter f=new Fighter() with IFighter f=new Fighter() is not enough. You must use a piece of code that decides for you what implementation of an interface will be used, something like IFighter f=GetMeA(typeof(IFighter)).

The principle is related to SRP because it says a boss should not be controlling or depending on what particular things the employees will do. Instead, he should just hire some trustworthy people and let them do their job. If a task depends so much on John that you can't ever fire him, you're in trouble. It is also related to OCP because the boss will behave like a boss no matter what employee changes occur. He may hire more people, replace some, fire some others, the boss does not change. Nor does he accept any responsibility, but that's a whole other principle ;)

Conclusions


I've explored some of the more common acronyms from hell related to software development and stuck more to SOLID, because ... well, you have to admit, calling it SOLID works! Seriously now, following principles such as these (choose your own, based on your own expertise and coding style) will help you a lot later on, when the complexity of your code will explode. In a company there is always that one poor guy who knows everything and can't work well because everybody else is asking him why and how something is like it is. If the code is properly separated on solid principles, you just need to look at it and understand what it does and keep the efficiency of your changes high. One small change for man , like. Let that unsung hero write their code and reach software Valhalla.

SRP keeps modules separated, OCP keeps code free of need for modifications, LSP and ISP make you use the basest of classes or interfaces, reinforcing the separation of modules, while DIP is helping you sharpen the boundaries between modules. Taken together, SOLID principles are obviously about focus, allowing you to work on small, manageable parts of the code without needing knowledge of others or having to make changes to them.

Keep coding!

IEnumerable/IEnumerator - the iterator design pattern implemented in the C# language


C# started up with the IEnumerable interface, which exposed one method called GetEnumerator. This method would return a specialized object (implementing IEnumerator) that could give you from a collection of items the current item and could also advance. Unlike Java on which C# is originally based, this interface was inherited by even the basest of collection objects, like arrays. This allowed a special construct in the language: foreach. One uses it very simply:
for (var item in ienumerable) // do something
. No need to know how or from where the item is exactly retrieved from the collection, just get the first, then the next and so on until there are no more items.

In .NET 2.0 generics came, along their own interfaces like IEnumerable<T>, holding items of a specific type, but the logic is the same. It also introduced another language element called yield. One didn't need to write an IEnumerator implementation anymore, they could just define a method that returned an IEnumerable and inside "yield return" values. Something like this:
public class Program
{
public static void Main(string[] args)
{
var enumerator = Fibonacci().GetEnumerator();
for (var i = 0; enumerator.MoveNext() && i < 10; i++)
{
var v = enumerator.Current;
Console.WriteLine(v);
}
Console.ReadKey();
}

public static IEnumerable<int> Fibonacci()
{
var i1 = 0;
var i2 = 1;
while (true)
{
yield return i2;
i2 += i1;
i1 = i2 - i1;
}
}
}

This looks a bit weird. A method is running a while(true) loop with no breaks. Shouldn't it block the execution of the program? No, because of the yield construct. While the Fibonacci series is infinite, we would only get the first 10 values. You can also see how the enumerator works, when used explicitly.

Iterators and generators in Javascript ES6


EcmaScript version 6 (or ES6 or ES2015) introduced the same concepts in Javascript. An iterator is just an object that has a next() method, returning an object containing the value and done properties. If done is true, value is disregarded and the iteration stops, if not, value holds the current value. An iterable object will have a method that returns an iterator, the method's name being Symbol.iterator. The for...of construct of the language iterates the iterable. String, Array, TypedArray, Map and Set are all built-in iterables, because the prototype objects of them all have a Symbol.iterator method. Example:
var iterable=[1,2,3,4,5];
for (v of iterable) {
console.log(v);
}

But what about generating values? Well, let's do it using the knowledge we already have:
var iterator={
i1:0,
i2:1,
next:function() {
var result = { value: this.i2 }
this.i2+=this.i1;
this.i1=this.i2-this.i1;
return result;
}
};

var iterable = {};
iterable[Symbol.iterator]=() => iterator;

var iterator=iterable[Symbol.iterator]();
for (var i=0; i<10; i++) {
var v=iterator.next();
console.log(v.value);
}

As you can see, it is the equivalent of the Fibonacci code written in C#, but look at how unwieldy it is. Enter generators, a feature that allows, just like in C#, to define functions that generate values and the iterable associated with them:
function* Fibonacci() {
var i1=0;
var i2=1;
while(true) {
yield i2;
i2+=i1;
i1=i2-i1;
}
}

var iterable=Fibonacci();

var iterator=iterable[Symbol.iterator]();
for (var i=0; i<10; i++) {
var v=iterator.next();
console.log(v.value);
}

No, that's not a C pointer, thank The Architect, it's the way Javascript ES6 defines generators. Same code, much clearer, very similar to the C# version.

Uses


OK, so these are great for mathematics enthusiasts, but what are we, regular dudes, going to do with iterators and generators? I mean, for better or for worse we already have for and .forEach in Javascript, what do we need for..of for? (pardon the pun) And what do generators help with?

Well, in truth, one could get away simply without for..of. The only object where .forEach works differently is the Map object, where it returns only the values, as different from for..of which returns arrays of [key,value]. However, considering generators are new, I would expect using for..of with them to be more clear in code and more inline with what foreach does in C#.

Generators have the ability to easily define series that may be infinite or of items which are expensive resources to get. Imagine a download of a large file where each chunk is a generated item. An interesting use scenario is when the .next function is used with parameters. This is Javascript, so an iterator having a .next method only means it has to be named like that. You can pass an arbitrary number of parameters. So here it is, a generator that not only dumbly spews out values, but also takes inputs in order to do so.

In order to thoroughly explore the value of iterators and generators I will use my extensive knowledge in googling the Internet and present you with this very nice article: The Hidden Power of ES6 Generators: Observable Async Flow Control which touches on many other concepts like async/await (oh, yeah, that should be another interesting C# steal), observables and others that are beyond the scope of this article.



I hope you liked this short intro into these interesting new features in ES6.

Can you tell me what is the difference between these to pieces of code?
var f=function() {
alert('f u!');
}

function f() {
alert('f u!');
}

It's only one I can think of and with good code hygiene it is one that should never matter. It is related to 'hoisting', or the idea that in Javascript a variable declaration is hoisted to the top of the function or scope before execution. In other words, I can see there is a function f before the declarations above. And now the difference: 'var f = function' will have its declaration hoisted, but not its definition. f will exist at the beginning of the scope, but it will be undefined; the 'function f' format will have both declaration and definition hoisted, so that at the beginning of the scope you will have available the function for execution.

Intro


I am not the authoritative person to go to for Javascript Promises, but I've used them extensively (pardon the pun) in Bookmark Explorer, my Chrome extension. In short, they are a way to handle methods that return asynchronously or that for whatever reason need to be chained. There are other ways to do that, for example using Reactive Extensions, which are newer and in my opinion better, but Rx's scope is larger and that is another story altogether.

Learning by examples


Let's take a classic and very used example: AJAX calls. The Javascript could look like this:
function get(url, success, error) {
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function () {
if (this.readyState == 4) {
if (this.status == 200) {
success(this.response);
} else {
error(this.status);
}
}
};
xhttp.open("GET", url, true);
xhttp.send();
}

get('/users',function(users) {
//do something with user data
},function(status) {
alert('error getting users: '+status);
})

It's a stupid example and not by far complete, but it shows a way to encapsulate an AJAX call into a function that receives an URL, a handler for success and one for error. Now let's complicate things a bit. I want to get the users, then for each active user I want to get the document list, then return the ones that contain a string. For simplicity's sake, let's assume I have methods that do all that and receive a success and an error handler:
var search = 'some text';
var result = [];
getUsers(function (users) {
users
.filter(function (user) {
return user.isActive();
})
.forEach(function (user) {
getDocuments(user, function (documents) {
result = result.concat(documents.filter(function (doc) {
return doc.text.includes(text);
}));
}, function (error) {
alert('Error getting documents for user ' + user + ': ' + error);
});
});
}, function (error) {
alert('Error getting users: ' + error);
});

It's already looking wonky. Ignore the arrow anti pattern, there are worse issues. One is that you never know when the result is complete. Each call for user documents takes an indeterminate amount of time. This async programming is killing us, isn't it? We would have much better liked to do something like this instead:
var result = [];
var search = 'some text';
var users = getUsers();
users
.filter(function (user) {
return user.isActive();
})
.forEach(function (user) {
var documents = getDocuments(user);
result = result.concat(documents.filter(function (doc) {
return doc.text.includes(text);
}));

});
First of all, no async, everything is deterministic and if the function for getting users or documents fails, well, it can handle itself. When this piece of code ends, the result variable holds all information you wanted. But it would have been slow, linear and simply impossible in Javascript, which doesn't even have a Pause/Sleep option to wait for stuff.

Now I will write the same code with methods that use Promises, then proceed on explaining how that would work.
var result = [];
var search = 'some text';
var userPromise = getUsers();
userPromise.then(function (users) {
var documentPromises = users
.filter(function (user) {
return user.isActive();
})
.map(function (user) {
return getDocuments(user);
});
var documentPromise = Promise.all(documentPromises);
documentPromise
.then(function (documentsArray) {
documentsArray.forEach(function (documents) {
result = result.concat(documents.filter(function (doc) {
return doc.text.includes(search);
});
});
// here the result is complete
})
.catch (function (reason) {
alert('Error getting documents:' + reason);
});
});
Looks more complicated, but that's mostly because I added some extra variables for clarity.

The first thing to note is that the format of the functions doesn't look like the async callback version, but like the synchronous version: var userPromise=getUsers();. It doesn't return users, though, it returns the promise of users. It's like a politician function. This Promise object encapsulates the responsibility of announcing when the result is actually available (the .then(function) method) or when the operation failed (the .catch(function) method). Now you can pass that object around and still use its result (successful or not) when available at whatever level of the code you want it.

At the end I used Promise.all which handles all that uncertainty we were annoyed about. Not only does it publish an array of all the document getting operations, but the order of the items in the array is the same as the order of the original promises array, regardless of the time it took to execute any of them. Even more, if any of the operations fails, this aggregate Promise will immediately exit with the failure reason.

To exemplify the advantages of using such a pattern, let's assume that getting the users sometimes fails due to network errors. The Internet is not the best in the world where the user of the program may be, so you want to not fail immediately, instead retry the operation a few times before you do. Here is how a getUsersWithRetry would look:
function getUserWithRetry(times, spacing) {
var promise = new Promise(function (resolve, reject) {
var f = function () {
getUsers()
.then(resolve)
.catch (function (reason) {
if (times <= 0) {
reject(reason);
} else {
times--;
setTimeout(f, spacing);
}
});
}
f();
});
return promise;
}

What happens here? First of all, like all the get* methods we used so far, we need to return a Promise object. To construct one we give it as a parameter a function that receives two other functions: resolve and reject. Resolve will be used when we have the value, reject will be used when we fail getting it. We then create a function so that it can call itself and in it, we call getUsers. Now, if the operation succeeds, we will just call resolve with the value we received. The operation would function exactly like getUsers. However, when it fails, it checks the number of times it must retry and only fails (calling reject) when that number is zero. If it still has retries, it calls the f function we defined, with a timeout defined in the parameters. We finally call the f function just once.



Here is another example, something like the original get function, but that returns a Promise:
function get(url) {
// Return a new promise.
return new Promise(function(resolve, reject) {
// Do the usual XHR stuff
var req = new XMLHttpRequest();
req.open('GET', url);

req.onload = function() {
// This is called even on 404 etc
// so check the status
if (req.status == 200) {
// Resolve the promise with the response text
resolve(req.response);
}
else {
// Otherwise reject with the status text
// which will hopefully be a meaningful error
reject(Error(req.statusText));
}
};

// Handle network errors
req.onerror = function() {
reject(Error("Network Error"));
};

// Make the request
req.send();
});
}
Copied it like a lazy programmer from JavaScript Promises: an Introduction.

Notes


An interesting thing to remember is that the .then() method also returns a Promise, so one can do stuff like get('/users').then(JSON.parse).then(function(users) { ... }). If the function called by .then() is returning a Promise, then that is what .then() will return, allowing for stuff like someOperation().then(someOtherOperation).catch(errorForFirstOperation).then(handlerForSecondOperation). There is a lot more about promises in the Introduction in the link above and I won't copy/paste it here.

The nice thing about Promises is that they have been around in the Javascript world since forever in various libraries, but only recently as native Javascript objects. They have reached a maturity that was tested through the various implementations that led to the one accepted today by the major browsers.

Promises solve some of the problems in constructing a flow of asynchronous operations. They make the code cleaner and get rid of so many ugly extra callback parameters that many frameworks used us to. This flexibility is possible because in Javascript functions are first class members, meaning they can be passed around and manipulated just like any other parameter type.

The disadvantages are more subtle. Some parts of your code will be synchronous in nature. You will have stuff like a=sum(b,c); in functions that return values. Suddenly, functions don't return actual values, but promised values. The time they take to execute is also unclear. Everything is in flux.

Conclusion


I hope this has opened your eyes to the possibilities of writing your code in a way that is both more readable and easy to encapsulate. Promises are not limited to Javascript, many other languages have their own implementations. As I was saying in the intro, I feel like Promises are a simple subcase of Reactive Extensions streams in the sense that they act like data providers, but are limited to only one possible result. However, this simplicity may be more easy to implement and understand when this is the only scenario that needs to be handled.

and has 0 comments
A few years ago I wrote a post titled Software Patterns are Useless in which I was drawing attention to the overhyping of the concept of software design patterns. In short, they are either too simple or too complex to bundle together into a concept to be studied, much less used as a lingua franca for coders.

I still feel this way, but in this post I want to explore what I think the differences are between software design patterns and building architecture design patterns. In the beginning, as you no doubt know from the maniacal introduction in design patternese by any of its enthusiasts, it was a construction architect that noticed common themes in solving the problems required to building something. This happened around the time I was born, so 40 years ago. In about ten years, software people started to import the concept and created this collection of general solutions to software problems. It was a great idea, don't get me wrong.

However, things have definitely changed in software since then. While we still meet some of the problems we invented software design patterns for, the problems or at least their scope has changed immensely. To paraphrase a well known joke, design patterns would be similar in construction and software if you would first construct a giant robot to build your house by itself. But it doesn't happen like this, instead the robot is actually the company itself, its purpose being to take your house specifications and build it. So how useful would a "pattern" be for a construction company saying "if you need a house, hire a construction company"?

Look at MVC, a software design pattern that is implemented everywhere in the web world, at different levels of abstractions. You use ASP.Net MVC, for example, to separate the logic from the rendering on the server, but then you use Angular to separate logic from rendering on the client. We are very clever young men, but it's MVCs all the way down! How is that a pattern anymore? "If you want to separate rendering from logic, use MVC" "How do I implement it?" "Oh, just use a framework that was built for you". At the other end of the spectrum there are things that are so simple and basically embedded in programming languages that thinking of them as patterns is pointless. Iterator is one of them. Now even Javascript has a .forEach construct, not to mention generators and iterators. Or Abstract Factory - isn't that just a factory for factories? Should I invent a new name for the pattern that creates a factory of factories for factories?

But surely there are patterns that are very useful and appropriate for our work today. Like the factory pattern, or the builder, the singleton or inversion of control and so many others. Of course they are, I am not debating that at all. I am going to present inversion of control soon to a group of developers. I even find it useful to split these patterns into categories like creational, structural, behavioral, concurrency, etc. Classification in general is a good thing. It's the particulars that bother me.

Take the seminal book Design Patterns: Elements of Reusable Object-Oriented Software (Gamma et al. 1994). And by seminal I mean so influential that the Wikipedia URL is Design_Patterns. Not (book) or the complete title or anything. If you followed the software design patterns link from higher up you noticed that design patterns are still organized around those 23 original ones from the book written by "the Gang of Four" more than twenty years ago.

I want to tell you that this is wrong, but I've seen it in so many other fields. There is one work that is so ground breaking that it becomes the new ground. Everything else seems to build upon it from then on. You cannot be taken seriously unless you know about it and what it contains. Any personal contribution of yours must needs reference the great work. All the turtles are stacked upon it. And it is dangerous, encouraging stagnation and trapping you in this cage that you can only get out of if you break it.

Does that mean that I am writing my own book that will break ground and save us all? No. The patterns, as described and explained in so many ways around the 'net are here to stay. However note the little changes that erode the staleness of a rigid classification, like "fluent interfaces" being used instead of "builder pattern", or the fact that no one in their right mind will try to show off because they know what an iterator is, or the new patterns that emerge not from evaluating multiple cases and extracting a common theme, but actually inventing a pattern to solve the problems that are yet to come, like MVVM.

I needed to write this post, which very much mirrors the old one from a few years ago, because I believe calling a piece of software that solves a problem a pattern increasingly sounds more like patent. Like biologists walking the Earth in the hope they will find a new species to name for themselves, developers are sometimes overdefining and overusing patterns. Remember it is all flexible, that a pattern is supposed to be a common theme in written code, not a rigid way of writing your code from a point on. If software patterns are supposed to be the basis of a common language between developers, remember that all languages are dynamic, alive, that no one speaks the way the dictionaries and manuals tell you to speak. Let your mind decide how to write and next time someone scoffs at you asking "do you know what the X pattern is?" you respond with "let me google that for you". Let patterns be guides for the next thing to come, not anchors to hold you back.

I watched this Beau teaches JavaScript video and I realized how powerful Proxy, this new feature of EcmaScript version 6, is. In short, you call new Proxy(someObject, handler) and you get an object that behaves just like the original object, but has your code intercept most of the access to it, like when you get/set a property or method or when you ask if the object has a member by name. It is great because I feel I can work with normal Javascript, then just insert my own logging, validation or some other checking code. It's like doing AOP and metaprogramming in Javascript.

Let's explore this a little bit. The video in the link above already shows some way to do validation, so I am going to create a function that takes an object and returns a proxy that is aware of any modification to the original object, adding the isDirty property and clearDirt() method.

function dirtify(obj) {
return new Proxy(obj,{
isDirty : false,
get : function(target, property, receiver) {
if (property==='isDirty') return this.isDirty;
if (property==='clearDirt') {
var self=this;
var f = function() {
self.isDirty=false;
};
return f.bind(target);
}
console.log('Getting '+property);
return target[property];
},
has : function(target, property) {
if (property==='isDirty'||property==='clearDirt') return true;
console.log('Has '+property+'?');
return property in target;
},
set : function(target, property, value, receiver) {
if (property==='isDirty'||property==='clearDirt') return false;
if (target[property]!==value) this.isDirty=true;
console.log('Setting '+property+' to '+JSON.stringify(value));
target[property]=value;
return true;
},
deleteProperty : function(target, property) {
if (property==='isDirty'||property==='clearDirt') return false;
console.log('Delete '+property);
if (target[property]!=undefined) this.isDirty=true;
delete target[property];
return true;
}
});
}
var obj={
x:1
};
var proxy=dirtify(obj);
console.log('x' in proxy); //true
console.log(proxy.hasOwnProperty('x')); //true
console.log('isDirty' in proxy); //true
console.log(proxy.x); //1
console.log(proxy.hasOwnProperty('isDirty')); //false
console.log(proxy.isDirty); //false

proxy.x=2;
console.log(proxy.x); //2
console.log(proxy.isDirty); //true

proxy.clearDirt();
console.log(proxy.isDirty); //false

proxy.isDirty=true;
console.log(proxy.isDirty); //false
delete proxy.isDirty;
console.log(proxy.isDirty); //false

delete proxy.x;
console.log(proxy.x); //undefined
console.log(proxy.isDirty); //true

proxy.clearDirt();
proxy.y=2;
console.log(proxy.isDirty); //true

proxy.clearDirt();
obj.y=3;
console.log(obj.y); //3
console.log(proxy.y); //3
console.log(proxy.isDirty); //false

So, here I am returning a proxy that logs any access to members to the console. It also simulates the existence of isDirty and clearDirt members, without actually setting them on the object. You see that when setting the isDirty property to true, it still reads false. Any setting of a property to a different value or deleting an existing property is setting the internal isDirty property to true and the clearDirt method is setting it back to false. To make it more interesting, I am returning true for the 'in' operator, but not for the hasOwnProperty, when querying if the attached members exist. Also note that this is a real proxy, if you change a value in the original object, the proxy will also reflect it, but without intercepting the change.

Imagine the possibilities!

More info:
ES6 Proxy in Depth
Metaprogramming with proxies
Metaprogramming in ES6: Part 3 - Proxies

and has 0 comments
As part of a self imposed challenge to write a blog post about code on average every day for about 100 days, I am discussing a link that I've previously shared on Facebook, but that I liked so much I feel the need to contemplate some more: The Caching Antipattern

Basically, what is says is that caching is done wrong in any of these cases:
  • Caching at startup - thus admitting that your dependencies are too slow to begin with
  • Caching too early in development - thus hiding the performance of the service you are developing
  • Integrated cache - cache is embedded and integral in the service code, thus breaking the single responsibility principle
  • Caching everything - resulting in an opaque service architecture and even recaching. Also caching things you think will be used, but might never be
  • Recaching - caches of caches and the nightmare of untangling and invalidating them in cascade
  • Unflushable cache - no method to invalidate the cache except restarting services

The alternatives are not to cache and instead know your data and service performance and tweak them accordingly. Try to take advantage of client caches such as the HTTP If-Modified-Since header and 304-not-modified return code whenever possible.

I've had the opportunity to work on a project that did almost everything in the list above. The performance bottleneck was the database, so the developers embedded an in code memory cache for resources that were not likely to change. Eventually other anti-patterns started to emerge, like having a filtered call (let's say a call asking for a user by id) getting all users and then selecting the one that had that id. Since it was in memory anyway, "getting" the entire list of records was just getting a reference to an already existing data structure. However, if I ever wanted to get rid of the cache or move it in an external service, I would have had to manually look for all cases of such presumptions in code and change them. If I wanted to optimize a query in the database I had no idea how to measure the performance as actually used in the product, in fact it led to a lack of interest in the way the database was used and a strong coupling of data provider implementation to the actual service code. Why use memory tables for often queried data? We have it cached anyway.

My take on this is that caching should always be separate from the service code. Have the code work, as slow as the data provider and external services allow it, then measure the performance and add caching where actually needed, as a separate layer of indirection. These days there are so many ways to do caching, from in memory tables in SQL Server, to distributed memory caches provided as a service by most cloud providers - such as Memcached or Redis in AWS, to content caches like Akamai, to html output caches like Varnish and to client caches, controlled by response and request headers like in the suggestion from the original article. Adding your own version is simply wasteful and error prone. Just like the data provider should be used through a thin interface that allows you to replace it at will, the caching layer should also be plug and play, thus allowing your application to remain unchanged, but able to upgrade any of its core features when better alternatives arrive.

There is also something to be said about reaching the limit of your resources. Let's say you cache everything your clients ask for in memory, when they ask for it. At one time or another you might reach the upper limit of your memory. At this time the cache should not fail, instead it should clear the data least used or the oldest inserted or something like that. A cache is not something that is supposed to hold all your data, only the part of it that is most efficient, performance wise, and it should never ever bring more problems like memory overflow crashes. Eek!

Now, I need to find a suitable name for my caching layer invalidation manager ;)

Other interesting resources about caching:
Cache (computing)
Caching Best Practices
Cache me if you can Powerpoint presentation
Caching guidance
Caching Techniques

The first thing that strikes anyone starting to use another IDE than the one they are used to is that all the key bindings are wrong. So they immediately google something like "X key bindings for Y" and they usually get an answer, since developers switching IDEs and preferring one in particular is quite a common situation. Not so with Eclipse. You have to install software, remove settings then still modify stuff. I am going to give you the complete answer here on how to switch Eclipse key bindings to the ones you are used to in Visual Studio.

Step 1
First follow the instructions in this Stack Overflow answer: How to Install Visual Studio Key Bindings in Eclipse (Helios onwards)
Short version: Go to Help → Install New Software, select your version in the Work with box, wait until the list populates, check the box next to Programming Languages → C/C++ Development Tools and install (with restart). After that go to Window → Preferences → General → Keys and change the Scheme in a dropdown to Microsoft Visual Studio.

Step 2
When Eclipse starts it shows you a Welcome screen. Disable the welcome screen by checking the box from the bottom-right corner and restart Eclipse. This is to avoid Ctrl-arrows not working in the editor as explained in this StackOverflow answer.

Step 3
While some stuff does work, others do not. It is time to go to Window → Preferences → General → Keys and start changing key bindings. It is a daunting task at first, since you have to find the command, set the shortcut in the zillion contexts that are available and so on. The strategy I found works best is this:
  • Right click on whatever item you want to affect with the keyboard shortcut
  • Find in the context menu whatever command you want to do
  • Remember the keyboard shortcut
  • Go to the key preferences and replace that shortcut everywhere (the text filter in the key bindings dialog allows searching for keyboard shortcuts)

You might want to share or at least backup your keyboard settings. No, the Export CSV option in the key bindings dialog gives you a file you can't import. The solution, as detailed here is to go to File → Export or Import → General → Preferences and work with .epf files. And if you think it gives you a nice list of key bindings that you can edit with a file editor, think again. The format holds the key binding scheme name, then only the custom changes, in a file that is what .ini and .xml would have if they decided on having children.

Now, the real decent thing would be to not go through Step 1 and instead just start from the default bindings and change them according to Visual Studio (2016, not 2005!!) and then export the .epf file in order for all people to enjoy a simple and efficient method of reaching their goal. I leave this as an exercise for the reader.

A short list of shortcuts that I found missing from the Visual Studio schema: rename variable on F2, go to declaration on F12, Ctrl-Shift-F for search across files, Ctrl-Minus to navigate backward ... and more to come I am sure.

and has 2 comments
Ugh! I will probably be working in Java for a while. Or I will kill myself, one of the two. So far I hate Eclipse, I can't even write code and I have to blog simple stuff like how to do regular expressions. Well, take it as a learning experience! Exiting my comfort zone! 2017! Ha, ha haaaaa [sob! sob!]

In Java you use Pattern to do regex:
Pattern p = Pattern.compile("a*b");
Matcher m = p.matcher("aaaaab");
boolean b = m.matches();

So let's do some equivalent code:

.NET:
var regDate=new Regex(@"^(?<year>(?:19|20)?\d{2})-(?<month>\d{1,2})-(?<day>\d{1,2})$",RegexOptions.IgnoreCase);
var match=regDate.Match("2017-02-14");
if (match.Success) {
var year=match.Groups["year"];
var month=match.Groups["month"];
var day=match.Groups["day"];
// do something with year, month, day
}

Java:
Pattern regDate=Pattern.compile("^(?<year>(?:19|20)?\\d{2})-(?<month>\\d{1,2})-(?<day>\\d{1,2})$", Pattern.CASE_INSENSITIVE);
Matcher matcher=regDate.matcher("2017-02-14");
if (matcher.find()) {
String year=matcher.group("year");
String month=matcher.group("month");
String day=matcher.group("day");
// do something with year, month, day
}

Notes:

The first thing to note is that there is no verbatim literal support in Java (the @"string" format from .NET) and there is no "var" (one always has to specify the type of the variable, even if it's fucking obvious). Second, the regular expression object Pattern doesn't do things directly, instead it creates a Matcher object that then does operations. The two bits of code above are not completely equivalent, as the Success property in a .NET Match object holds the success of the already performed operation, while .find() in the Java Matcher object actually performs the match.

Interestingly, it seems that Pattern is automatically compiling the regular expression, something that .NET must be directed to do. I don't know if the same term means the same thing for the two frameworks, though.

Another important thing is that it is more efficient to reuse matchers rather than recreate them. So when you want to use the matcher on another string, use matcher.reset("newstring").

And lastly, the string class itself has quick and dirty regular expression methods like .matches, replaceFirst and .replaceAll. The matches method only returns a bool if the string is a perfect match (equivalent to a Pattern match with ^ at the beginning and $ at the end).

Often we need to attach functions on Javascript events, but we need them to not be executed too often. Mouse move or scroll events can fire several times a second and executing some heavy computation directly would make everything slow and unresponsive. That's why we use a method called debounce, that takes your desired function and returns another function that will only get executed so many times in a time interval.



It has reached a certain kind of symmetry, so I like it this way. Let me explain how I got to it.

First of all, there is often a similar function used for debouncing. Everyone remembers it and codes it off the top of the head, but it is partly wrong. It looks like this:
function debounce(fn, wait) {
var timeout=null;
return function() {
var context=this;
var args=arguments;
var f=function(){ fn.apply(context,args); };
clearTimeout(timeout);
timeout=setTimeout(f,wait);
};
}
It seems OK, right? Just extend the time until the function gets executed. Well, the problem is that the first time you call the function you will have to wait before you see any result. If the function is called more often than the wait period, your code will never get executed. That is why Google shows this page as *the* debounce reference: JavaScript Debounce Function. And it works, but good luck trying to understand its flow so completely that you can code it from memory. My problem was with the callNow variable, as well as the rarity of cases when I would need to not call the function immediately the first time, thus making the immediate variable redundant.

So I started writing my own code. And it looked like the "casual" debounce function, with an if block added. If the timeout is already set, then just reset it; that's the expected behavior. When isn't this the expected behavior? When calling it the first time or after a long period of inactivity. In other words when the timeout is not set. And the code looked like this:
function debounce(fn, wait) {
var timeout=null;
return function() {
var context=this;
var args=arguments;
var f=function(){ fn.apply(context,args); };
if (timeout) {
clearTimeout(timeout);
timeout=setTimeout(f,wait);
} else {
timeout=setTimeout(function() {
clearTimeout(timeout);
timeout=null;
});
f();
}
};
}

The breakthrough came with the idea to use the timeout anyway, but with an empty function, meaning that the first time it is called, the function will execute your code immediately, but also "occupy" the timeout with an empty function. Next time it is called, the timeout is set, so it will be cleared and reset with a timeout using your initial code. If the interval elapses, then the timeout simply gets cleared anyway and next time the call of the function will be immediate. If we abstract the clearing of timeout and the setting of timeout in the functions c and t, respectively, we get the code you saw at the beginning of the post. Note that many people using setTimeout/clearTimeout are in the scenario in which they set the timeout immediately after they clear it. This is not always the case. clearTimeout is a function that just stops a timer, it does not change the value of the timeout variable. That's why, in the cases when you want to just clear the timer, I recommend also setting the timeout variable to null or 0.

For the people wanting to look cool, try this version:
function debounce(fn, wait) {
var timeout=null;
var c=function(){ clearTimeout(timeout); timeout=null; };
var t=function(fn){ timeout=setTimeout(fn,wait); };
return function() {
var context=this;
var args=arguments;
var f=function(){ fn.apply(context,args); };
timeout
? c()||t(f)
: t(c)||f();
}
}

Now, doesn't this look sharp? The symmetry is now obvious. Based on the timeout, you either clear it immediately and time out the function or you time out the clearing and execute the function immediately.

Update 26 Apr 2017: Here is an ES6 version of the function:
function debounce(fn, wait) {
let timeout=null;
const c=()=>{ clearTimeout(timeout); timeout=null; };
const t=fn=>{ timeout=setTimeout(fn,wait); };
return ()=>{
const context=this;
const args=arguments;
let f=()=>{ fn.apply(context,args); };
timeout
? c()||t(f)
: t(c)||f();
}
}

I am working on this issue that I can't seem to properly be able to solve. No matter what I do I can't shake the feeling of code smell from it. It involves a chain of validations and business operations that I want to separate. The simple scenario is this: I have a web API that receives a data transfer object and needs to perform some action, but then it gets murkier and murkier.

Simply code


The first implementation is pretty obvious. I get the DTO, perform checks on it, access the database to get more info, perform more checks, do more work until everything is done. It's easy to do, clear imperative linear programming, it does the job. However I get a bad smell from the code because I have mashed together validation and business logic. I would also like to reuse the validation in other areas of the project.

Manager and Validator


So the natural improvement is to create two flows instead of one, each with their own concern: a validator will perform checks so I know an operation is valid, then the manager performs the operation. But this soon hits a snag. Let say I want to validate that a record exists from the id that I send as a property in the DTO. If I do this check in the validator, then it accesses the database. It stinks! I should probably get the record with the manager, but then I have to validate that the record was returned and that the data in it is valid in the context of my DTO. What do I do?

Manager and Validator classes


OK, so I can split the validation and operational flows in methods that will live in their respective classes. In this scenario a validator checks the record id is not malformed, a manager attempts to get the record, then the validator executes another method to check the record exists and conforms to the expectations, and so on. Problem solved, right? But no. That means I have to pass the record to the validating method, or any number of records that I need to perform the validation. For example I want to check a large chain of records that need to be validated based on conditions in the DTO. I could get the list of all records, pass them to the validator and then check if they exist, but what if the database is very large?

Manager methods that help validation and Validator


Wait, I can be smart about it. I know that essentially I want to get a list of records then validate something. Can't I just create a specific method in the manager that gets only the records involved in validation? Just do manager.GetRelatedRecords(dto.Id) to get a list of records from the database, then pass them as parameters to the validator methods. Fixed it, right? Nope. What if there is a breaking condition that is related to validation? The original linear method would just go through the steps and exit when finished. It wouldn't get a bunch of unnecessary records then fail at the first later on. And there is even a worse problem: what if while I was busy getting the records and preparing for validation the records have changed? I am basically introducing a short lived cache, with all the problems arising from it, like concurrency and invalidation.

Transactions


Hey, I can just new TransactionScope the shit out of it, right? Do steps of Validate, GetMoreData and repeat all inside a transaction. Anything fails, just rollback, what is the big deal? Well, concurrent access being slowed or deadlocks would be problematic, but more than that I am terrified of the result. I started with a linear flow in a single method and now I have two classes, each with a lot of methods which get more and more parameters as I move on: first I validate the DTO values, then a record in relation to the DTO, then two records in relation to each other and the DTO and so on. All these methods are being executed inside a transaction using a local ad-hoc memory cache to hold all the data I presume I will need. The manager methods are either using a big list of parameters themselves or repeatedly instantiating the same providers for the data they get. It's a lot more complex than the original, working design. It's also duplicated flow, as it needs to at least partially go through the data to populate what I need, then it goes through the same flow during the validation.

Context


Ha! I know! Instead of passing an increasing number of parameters to validation methods, create a single context object in which I just populate properties as I go. No more mega methods. Anything can be solved by adding another layer of indirection, correct? Well, now I have three classes, not two, with the context class practically constituting a replacement of the local scope in the original method. The validation methods are now completely obscure unless you look inside them: they all receive the DTO and the context class. Who knows which properties in the context are being used and how?

Interfaces and Flow


Ok, maybe I rushed into things. Sometimes several layers of indirection are required to solve a single problem. In this case I will create an interface with the properties and methods used by every validation method. So the name validation will look like ValidateName(MyDTO dto, INameContext nameContext), the graph validation will look like ValidateGraph(MyDTO dto, IGraphStructure graph) and my context class will implement INameContext and IGraphStructure, even when the two interfaces have common members. I will do the same with the manager class, which will implement the interfaces required to populate the specific context interfaces.

I am not kidding!


This isn't one of those "how the junior and the senior write the same code" jokes. I actually don't know what to do. The most "elegant" solution turned a working method from an API class into three classes, transactions, a dozen interfaces and, best of all, still keeping the entire logical flow in the original method. You know how when a code block becomes too complex it needs to be split into several short methods, right? Well I feel like in this case I somehow managed to split each line in the original block into several methods. I can see the Medium article title now: "If your method has one line of code, it's too long!".

Unit Tests


Any software developer worth his salt will write unit tests, especially for a code such as this. That means any large validation method will need to be split just in order to not increase unit test size exponentially. Can you imagine writing unit tests for the methods that use the huge blob-like context class?

Cry for help


So help me out here, is there an general, elegant but simple solution to API separation of concerns in relation to validation?

Sometimes we want to serialize and deserialize objects that are dynamic in nature, like having different types of content based on a property. However, the strong typed nature of C# and the limitations of serializers make this frustrating.


In this post I will be discussing several things:
  • How to deserialize a JSON property that can have a value of multiple types that are not declared
  • How to serialize and deserialize a property declared as a base type in WCF
  • How to serialize it back using JSON.Net

The code can all be downloaded from GitHub. The first phase of the project can be found here.

Well, the scenario was this: I had a string in the database that was in JSON format. I would deserialize it using JSON.Net and use the resulting object. The object had a property that could be one of several types, but no special notation to describe its concrete type (like the $type notation for JSON.Net or the __type notation for DataContractSerializer). The only way I knew what type it was supposed to be was a type integer value.

My solution was to declare the property as JToken, meaning anything goes there, then deserialize it manually when I have both the JToken value and the integer type value. However, passing the object through a WCF service, which uses DataContractSerializer and which would throw the exception System.Runtime.Serialization.InvalidDataContractException: Type 'Newtonsoft.Json.Linq.JToken' is a recursive collection data contract which is not supported. Consider modifying the definition of collection 'Newtonsoft.Json.Linq.JToken' to remove references to itself..

I thought I could use two properties that pointed at the same data: one for JSON.Net, which would use JToken, and one for WCF, which would use a base type. It should have been easy, but it was not. People have tried this on the Internet and declared it impossible. Well, it's not!

Let's work with some real data. Database JSON:
{
"MyStuff": [
{
"Configuration": {
"Wheels": 2
},
"ConfigurationType": 1
},
{
"Configuration": {
"Legs": 4
},
"ConfigurationType": 2
}
]
}

Here is the code that works:
[DataContract]
[JsonObject(MemberSerialization.OptOut)]
public class Stuff
{
[DataMember]
public List<Thing> MyStuff;
}

[DataContract]
[KnownType(typeof(Vehicle))]
[KnownType(typeof(Animal))]
public class Configuration
{
}

[DataContract]
public class Vehicle : Configuration
{
[DataMember]
public int Wheels;
}

[DataContract]
public class Animal : Configuration
{
[DataMember]
public int Legs;
}

[DataContract]
public class Thing
{
private Configuration _configuration;
private JToken _jtoken;

[DataMember(Name = "ConfigurationType")]
public int ConfigurationType;

[IgnoreDataMember]
public JToken Configuration
{
get
{
return _jtoken;
}
set
{
_jtoken = value;
_configuration = null;
}
}

[JsonIgnore]
[DataMember(Name = "Configuration")]
public Configuration ConfigurationObject
{
get
{
if (_configuration == null)
{
switch (ConfigurationType)
{
case 1: _configuration = Configuration.ToObject<Vehicle>(); break;
case 2: _configuration = Configuration.ToObject<Animal>(); break;
}
}
return _configuration;
}
set
{
_configuration = value;
_jtoken = JRaw.FromObject(value);
}
}
}

public class ThingContractResolver : DefaultContractResolver
{
protected override JsonProperty CreateProperty(System.Reflection.MemberInfo member,
Newtonsoft.Json.MemberSerialization memberSerialization)
{
var prop = base.CreateProperty(member, memberSerialization);
if (member.Name == "Configuration") prop.Ignored = false;
return prop;
}
}

So now this is what happens:
  • DataContractSerializer ignores the JToken property, and doesn't throw an exception
  • Instead it takes ConfigurationObject and serializes/deserializes it as "Configuration", thanks to the KnownTypeAttributes decorating the Configuration class and the DataMemberAttribute that sets the property's serialization name (in WCF only)
  • JSON.Net ignores DataContract as much as possible (JsonObject(MemberSerialization.OptOut)) and both properties, but then un-ignores the Configuration property when we use the ThingContractResolver
  • DataContractSerializer will add a property called __type to the resulting JSON, so that it knows how to deserialize it.
  • In order for this to work, the JSON deserialization of the original text needs to be done with JSON.Net (no __type property) and the JSON coming to the WCF service to be correctly decorated with __type when it comes to the server to be saved in the database

If you need to remove the __type property from the JSON, try the solution here: JSON.NET how to remove nodes. I didn't actually need to do it.

Of course, it would have been simpler to just replace the default serializer of the WCF service to Json.Net, but I didn't want to introduce other problems to an already complex system.


A More General Solution


To summarize, it would have been grand if I could have used IgnoreDataMemberAttribute and JSON.Net would just ignore it. To do that, I could use another ContractResolver:
public class JsonWCFContractResolver : DefaultContractResolver
{
protected override JsonProperty CreateProperty(System.Reflection.MemberInfo member,
Newtonsoft.Json.MemberSerialization memberSerialization)
{
var prop = base.CreateProperty(member, memberSerialization);
var hasIgnoreDataMember = member.IsDefined(typeof(IgnoreDataMemberAttribute), false);
var hasJsonIgnore = member.IsDefined(typeof(JsonIgnoreAttribute), false);
if (hasIgnoreDataMember && !hasJsonIgnore)
{
prop.Ignored = false;
}
return prop;
}
}
This class now automatically un-ignores properties declared with IgnoreDataMember that are also not declared with JsonIgnore. You might want to create your own custom attribute so that you don't modify JSON.Net's default behavior.

Now, while I thought it would be easy to implement a JsonConverter that works just like the default one, only it uses this contract resolver by default, it was actually a pain in the ass, mainly because there is no default JsonConverter. Instead, I declared the default contract resolver in a static constructor like this:
JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
ContractResolver=new JsonWCFContractResolver()
};

Now the JSON.Net operations don't need an explicitly declared contract resolver. Updated source code on GitHub


Links that helped

Configure JSON.NET to ignore DataContract/DataMember attributes
replace WCF built-in JavascriptSerializer with Newtonsoft Json.Net json serializer
Cannot return JToken from WCF Web Service
Serialize and deserialize part of JSON object as string with WCF

and has 0 comments
I am not the manager type: I do software because I like programming. Yet lately I found myself increasingly be the go-to guy for development processes. And I surprisingly knew of what people were asking of me. And it's true, no matter how crappy your jobs are and how little you get to do something truly impressive or challenging, you always gain ... operational experience. I've been working in almost every software environment possible: chaotic, amateurish, corporate, agile, state and everything in between. Therefore I decided to write a blog post about what I thought worked in the development process, rather than talking about code.

First things first


The first thing I need to say is that process, no matter which one you choose, will never work without someone to manage it and perhaps even enforce it. Even a team that agrees unanimously on how they are going to organize the work will slack off eventually. Process is never pleasant to define and follow to the letter, but it is absolutely necessary. In Scrum, for example, they defined the role of Scrum Master exactly for this reason. Even knowing fully that Scrum helps the team, developers and managers alike would cut corners and mess it all up.

The second thing I feel important to put out there before I describe a working process is that doing process for the sake of process is just stupid. The right way to think about it is like playing a game. The best way to play it may be determined by analyzing its rules and planning the actions that will benefit the player most, but no one will ever play the game if it's not fun. Therefore the first thing when determining your process is to make it work for your exact team. The second is to follow the plan as well as possible, because you know and your team knows that is the best way to achieve your goals.

A process


From my personal experience, the best way to work in a team was when we were using Scrum. I have no experience with Extreme Programming or Kanban, though, so I can't tell you which one is "best". As noted above, the best is determined by your team composition, not by objective parameters. For me Scrum was best for two main reasons.

Planning was done within the team, meaning that we both estimated and discussed the issues of the following sprint together. There was no one out of the loop, unless they chose to not pay attention, and it was always clear what we were meant to do. As with every well implemented Scrum process, we had all the development steps for our tasks done during the same sprint: planning, development, unit testing, code review, testing, demo, refactoring and documenting. At the end of the sprint we were confident we did a good job (or aborted the sprint) and that we had delivered something, no matter how trivial, and showed it working. This might not seem so important, but it is, for morale reasons. At the end of the sprint the team was seeing the fruits of their labor. So this is one reason: planned things together, made them work, showed them working and saw the results of our labor and got feedback for it.

The second reason is specific to Scrum because Scrum sprints are timeboxed. You either do what you planned to do in that time or you abort the sprint. This might sound like some pedantic implementation of a process that doesn't take into account the realities of software development, but it's not. Leaving aside the important point that one cannot judge results of work that changed direction since planning, timeboxing makes it difficult and painful to change things from the original planning. That's the main reason for it, in my opinion, and forces the team to work together from beginning to end. The product owner will not laze out and give you a half thought feature to implement because they know it will bite them in the ass if the sprint aborts midway. The developers will not postpone things because they feel they have other things more important to implement. The managers attached to the project will get that nice report they all dream about: starting with clear goals and deadlines and finishing with demonstrable results.

Personally, I also like timeboxing because I get some stuff to do and when I get it done I can do whatever the hell I want. There is no pressure to work withing a fixed daily schedule and encourages business owners to think outside the box, leave people working in their own ways as long as they produce the required results. I liked the philosophy of Scrum so much that I even considered using it for myself, meaning a Scrum of one person, working for my own backlog. It worked for a while quite well, but then I cut corners and slowly abandoned it. As I said before, you need someone to account for the proper implementation of the process. I didn't have it in me to be that person for myself. Not schizoid enough, I guess. Or my other personalities just suck.

Tooling


Process can be improved significantly by tooling. The friction caused by having to follows rules (any rules) can be lessened a lot by computer programs that by definition do just that: follow rules. There is a software developer credo that says "automate everything that you can", meaning you let the computer do as much as possible for you, the rest being what you use your time for. Do the same for your development team. Use the proper tools for the issue at hand.

A short list of tools that I liked working with:
  • Microsoft Visual Studio - great IDE for .NET and web development
  • JetBrains ReSharper - falling in and out of love with it on a regular basis, it feels like magic in the hands of a developer. It also helps with a lot of the administrative part of development like company code practices, code consistency, overall project analysis and refactoring.
  • Atlassian JIRA - "The #1 software development tool used by agile teams". It helps with tasks, management, bugs, reporting. It's web based and blends naturally with collaborative work for teams of any size. It allows for both project management and bug tracking, which is great.
  • Atlassian Confluence - an online document repository. It works like a wiki, with easy to use interface for creating documents and a lot of addons for all kinds of things.
  • Smartbear Code Collaborator - it makes code reviews easy to create, track and integrate.
  • Version control system - this is not a product, it's a class of software. I didn't want to recommend one because people are very attached and vocal about their source control software. I worked with Perforce, SVN, Git, etc. Some are better than others, but it's a long discussion not suitable for this post. Having source control, though, is necessary for any development project.
  • Some chat system - one might think this is trivial, but it's not. People need to be able to communicate instantly without shouting over their desks. Enough said.
  • Jenkins - source automation server. It manages stuff like deployment, but also helps with Continuous Integration and Delivery. I liked it because I worked with it, didn't have to make it work myself. It's written in Java, after all :)
  • Microsoft Exchange - used not only for emails, but also for planning of meetings between people.
  • Notepad - not kidding. With all those wonderful collaborative tools I described above it is easy to forget that individuals too need to plan their work. I found it very helpful to always split my work into little things to do which I write in a text file, then follow the plan. It helps a lot when someone interrupts you from your work and then you need to know where you left off.

This is by no means a comprehensive "all you ever need" list, just some of the tools that I found really useful. However, just paying for them and installing them is not enough. They need to work together. And that's where you absolutely need a role, a person who will take care of all the software, bind them together like the ring of Sauron, and make them seamless for your team which, after all, has people with other stuff to concern themselves with.

My experience


At the beginning at the sprint we would have a discussion on stories that we would like to address in it. Stories are descriptions of features that need to be implemented. They start as business cases "As [some role], I want [something] in order to achieve [some goal]" and together with the product owner (the person in your team that is closest to the client and represents their interests) the whole team discusses overall details and splits it into tasks of various general complexities. About here the team has a rough idea of what will be part of the sprint. Documents are created with the story specifications (using Confluence, for example).

Tasks are technical in nature, so only the technical part of the team, like developers and testers, continue to discuss them. The important part here is to determine the implementation details and estimate complexity in actual time. At the end of this meeting, stories are split into tasks that have people attached to them and are estimated in hours. Now the team can say with some semblance of certainty what tasks can and which cannot be part of the sprint, taking into consideration the skill of the people in the team and their availability in the coming sprint.

Now, with the sprint set, you start work. Developers are writing technical briefs on the desired implementation, eventually the changes, comments, difficulties encountered, etc. (also using Confluence). For each feature brief you will have a technical brief, split into the tasks above. Why waste time with writing documentation while you are developing? Because whenever someone comes and asks what that feature is about, you send them a link. Because whenever you want to move the project to another team, they only have to go to one place and see what has been planned, developed and what is the status of work. The testing team also writes documents on how to proceed with testing. Testing documents will be reviewed by developers and only when that is done, the testing can proceed. Testing documents may seem a bit much, but with this any tester can just do any of the work. With a clear plan, you can take a random person and ask them to test your software. I am not trying to minimize the importance of professional testing skills, but some times you actually want people that are not associated with the development to do the testing. They will note any friction in using your software exactly because they are not close to you and have never used your product before. Also, an important use of testing documents is creating unit tests, which is a developer task.

When a piece of code has been submitted to source control, it must be reviewed (Code Collab) and the deployment framework (Jenkins) will create a task for code review (Code Collab) automate a deploy and run all unit tests and static analysis. If any of them fail, a task will be created in the task manager (Jira). When testing finds bugs, they create items in the bug tracker (Jira). Developers will go over the tasks in Jira and fix what needs fixing. Email will be sent to notify people of new tasks, comments or changes in task status. Meetings to smooth things over and discuss things that are not clear will be created in the meeting software (Exchange), where conflicts will be clear to see and solve. BTW, the room where the meeting is to take place also needs to be tracked, so that meetings don't overlap in time and space.

Wait, you will say, but you are talking more of writing documents and going to meetings and less of software writing. Surely this is a complete waste of time!

No one reasonably expects to work the entire time writing code or testing or whatever their main responsibility is. Some companies assume developers will write code only four hours out of eight. The rest is reserved for planning, meetings, writing documentation and other administrative tasks. Will you write twice as much if you code for eight hours a day? Sure. Will anyone remember what you wrote in two weeks time? No. So the issue is not if you are writing, reading documents and going to meetings instead of coding, but if you are doing all that instead of trying to figure out what the code wanted to do, what it actually does and who the hell wrote that crap (Oh, it was me, shit!). Frankly, in the long run, having a clear picture of what you want and what the entire team did wins big time.

Also please take note on how software integrates with everything to make things better for you. At my current place of work they use the same software, but it is not integrated. People create code reviews if they feel like it, bugs are not related to commits, documentation is vague and out of date, unit tests are run if someone remembers to run them and then no one does because some of them fail and no one remembers why or even wants to take a look. Tooling is nothing if it doesn't help you, if it doesn't work well together. Automate everything!

What I described above worked very well. Were people complaining about too many meetings? Yes. Was the Scrum Master stopping every second discussion in daily meetings because they were meandering out of scope? Sure. Was everybody hating said Scrum Master for interrupting them when saying important stuff then making them listen to all that testing and design crap? Of course. Ignore for a moment you will never get three people together and not have one complain. The complaints were sometimes spot on. As the person responsible for the process you must continuously adapt it to needs. Everybody in an agile team needs to be agile. Sometimes you need to remove or shorten a step, then restore them to their original and then back again. You will never have all needs satisfied at the same time, but you need to take care of the ones that are most important (not necessarily more urgent).

For example, I experienced this process for a web software project that had one hundred thousand customers that created web sites for millions of people that would visit them. There were over seventy people working on it. The product was solid and in constant use for years and years. Having a clear understanding of what is going on in it was very important. Your project might be different. A small tool that you use internally doesn't need as much testing, since bugs will appear with use. It does need documentation, so people know how to use it. A game that you expect people to play en masse for a year then forget about it also has some other requirements.

For me, the question for all cases is the same: "What would you rather be doing?". The process needs to take into account the options and choose the best ones. Would you rather have a shitty product that sells quickly and then is thrown away? Frankly, some people reply wholeheartedly yes, but that's because they would rather work less and earn more. But would you be rather reading code written by someone else and trying to constantly merge your development style with what you think they wanted to do there or reading a document that explains clearly what has been done and why? Would you rather expect people to code review when they feel like it or whenever something is put in source control? Answer this question for all members of the team and you will clearly see what needs to be done. The hard part will be to sell it to your boss, as always.

Daily meeting may not be that important if all the tasks take more than three days, but then you have to ask yourself why do tasks take so long. Could you have split development more finely? Aftermath or postmortem meetings are also essential. In them you discuss what went well and what went wrong in the previous sprint. It gives you the important feedback you need in order to adapt the process to your team and to always, always improve.

Development


I mentioned Continuous Integration above. It's the part that takes every bit of code committed, deploys that changelist, and runs all unit tests to see if anything fails. At the end you either have all green or the person that committed the code is notified that they broke something. Not only it gives more confidence for the code written, but it also promotes unit testing, which in turn promotes a modular way of writing code. When you know you will write unit tests for all written code you will plan that code differently. You will not make methods depend on the current time, you will take all dependencies and plug them in your class without hardcoding them inside. Modularity promotes code that is easy to understand by itself, easy to read, easy to maintain and modify.

Documenting work before it starts makes it easy to find any issues before code writing even began, it makes for easy development of tasks that are implemented one after the other. Reviewing test plans makes developers see their work from a different perspective and have those nice "oh, I didn't think of that" moments.

And code review. If you need to cut as much as possible from process, remove everything else before you abandon code review. Having people read your code and shit all over it may seem cruel and unusual punishment, but it helps a lot. You cannot claim to have easy to understand code if no one reads it. And then there are those moments when you are too tired to write good code. "Ah, it works just the same" you think, while you vomit some substandard code and claim the task completed. Later on, when you are all rested, you will look at the code and wonder "What was I thinking?". If someone else writes something exactly like that you will scold them fiercely for being lazy and not elegant. People don't crap all over you because they hate you, but because they see problems in your code. You will take that constructive criticism and become a better developer, a better team player and a better worker for it.

Conclusions


In this post I tried to show you less how things ought to be, but how things can be. It must be your choice to change things and how to do it. Personally I enjoyed a lot this kind of organization that I felt was freeing me from all the burdens of daily work except the actual software writing that I love doing. From the managerial standpoint it also makes sense to never waste someone's skills for anything else than what they are good at. I was amazed to see how people not only not work like this, but a lot of them don't even knew this was possible and some even outright reject it.

In the way of criticism I've heard a lot of "People before process" and "We want to work here" and "We are here to have fun" and other affirmations that only show how people don't really get it. I've stated many a time above that the process needs to adapt to people; yes, it is very important and it must be enforced, but with the needs of the team put always first. This process speeds up, makes more clear and improves the work, it doesn't stop or delay it. As for fun, working with clear goals in mind, knowing you have the approval of everyone in the team and most wrinkles in the design have been smoothed out before you even began writing code is nothing but fun. It's like a beautiful woman with an instruction manual!

However, just because process can help you, doesn't mean it actually does. It must be implemented properly. It is extremely easy to fall into the "Process over people" trap, for example, and then the criticism above is not only right, but pertinent. You need to do something about it. It doesn't help that process improves productivity ten fold if all the people in the team are malcontent and work a hundred times slower just because you give them no alternative.

Another pitfall is to conceive the most beautiful process in the world, make all agile teams be productive and happy, then fail to synchronize them. In other words you will have several teams that all work as one, but the group of teams is itself not a team. That's why we had something called the Scrum of Scrums, when representatives from each team would meet and discuss goals and results.

Now ask yourself, what would you rather be doing?

I am mentally preparing for giving a talk about dependency injection and inversion of control and how are they important, so I intend to clarify my thoughts on the blog first. This has been spurred by seeing how so many talented and even experienced programmers don't really understand the concepts and why they should use them. I also intend to briefly explore these concepts in the context of programming languages other than C#.

And yes, I know I've started an ASP.Net MVC exploration series and stopped midway, and I truly intend to continue it, it's just that this is more urgent.

Head on intro


So, instead of going to the definitions, let me give you some examples, instead.
public class MyClass {
public IEnumerable<string> GetData() {
var provider=new StringDataProvider();
var data=provider.GetStringsNewerThan(DateTime.Now-TimeSpan.FromHours(1));
return data;
}
}
In this piece of code I create a class that has a method that gets some text. That's why I use a StringDataProvider, because I want to be provided with string data. I named my class so that it describes as best as possible what it intends to do, yet that descriptiveness is getting lost up the chain when my method is called just GetData. It is called so because it is the data that I need in the context of MyClass, which may not care, for example, that it is in string format. Maybe MyClass just displays enumerations of objects. Another issue with this is that it hides the date and time parameter that I pass in the method. I am getting string data, but not all of it, just for the last hour. Functionally, this will work fine: task complete, you can move to the next. Yet it has some nagging issues.

Dependency Injection


Let me show you the same piece of code, written with dependency injection in mind:
public class MyClass {
private IDataProvider _dataProvider;
private IDateTimeProvider _dateTimeProvider;

public void MyClass(IDataProvider dataProvider, IDateTimeProvider dateTimeProvider) {
this._dataProvider=dataProvider;
this._dateTimeProvider=dateTimeProvider;
}

public IEnumerable<string> GetData() {
var oneHourBefore=_dateTimeProvider.Now-TimeSpan.FromHours(1);
var data=_dataProvider.GetDataNewerThan(oneHourBefore);
return data;
}
}
A lot more code, but it solves several issues while introducing so many benefits that I wonder why people don't code like this from the get go.

Let's analyse this for a bit. First of all I introduce a constructor to MyClass, one that accepts and caches two parameters. They are not class types, but interfaces, which declare the intention for any class implementing them. The method then does the same thing as in the original example, using the providers it cached. Now, when I write the code of the class I don't actually need to have any provider implementation. I just declare what I need and worry about it later. I also don't need to inject real providers, I can mock them so that I can test my class as standalone. Note that the previous implementation of the class would have returned different data based on the system time and I had no way to control that behavior. The best benefit, for me, is that now the class is really descriptive. It almost reads like English: "Hi, folks, I am a class that needs someone to give me some data and the time of day and I will give you some processed data in return!". The rule of thumb is that for each method, external factors that may influence its behavior must be abstracted away. In our case if the date time provider provides the same time and the data provider the same data, the effect of the method is always the same.

Note that the interface I used was not IStringDataProvider, but IDataProvider. I don't really care, in my class, that the data is a bunch of strings. There is something called the Single Responsibility Principle, which says that a class or a method or some sort of unit of computation should try to only have one responsibility. If you change that code, it should only affect one area. Now, real life is a little different and classes do many things in many directions, yet they can implement any number of interfaces. The interfaces themselves can declare only one responsibility, which is why this is so nice. I don't actually have to have a class that is only a data provider, but in the context of my class, I only need that part and I am clearly declaring my intent in the code.

This here is called dependency injection, which is a fancy expression for saying "my code receives all third party instances as parameters". It is also in line with the Single Responsibility Principle, as now your class doesn't have to carry the responsibility of knowing how to instantiate the classes it needs. It makes the code more modular, easier to test, more legible and more maintainable.

But there is a problem. While before I was using something like new MyClass().GetData(), now I have to push the instantiation of the providers somewhere up the stream and do maybe something like this:
var dataProvider=new StringDataProvider();
var dateTimeProvider=new DateTimeProvider();
var myClass=new MyClass(dataProvider,dateTimeProvider);
myClass.GetData();
The apparent gains were all for naught! I just pushed the same ugly code somewhere else. But here is where Inversion of Control comes in. What if you never need to instantiate anything again? What it you never actually had to write any new Something() code?

Inversion of Control


Inversion of Control actually takes over the responsibility of creating instances from you. With it, you might get this code instead:
public interface IMyClass {
IEnumerable<string> GetData();
}

public class MyClass:IMyClass {
private IDataProvider _dataProvider;
private IDateTimeProvider _dateTimeProvider;

public void MyClass(IDataProvider dataProvider, IDateTimeProvider dateTimeProvider) {
this._dataProvider=dataProvider;
this._dateTimeProvider=dateTimeProvider;
}

public IEnumerable<string> GetData() {
var oneHourBefore=_dateTimeProvider.Now-TimeSpan.FromHours(1);
var data=_dataProvider.GetDataNewerThan(oneHourBefore);
return data;
}
}
Note that I created an interface for MyClass to implement, one that declares my GetData method. Now, to use it, I could write something like this:
var myClass=Dependency.Get<IMyClass>();
myClass.GetData();

Wow! What happened here? I just used a magical class called Dependency that gets me an instance of IMyClass. And I really don't care how it does it. It can discover implementations by itself or maybe I am manually binding interfaces to implementations when the application starts (for example Dependency.Bind<IMyClass,MyClass>();). When it needs to create a new MyClass it automatically sees that it needs two other interfaces as parameters, so it gets implementations for those first and continues up the chain. It is called a dependency chain and the container will go through it all to simply "Get" you what you need. There are many inversion of control frameworks out there, but the concept is so simple that one can make their own easily.

And I get another benefit: if I want to display some other type of data, all I have to do is instruct the dependency container that I want another implementation for the interface. I can even think about versioning: take a class that I know does the job and compare it with a new implementation of the same interface. I can tell it to use different versions based on the client used. And all of this in exactly one place: the dependency container bindings. You may want to plug different implementations provided by third parties and all they have to care about is respecting the contract in your interface.



Solution structure


This way of writing code forces some changes in the structure of your projects. If all you have is written in a single project, you don't care, but if you want to split your work in several libraries, you have to take into account that interfaces need to be referenced by almost everything, including third party modules that you want to plug. That means the interfaces need their own library. Yet in order to declare the interfaces, you need access to all the data objects that their members need, so your Interfaces project needs to reference all the projects with data objects in them. And that means that your logic will be separated from your data objects in order to avoid circular dependencies. The only project that will probably need to go deeper will be the unit and integration test project.

Bottom line: in order to implement this painlessly, you need an Entities library, containing data objects, then an Interfaces library, containing the interfaces you need and, maybe, the dependency container mechanism, if you don't put it in yet another library. All the logic needs to be in other projects. And that brings us to a nice side effect: the only connection between logic modules is done via abstractions like interfaces and simple data containers. You can now substitute one library with another without actually caring about the rest. The unit tests will work just the same, the application will function just the same and functionality can be both encapsulated and programatically described.

There is a drawback to this. Whenever you need to see how some method is implemented and you navigate to definition, you will often reach the interface declaration, which tells you nothing. You then need to find classes that implement the interface or to search for uses of the interface method to find implementations. Even so, I would say that this is an IDE problem, not a dependency injection issue.

Other points of view


Now, the intro above describes what I understand by dependency injection and inversion of control. The official definition of Dependency Injection claims it is a subset of Inversion of Control, not a separate thing.

For example, Martin Fowler says that when he and his fellow software pattern creators thought of it, they called it Inversion of Control, but they decided that it was too broad a term, so they moved to calling it Dependency Injection. That seems strange to me, since I can describe situations where dependencies are injected, or at least passed around, but they are manually instantiated, or situations where the creation of instances is out of the control of the developer, but no dependencies are passed around. He seems to see both as one thing. On the other hand, the pattern where dependencies are injected by constructor, property setters or weird implementation of yet another set of interfaces (which he calls Dependency Injection) is different from Service Locator, where you specifically ask for a type of service.

Wikipedia says that Dependency Injection is a software pattern which implements Inversion of Control to resolve dependencies, while it calls Inversion of Control a design principle (so, not a pattern?) in which custom-written portions of a computer program receive the flow of control from a generic framework. It even goes so far as to say Dependency Injection is a specific type of Inversion of Control. Anyway, the pages there seem to follow the general definitions that Martin Fowler does, which pits Dependency Injection versus Service Locator.

On StackOverflow a very well viewed answer sees dependency injection as "giving an object its instance variables". I tend to agree. I also liked another answer below that said "DI is very much like the classic avoiding of hardcoded constants in the code." It makes one think of a variable as an abstraction for values of a certain type. Same page holds another interesting view: "Dependency Injection and dependency Injection Containers are different things: Dependency Injection is a method for writing better code, a DI Container is a tool to help injecting dependencies. You don't need a container to do dependency injection. However a container can help you."

Another StackOverflow question has tons of answers explaining how Dependency Injection is a particular case of Inversion of Control. They all seem to have read Fowler before answering, though.

A CodeProject article explains how Dependency Injection is just a flavor of Inversion of Control, others being Service Locator, Events, Delegates, etc.

Composition over inheritance, convention over configuration


An interesting side effect of this drastic decoupling of code is that it promotes composition over inheritance. Let's face it: inheritance was supposed to solve all of humanity's problems and it failed. You either have an endless chain of classes inheriting from each other from which you usually use only one or two or you get misguided attempts to allow inheritance from multiple sources which complicates understanding of what does what. Instead interfaces have become more widespread, as declarations of intent, while composition has provided more of what inheritance started off as promising. And what is dependency injection if not a sort of composition? In the intro example we compose a date time provider and a data provider into a time aware data provider, all the time while the actors in this composition need to know nothing else than the contracts each part must abide by. Do that same thing with other implementations and you get a different result. I will go as far as to say that inheritance defines what classes are, while composition defines what classes do, which is what matters in the end.

Another interesting effect is the wider adoption of convention over configuration. For example you can find the default implementation of an interface as the class that implements it and has the same name minus the preceding "I". Rather than explicitly tell the framework that we want to use the Manager class each time someone needs an IManager implementation, it can figure it out for itself by naming alone. This would never work if the responsibility of getting class instances resided with each method using them.

Real life examples


Simple Injector


If you look on the Internet, one of the first dependency injection frameworks you find for .Net is Simple Injector, which works on every flavor of .Net including Mono and Core. It's as easy to use as installing the NuGet package and doing something like this:
// 1. Create a new Simple Injector container
var container = new Container();

// 2. Configure the container (register)
container.Register<IUserRepository, SqlUserRepository>(Lifestyle.Transient);
container.Register<ILogger, MailLogger>(Lifestyle.Singleton);

// 3. Optionally verify the container's configuration.
container.Verify();

// 4. Get the implementation by type
IUserService service = container.GetInstance<IUserService>();

ASP.Net Core


ASP.Net Core has dependency injection built in. You configure your bindings in ConfigureServices:
public void ConfigureServices(IServiceCollection svcs)
{
svcs.AddSingleton(_config);

if (_env.IsDevelopment())
{
svcs.AddTransient<IMailService, LoggingMailService>();
}
else
{
svcs.AddTransient<IMailService, MailService>();
}

svcs.AddDbContext<WilderContext>(ServiceLifetime.Scoped);

// ...
}
then you use any of the registered classes and interfaces as constructor parameters for controllers or even using them as method parameters (see FromServicesAttribute)

Managed Extensibility Framework


MEF is a big beast of a framework, but it can simplify a lot of work you would have to do to glue things together, especially in extensibility scenarios. Typically one would use attributes to declare which interface something "exports" and then use other attributes to "import" implementations in properties and values. All you need to do is put them in the same place. Something like this:
[Export(typeof(ICalculator))]
class SimpleCalculator : ICalculator {
//...
}

class Program {

[Import(typeof(ICalculator))]
public ICalculator calculator;

// do something with calculator
}
Of course, in order for this to work seamlessly you need stuff like this, as well:
private Program()
{
//An aggregate catalog that combines multiple catalogs
var catalog = new AggregateCatalog();
//Adds all the parts found in the same assembly as the Program class
catalog.Catalogs.Add(new AssemblyCatalog(typeof(Program).Assembly));
catalog.Catalogs.Add(new DirectoryCatalog("C:\\Users\\SomeUser\\Documents\\Visual Studio 2010\\Projects\\SimpleCalculator3\\SimpleCalculator3\\Extensions"));


//Create the CompositionContainer with the parts in the catalog
_container = new CompositionContainer(catalog);

//Fill the imports of this object
try
{
this._container.ComposeParts(this);
}
catch (CompositionException compositionException)
{
Console.WriteLine(compositionException.ToString());
}
}

Dependency Injection in other languages


Admit it, C# is great, but it is not by far the most used computer language. That place is reserved, at least for now, for Javascript. Not only is it untyped and dynamic, but Javascript isn't even a class inheritance language. It uses the so called prototype inheritance, which uses an instance of an object attached to a type to provide default values for the instance of said type. I know, it sounds confusing and it is, but what is important is that it has no concept of interfaces or reflection. So while it is trivial to create a dictionary of instances (or functions that create instances) of objects which you could then use to get what you need by using a string key (something like var manager=Dependency.Get('IManager');, for example) it is difficult to imagine how one could go through the entire chain of dependencies to create objects that need other objects.

And yet this is done, by AngularJs, RequireJs or any number of modern Javascript frameworks. The secret? Using regular expressions to determine the parameters needed for a constructor function after turning it to string. It's complicated and beyond the scope of this blog post, but take a look at this StackOverflow question and its answers to understand how it's done.

Let me show you an example from AngularJs:
angular.module('myModule', [])
.directive('directiveName', ['depService', function(depService) {
// ...
}])
In this case the key/type of the service is explicit using an array notation that says "this is the list of parameters that the dependency injector needs to give to the function", but this might be have been written just as the function:
angular.module('myModule', [])
.directive('directiveName', function(depService) {
// ...
})
In this case Angular would use the regular expression approach on the function string.


What about other languages? Java is very much like C# and the concepts there are similar. Even if all are flavors of C, C++ is very different, yet Dependency Injection can be achieved. I am not a C++ developer, so I can't tell you much about that, but take a look at this StackOverflow question and answers; it is claimed that there is no one method, but many that can be used to do dependency injection in C++.

In fact, the only languages I can think of that can't do dependency injection are silly ones like SQL. Since you cannot (reasonably) define your own types or pass functions along, the concept makes no sense. Even so, one can imagine creating dummy stored procedures that other stored procedures would use in order to be tested. There is no reason why you wouldn't use dependency injection if the language allows for it.

Testability


I mentioned briefly unit testing. Dependency Injection works hand in hand with automated testing. Given that the practice creates modules of software that give reproducible results for the same inputs and account for all the inputs, testing becomes a breeze. Let me give you some examples using Moq, a mocking library for .Net:
var dateTimeMock=new Mock<IDateTimeProvider>();
dateTimeMock
.Setup(m=>m.Now)
.Returns(new DateTime(2016,12,03));

var dataMock=new Mock<IDataProvider>();
dataMock
.Setup(m=>m.GetDataNewerThan(It.IsAny<DateTime>()))
.Returns(new[] { "test","data" });

var testClass=new MyClass(dateTimeMock.Object, dataMock.Object);

var result=testClass.GetData();
AssertDeepEqual(result,new[] { "test","data" });

First of all, I take care of all dependencies. I create a "mock" for each of them and I "set up" the methods or property setters/getters that interest me. I don't really need to set up the date time mock for Now, since the data from the data provider is always the same no matter the parameter, but it's there for you to see how it's done. Second, I instantiate the class I want to test using the Object property of my mocks, which returns an object that implements the type given as a generic parameter in the constructor. Third I assert that the side effects of my call are the ones I expect. The mocks need to be as dumb as possible. If you feel you need to write code to define your mocks you are probably doing something wrong.

The type of the tests, for people who are not familiar with this concept, is usually a fully positive one - that is give full valid data and expect the correct result - followed by many negative ones, where the correct data is made incorrect in all possible ways and it is tested that the method fails. If there are many types of combinations of data that would be considered valid, you need a test for as many of them.

Note that the test is instantiating the test class directly, using the constructor. We are not testing the injector here, but the actual class.

Conclusions


What I appreciate most with Dependency Injection is that it forces you to write code that has clear boundaries defined by interfaces. Once this is achieved, you can go write your own stuff and not care about what other people do with theirs. You can test your modules without even caring if the rest of the project even exists. It allows to refactor code in steps and with a lot more confidence since you are covered by unit tests.

While some people work on fire-and-forget projects, like small games or utilities, and they don't care about maintainability, one of the most touted reasons for using unit tests and dependency injection, these practices bring so many other benefits that are almost impossible to get otherwise.

The entire point of this is reducing the complexity of dependencies, which include not only the modules in your application, but also the support frame for them, like people working on them. While some managers might not see the wisdom of reducing friction between software components, surely they can see the positive value of reducing friction between people.

There was one other topic that I wanted to touch, but it is both vast and I have not enough experience with it, however it feels very attractive to me: refactoring old code in order to use dependency injection. Best practices, how to make it safe enough and fast enough to make managers approve it and so on. Perhaps another post later on. I was thinking of a combination of static analysis and automated methods, like replacing all usages of "new" with a single point of instantiation, warning about static methods and properties, automatically replacing known bad practices like DateTime.Now and so on. It might be interesting, right?

I hope I wasn't too confusing and I appreciate any feedback you have. I will be working on a presentation file with similar content, so any help will go into doing a better job explaining it to others.