# Interview question: generate all valid combinations of n pairs of parentheses

## Definition

So, the task at hand is the subject of a common interview question: Implement an algorithm to get all valid (opened and closed) combinations of n pairs of parentheses. This means that for n=1 there is only one solution: "()". "((" or "))" are not valid, for 2 you will have "(())" and "()()" and so on. The question is trying to test how the interviewee handles recursion and what is commonly called backtracking. But as usual, there's more than one way to skin a cat, although for the life of me I can't see why you would want to do that.

The solutions here will be in C# and the expected result is an enumeration of strings containing open and closed parentheses. The code can be easily translated into other languages, including Javascript (ECMAScript 2015 introduced iterators and generator functions), but that's left to the reader. Let's begin.

## Analysis

Before we solve any problem we need to analyse it and see what are the constraints and the expected results. In this case there are several observations that can be made:

• the resulting strings will be of length n*2 (n pairs)
• they will contain n '(' characters and n ')' characters
• they cannot start with a ')' or end in a '('
• in order to generate such a string, we can start with a smaller string to which we add '(' or ')'
• we cannot add a ')' if there isn't at least one corresponding unclosed '('
• if we add a '(' we need to have enough characters left to close the parenthesis, so the number of unclosed parentheses cannot exceed the characters left to fill
• we could count the open and closed parentheses, but we only care about the number of unclosed ones, so instead of "closed" and "open" values, we can only use "open" to represent unclosed parentheses

Let's go for some variables and calculations:

• n = number of pairs
• open = number of unclosed parentheses in a string
• open cannot be negative
• one cannot add ')' if open = 0
• one cannot add '(' if open >= n*2 - substring.length

## Recursive solution

A simple implementation of these requirements can done with recursion:

``````public IEnumerable<string> GenerateRecursive(int n, string root = "", int open = 0)
{
// substring is long enough, return it and exit
if (root.Length == n * 2)
{
yield return root;
yield break;
}
// if we can add '(' to existing substring, continue the process with the result
if (open < n * 2 - root.Length)
{
// if only C# could just 'yield IteratorFunction()' this would look sleeker
foreach (var s in GenerateRecursive(n, root + "(", open + 1))
{
yield return s;
}
}
// if we can add ')' to existing substring, continue the process with the result
if (open > 0)
{
foreach (var s in GenerateRecursive(n, root + ")", open - 1))
{
yield return s;
}
}
}
``````

However, every time you see recursion you have to ask yourself: could n be large enough to cause a stack overflow? For example this fails for n=3000. The nice thing about this method, though, is that it can be limited to the number of items you want to see. For example `var firstTen = GenerateRecursive(1000).Take(10)` is very fast, as the generation is depth first and only computes the first ten values and exits.

So, can we replace the recursion with iteration?

## Iterative solution

In order to do thing iteratively, we need to store the results of the previous step and use them in the current step. This means breadth first generation, which has its own problems. Let's see some code:

``````public IEnumerable<string> GenerateIteration(int n)
{
// using named tuples to store the unclosed parentheses count with the substring
var results = new List<(string Value,int Open)>() { ("",0) };
for (int i = 0; i < n*2; i++)
{
// each step we compute the list of new strings from the list in the previous step
var newResults = new List<(string Value, int Open)>();
foreach (var (Value, Open) in results)
{
if (Open < n * 2 - Value.Length)
{
newResults.Add((Value + "(", Open + 1));
}
if (Open > 0)
{
newResults.Add((Value + ")", Open - 1));
}
}
results = newResults;
}
return results.Select(r=>r.Value);
}``````

It's pretty sleek, but if you try something like `var firstTen = GenerateRecursive(1000).Take(10)` now it will take forever since all combinations of 1000 parentheses need to be computed and stored before taking the first 10! BTW, we can write this much nicer with LINQ, but be careful at the gotcha in the comment:

``````public IEnumerable<string> GenerateLinq(int n)
{
// looks much nicer with LINQ
IEnumerable<(string Value, int Open)> results = new[] { ("", 0) };
for (var i = 0; i < n * 2; i++)
{
results =
results
.Where(r => r.Open < n * 2 - r.Value.Length)
.Select(r => (Value: r.Value + "(", Open: r.Open + 1))
.Concat(results
.Where(r => r.Open > 0)
.Select(r => (Value: r.Value + ")", Open: r.Open - 1))
);  // but if you do not end this with a .ToList()
// it will generate a huge expression that then will be evaluated at runtime! Oops!
}
return results.Select(r => r.Value);
}``````

But can't we do better? One is going to stack overflow, the other memory overflow and the last one kind of does both.

## Incremental solution

They did say this requires an incremental solution, right? So why don't we take this literally? '(' and ')' are like 0 and 1, as ')' must always follow a '('. If you view a parenthesis string as a binary number, then all possible combinations can be encoded as numbers. This means that we could conceivably write a very fast function that would compute all possible combinations using bit operations, maybe even special processor instructions that count bits and so on. However, this would work only for n<=32 or 64 depending on the processor architecture and we don't want to get into that. But we can still use the concept!

If a string represents a fictional number, then you can start with the smallest one, increment it and check for validity. If you combine the incremental operation with the validity check you don't need to go through 2n operations to get the result. It doesn't use any memory except the current string and it is depth first generation. The best of both worlds! Let's see some code:

``````public IEnumerable<string> GenerateIncrement(int n)
{
// the starting point is n open parentheses and n closing ones
// we use the same array of characters to generate the strings we display
var arr = (new string('(', n) + new string(')', n)).ToCharArray();
// iteration will stop when incrementation reaches the "maximum" valid combination
var success = true;
while (success)
{
yield return new string(arr);
success = Increment(arr, n);
}
}

private bool Increment(char[] arr, int n)
{
// we begin with a valid string, which means there are no unclosed parentheses
var open = 0;
// we start from the end of the string
for (var i = arr.Length - 1; i >= 0; i--)
{
// ')' is equivalent to a 1. To "increment" this string we need to go to the previous position
// incrementing 01 in binary results in 10
if (arr[i] == ')')
{
open++;
continue;
}

// '(' is equivalent to a 0. We will turn it into a ')' to increment it,
// but only if there are unclosed parentheses to close
open--;
if (open == 0) continue;

// we 'increment' the value
arr[i] = ')';
// now we need to reset the rest of the array
var k = n - (open + i) / 2;
// as many opening parenthesis as possible
for (var j = i + 1; j < i + 1 + k; j++)
{
arr[j] = '(';
}
// the rest are closing parentheses
for (var j = i + 1 + k; j < n * 2; j++)
{
arr[j] = ')';
}
return true;
}
// if we reached this point it means we got to a maximum
return false;
}``````

Now doing `GenerateIncrement(1000000).Take(10)` took more to display the results than to actually compute them.

## More solutions

As this is a classic interview question, there are a billion solutions to it at LeetCode. Yet the purpose of interview questions is to find out how one thinks, not what the solution of the problem actually is. I hope this helps.

# Is lazy dependency injection a good idea?

## Intro

When talking Dependency Injection, if a class implementing Interface1 needs an implementation of Interface2 in its constructor and the implementation for Interface2 needs an implementation of Interface1 you get a circular dependency error. This could be fixed, though, by providing lazy proxy implementations, which would also fix issues with resources getting allocated too early and other similar issues.  Now, theoretically this is horrible. Yet in practice one meets this situation a lot. This post will attempt to clarify why this happens and how practice may be different from theory.

## Problem definition

Let's start with defining what an interface is. Wikipedia says it's a shared boundary between components. In the context of dependency injection you often hear about the Single Responsibility Principle, which stipulates that a class (and by extension an interface) should only do one thing. Yet even in this case, the implementation for any of the Facade, Bridge, Decorator, Proxy and Adapter software patterns would do only one thing: to proxy, merge or split the functionality of other components, regardless of how many and how complex they are. Going to the other extreme, one could create an interface for every conceivable method, thus eliminating the need for circular dependencies and also loading code that is not yet needed. And then there are the humans writing the code. When you need a service to provide the physical location of the application you would call it ILocationService and when you want to compute the distance between two places you would use the same, because it's about locations, right? Having an ILocationProviderService and an ILocationDistanceCalculator feels like overkill. Imagine trying to determine if a functionality regarding locations is already implemented and going through all the ILocation... interfaces to find out, then having to create a new interface when you write the code for it and spending sleepless nights wondering if you named things right (and if you need to invalidate their cache).

In other words, depending on context, an interface can be anything, as arbitrarily complex as the components it separates. They could contain methods that are required by other components together with methods that require other components. If you have more such interfaces, you might end up with a circular dependency in the instantiation phase even if the execution flow would not have this problem. Let's take a silly example.

We have a LocationService and a TimeService. One handles points in space the other moments in time. And let's say we have the entire history of someone's movements. You could get a location based on the time provided (GetLocation) or get the time based on a provided location (GetTime). Now, the input from the user is text, so we need the LocationService and the TimeService to translate that text into actual points in space and moments in time, so GetLocation would use an ITimeService, while GetTime would use an ILocationService. You start the program and you get the circular dependency error. I told you it would be silly. Anyway, you can split any of the services into ITimeParser and ITimeManager or whatever, you can create a new interface called ITextParser, there are a myriad refactoring solutions. But what if you don't have the luxury to refactor and why do you even need to do anything? Surely if you call GetLocation you only need to parse the time, you never call GetTime, and the other way around.

## Solution?

A possible solution is to only actually get the dependency implementation when you use it. Instead of providing the actual implementation for the interface you need, you provide a lazy proxy. Here is an example of a generic (and lazy one liner) LazyProxy implementation:

``````public class LazyProxy<TInterface>:Lazy<TInterface>
{
public LazyProxy(IServiceProvider serviceProvider) : base(() => serviceProvider.GetService<TInterface>()) { }
}``````

Problem solved, right? LocationService would ask for a `LazyProxy<ITimeService>` implementation, GetLocation would do `_lazyTimeService.Value.ParseTime(input)` which would instantiate a TimeService for the first time, which would ask for a `LazyProxy<ILocationService>` and in GetTime it would use `_lazyLocationService.Value.ParseLocation(input)` which would get the existing instance of LocationService (if it's registered as Singleton). Imagine either of these services would have needed a lot of other dependencies.

Now, that's what called a "leaky abstraction". You are hiding the complexity of instantiating and caching a service (and all of its dependencies) until you actually use it. Then you might get an error, when the actual shit hits the actual fan. I do believe that the term "leaky" might have originated from the aforementioned idiom. Yuck, right? It's when the abstraction leaked the complexity that lies beneath.

There are a number of reasons why you shouldn't do it. Let's get through them.

## Criticism

The most obvious one is that you could do better. The design in the simple and at the same time contrived example above is flawed because each of the services are doing two very separate things: providing a value based on a parameter and interpreting text input. If parsing is a necessary functionality of your application, then why not design an ITextParser interface that both services would use? And if your case is that sometimes you instantiate a class to use one set of functions and sometimes to use another set of functions, maybe you should split that up into two. However, in real life situations you might not have full control over the code, you might not have the resources to refactor the code. Hell, you might be neck deep in spaghetti code! Have you ever worked in one of those house of cards companies where you are not allowed to touch any piece of code for fear it would all crash?

The next issue is that you would push the detection for a possible bug to a particular point of the execution of your code. You would generate a Heisenbug, a bug that gets reproduced inconsistently. How appropriate this would have been if an IMomentumService were used as well. Developers love Heisenbugs, as the time for their resolution can vary wildly and they would be forced to actually use what they code. Oh, the humanity! Yet, the only problem you would detect early is the cycle in the dependency graph, which is more of a design issue anyway. A bug in the implementation would still be detected when you try to use it.

One other issue that this pattern would solve should not be there in the first place: heavy resource use in constructors. Constructors should only construct, obviously, leaving other mechanisms to handle the use of external resources. But here is the snag: if you buy into this requirement for constructors you already use leaky abstractions. And again, you might not be able to change the constructors.

Consider, though, the way this pattern works. It is based on the fact that no matter when you request the instantiation of a class, you would have a ready implementation of IServiceProvider. The fact that the service locator mechanism exists is already lazy instantiation on the GetService method. In fact, this lazy injection pattern is itself a constructor dependency injection abstraction of the service provider pattern. You could just as well do `var timeService = _serviceProvider.GetService<ITimeService>()` inside your GetLocation method and it would do the exact same thing. So this is another reason why you should not do it: mixing the metaphors. But hey! If you have read this far, you know that I love mixing those suckers up!

## Conclusion

In conclusion, I cannot recommend this solution if you have others like refactoring available. But in a pinch it might work. Let me know what you think!

BTW, this issue has been also discussed on Stack Overflow, where there are some interesting answers.

# Using Kendo UI for jQuery MultiSelect with dynamic values

MultiSelect is a Kendo UI control that transforms a select element into a nice dropdown with text filtering which allows the selection of multiple items. This is how you use the same control to write values directly in the list, something akin to the Outlook address bar functionality.

Long story short: the control exposes some events like: 'filtering','open','close' and 'change'. In the filtering event, which is fired by someone writing or pasting text in order to filter the list of items, we dynamically create a list item that holds that value, so that the user can just press Enter and enter the value in the list. The code also allows for a custom transformation function, so for example someone could enter "1,2,3" and it would be translated into three values 1, 2 and 3 instead of an item with the value "1,2,3". On the close and change events we clear the items in the list that have not been selected. This means you cannot use this code as is to show an autocomplete list and also add dynamic values, but it is easy to tweak for that purpose.

In order to use it, instead of doing `\$(selector).kendoMultiSelect(options)`, just use `\$(selector).kendoDynamicMultiSelect(options)`. Here is the code:

``````\$.fn.kendoDynamicMultiSelect = function (options) {
var multiSelect = \$(this).kendoMultiSelect(options).getKendoMultiSelect();

multiSelect.bind('filtering', function (ev) {
var val = ev.filter && ev.filter.value;
if (!val) return;

var dataSource = ev.sender.dataSource;
var items = dataSource.data();

// if there is an existing item in the list, don't create a new one
var existingItem = items.filter(function (i) {
return i.value == val;
})[0];
if (existingItem) return;

// find or create the item that will hold the current filter value
var inputItem = items.filter(function (i) {
return i.isInput;
})[0];
if (!inputItem) {
inputItem = dataSource.insert(0, { isInput: true });
// when inserting a value the input gets cleared in some situations
// so set it back
ev.sender.input.val(ev.filter.value);
}
inputItem.value = val;
});

// cleans input items and also applies an optional value transformation function
var updateValues = function (ev) {
var values = ev.sender.value();
if (typeof options.valueTransformationFunction === 'function') {
// for example split comma separated values
values = options.valueTransformationFunction(values);
}

var dataSource = ev.sender.dataSource;
var items = dataSource.data();
for (var i = 0; i < items.length; i++) {
var item = items[i];
item.shouldBeKept = false;
}

// add items for existing values
for (var i = 0; i < values.length; i++) {
var value = values[i];

var item = items.filter(function (i) { return i.value == value; })[0];
if (!item) {
item = dataSource.add({ value: value });
}
item.isInput = false;
item.shouldBeKept = true;
}

ev.sender.value(values);

// delete all others
for (var i = 0; i < items.length; i++) {
var item = items[i];
if (!item.shouldBeKept) {
dataSource.remove(item);
}
}
};

multiSelect.bind('change', updateValues);
multiSelect.bind('close', updateValues);
};
``````

I kind of copied this code by hand and tried it on another computer. If you find any bugs, let me know. Also, I know this is old time tech, but they use it in my company and I couldn't find this functionality by googling it, so here it is.

I hope it helps.

# Optimizing an SQL Server query

OK, so I played a little with SQL and I found an interesting flow for analysing queries. It uses the SET STATISTICS PROFILE functionality, but the results of this are usually hard to read and handle in any meaningful way. There are applications that help out with this, but this blog post is trying to show you a method that doesn't need any extra software (for when you are working for a paranoid company that doesn't allow you to install what you need to do your work, for example).

This works in the query itself, so no need of any extra tool except SQL Server Management Studio and Excel:

1. Add SET STATISTICS PROFILE OFF at the start of the query (because you don’t need to profile the setup)
2. Add SET STATISTICS PROFILE ON just before the SELECT that you want to optimize
3. Clear cache and stats - this is optional, but good practice. There are multiple ways of doing this and it depends on your environment and preferences, so I am not covering this here.
4. Execute the query -> In the query results you will get the results of the query, but also the profiling statistics of the query execution, also in table form
5. Copy the entire statistics table with headers and insert it into a new Excel sheet
6. Add a new column right after Parent, call it IsLeaf
7. Fill the IsLeaf column with a formula to see if the value in NodeId exists in the Parent column
1. Write "=COUNTIF(\$F\$2:\$F\$10000,E2)=0" as the first value of the column
2. Keep pressing Shift, then press End and Down arrow (and release Shift) – you should have the entire column selected
3. Press Ctrl-D
8. Select the header row of the table
9. Click on "Sort and Filter"
10. Select "Filter"
11. Click on a random cell, click on "Sort and Filter" again
12. Click on "Custom sort"
13. Select TotalTreeSubcost and "From largest to smallest"
14. Now click on the filter on the IsLeaf column and filter on value TRUE (only the leaves)

You should now have the rows of the final tree branch nodes, ordered descending by the cost to the query.

Here you can look at the IO cost, CPU cost and Rows columns to find the places you need to work on. These values need to be as small as possible.

I hope this helps.

# SQL: using IN or a bunch of ORs ?

Update: more analysis shows that the change was not because of the OR clauses, but because of some debug OPTIONs that I had used. This post is thus wrong.

Original post:

So I had this stored procedure that would calculate counts from a table, based on a specific column which was also indexed. Something like this:

``````SELECT Code, COUNT(*) as Nr
FROM MyTable``````

The code would take the result of this query and only use the counts for some of the codes, let's say 'A', 'B' and 'C'. There was also a large number of instructions that had a NULL Code. So the obvious optimizations was:

``````SELECT Code, COUNT(*) as Nr
FROM MyTable
WHERE Code IN ('A','B','C')``````

And it worked, however I was getting this annoying warning in the execution plan: "Operator used tempdb to spill data during execution". What the hell was that?

Long story short, I found a very nice SO answer that explains it: SQL Server cardinality estimation can use two types of statistics to guess how many rows will get through a predicate filter:

• about the column on average using the density vector
• about particular values for that column using the histogram

When a literal is used, the cardinality estimator can search for that literal in the histogram. When a parameter is used, its value is not evaluated until after cardinality estimation, so the CE has to use column averages in the density vector.

Probably, behind the scenes, ('A','B','C') is treated as a variable, so it only uses the density vector. Also, because how the IN operator is implemented, what happens to the query next is very different than replacing it with a bunch of ORs:

``````SELECT Code, COUNT(*) as Nr
FROM MyTable
WHERE (Code='A' OR Code='B' OR Code='C')``````

Not only the warning disappeared, but the execution time was greatly reduced!

You see, the query here is simplified a lot, but in real life it was part of a larger one, including complicated joins and multiple conditions. With an IN clause, the execution plan would only show me one query, containing multiple joins and covering all of the rows returned. By using OR clauses, the execution plan would show me three different queries, one for each code.

This means that in certain situations, this strategy might not work, especially if the conditions are not disjunct and have rows that meet multiple ones. I am also astonished that for such a simple IN clause, the engine did not know to translate it automatically into a series of ORs! My intent as a developer is clear and the implementation should just take that and turn it into the most effective query possible.

I usually tell people to avoid using OR clauses (and instead try to use ANDs) or query on values that are different (try for equality instead) or using NOT IN. And the reason is again the execution plan and how you cannot prove a general negative claim. Even so, I've always assumed that IN will work as a series or ORs. My reasoning was that, in case of an OR, the engine would have to do something like an expensive DISTINCT operation, something like this:

``````SELECT *
FROM MyTable
WHERE (Code='A' OR Code='B')

-- the above should equate to
SELECT *
FROM MyTable
WHERE Code='A'
UNION -- not a disjunct union so engine would have to eliminate duplicates
SELECT *
FROM MyTable
WHERE Code='B'

-- therefore an optimal query is
SELECT *
FROM MyTable
WHERE Code='A'
UNION ALL - disjunct union
SELECT *
FROM MyTable
WHERE Code='B'
-- assuming that there are no rows that meet both conditions (code A and code B)``````

In this case, however, SQL did manage to understand that the conditions were disjunct so it split the work into three, correctly using the index and each of them being quite efficient.

I learned something today!

# Careful with dates in Javascript

I was working on a grid display and I had to properly sort date columns. The value provided was not a datetime, but instead a string like "20 Jan 2017" or "01 Feb 2020". Obviously sorting them alphabetically would not be very useful. So what I did was implement a custom sorting function that first parsed the strings as dates, then compared them. Easy enough, particularly since the Date object in Javascript has a Parse function that understands this format.

The problem came with a string with the value "01 Jan 0001" which appeared randomly among the existing values. I first thought it was an error being thrown somewhere, or that it would not parse this string or even that it would be an overflow. It was none of that. Instead, it was about handling the year part.

A little context first:

``````Date.parse('01 Jan 0001') //978300000000
new Date(0) //Thu Jan 01 1970 00:00:00

Date.parse('01 Jan 1950') //-631159200000
new Date(Date.parse('01 Jan 1950')) //Sun Jan 01 1950 00:00:00

Date.parse('31 Dec 49 23:59:59.999') //2524600799999
Date.parse('1 Jan 50 00:00:00.000') //-631159200000

new Date(Date.parse('01 Jan 0001')) //Mon Jan 01 2001 00:00:00

``````

The first two lines almost had me convinced Javascript does not handle dates lower than 1970. The next two lines disproved that and made me think it was a case of numerical overflow. The next two demonstrated it was not so. Now look closely at the last line. What? 2001?

The problem was with the handling of years that are numerically smaller than 50. The parser assumes we used a two digit year and translates it into Date.parse('01 Jan 01') which would be 2001. We get a glimpse into how it works, too, because everything between 50 and 99 would be translated into 19xx and everything between 00 and 49 is considered 20xx.

Note that .NET does not have this problem, correctly making the difference between a 2 digit and 4 digit year.

Hope it helps people.

# Creating a Discord chat bot in .NET

Update: due to popular demand, I've added Tyrion as a Github repo.

## Intro

Discord is something I have only vaguely heard about and when a friend told me he used it for chat with friends, I installed it, too. I was pleasantly surprised to see it is a very usable and free chat application, which combines feature of IRC, other messenger applications and a bit of Slack. You can create servers and add channels to them, for example, where you can determine the rights of people and so on. What sets Discord apart from anything, perhaps other than Slack, is the level of "integration", the ability to programatically interact with it. So I create a "bot", a program which stays active and responds to user chat messages and can take action. This post is about how to do that.

Before you implement a bot you obviously need:

All of this has been done to death and you can follow the links above to learn how to do it. Before we continue, a little something that might not be obvious: you can edit a Discord chat invite so that it never expires, as it is the one on this blog now.

## Writing code

One can write a bot in a multitude of programming languages, but I am a .NET dev, so Discord.NET it is. Note that this is an "unofficial" library, so it may not (and it is not) completely in sync with all the options that the Discord API provides. One such feature, for example, is multiple attachments to a message. But I digress.

Since my blog is also written in ASP.NET Core, it made sense to add the bot code to that. Also, in order to make it all clean code, I will use dependency injection as much as possible and use the built-in system for commands, even if it is quite rudimentary.

### Step 1 - making dependencies available

We are going to need these dependencies:

• DiscordSocketClient - the client to connect to Discord
• CommandService - the service managing commands
• BotSettings - a class used to hold settings and configuration
• BotService - the bot itself, which we are going to make implement IHostedService so we can add it as a hosted service in ASP.Net

In order to keep things separated, I will not add all of this in Startup, instead encapsulating them into a Bootstrap class:

``````public static class Bootstrap
{
public static IWebHostBuilder UseDiscordBot(this IWebHostBuilder builder)
{
return builder.ConfigureServices(services =>
{
services
});
}
}
``````

This allows me to add the bot simply in CreateWebHostBuilder as:

``````WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.UseDiscordBot();``````

### Step 2 - the settings

The BotSettings class will be used not only to hold information, but also communicate it between classes. Each Discord chat bot needs an access token to connect and we can add that as a configuration value in appsettings.config:

``````{
...
"DiscordBot": {
"Token":"<the token value>"
},
...
}
``````
``````public class BotSettings
{
public BotSettings(IConfiguration config, IHostingEnvironment hostingEnvironment)
{
Token = config.GetValue<string>("DiscordBot:Token");
RootPath = hostingEnvironment.WebRootPath;
BotEnabled = true;
}

public string Token { get; }
public string RootPath { get; }
public bool BotEnabled { get; set; }
}``````

As you can see, no fancy class for getting the config, nor do we use IOptions or anything like that. We only need to get the token value once, let's keep it simple. I've added the RootPath because you might want to use it to access files on the local file system. The other property is a setting for enabling or disabling the functionality of the bot.

### Step 3 - the bot skeleton

Here is the skeleton for a bot. It doesn't change much outside the MessageReceived and CommandReceived code.

``````public class BotService : IHostedService, IDisposable
{

public BotService(DiscordSocketClient client,
CommandService commandService,
IServiceProvider services,
BotSettings settings)
{
_client = client;
_commandService = commandService;
_services = services;
_settings = settings;
}

// The hosted service has started
{
_commandService.CommandExecuted += CommandExecuted;
_client.Log += Log;
_commandService.Log += Log;
// look for classes implementing ModuleBase to load commands from
// start bot
await _client.StartAsync();
}

// logging
{
// do some logging
}

// bot has connected and it's ready to work
{
// some random stuff you can do once the bot is online:

// set status to online
await _client.SetStatusAsync(UserStatus.Online);
// Discord started as a game chat service, so it has the option to show what games you are playing
// Here the bot will display "Playing dead" while listening
}
{
// message retrieved
}
private async Task CommandExecuted(Optional<CommandInfo> command, ICommandContext context, IResult result)
{
// a command execution was attempted
}

// the hosted service is stopping
{
await _client.SetGameAsync(null);
await _client.SetStatusAsync(UserStatus.Offline);
await _client.StopAsync();
_client.Log -= Log;
_commandService.Log -= Log;
_commandService.CommandExecuted -= CommandExecuted;
}

public void Dispose()
{
_client?.Dispose();
}
}``````

### Step 4 - adding commands

In order to add commands to the bot, you must do the following:

• create a class to inherit from ModuleBase
• add public methods that are decorated with the CommandAttribute
• don't forget to call commandService.AddModuleAsync like above

Here is an example of an enable/disable command class:

``````public class BotCommands:ModuleBase
{

public BotCommands(BotSettings settings)
{
_settings = settings;
}

[Command("bot")]
{
if (string.Equals(rest, "enable",StringComparison.OrdinalIgnoreCase))
{
_settings.BotEnabled = true;
}
if (string.Equals(rest, "disable", StringComparison.OrdinalIgnoreCase))
{
_settings.BotEnabled = false;
}
await this.Context.Channel.SendMessageAsync("Bot is "
+ (_settings.BotEnabled ? "enabled" : "disabled"));
}
}``````

When the bot command will be issued, then the state of the bot will be sent as a message to the chat. If the parameter of the command is enable or disable, the state will also be changed accordingly.

Yet, in order for this command to work, we need to add code to the bot MessageReceived method:

``````private async Task MessageReceived(SocketMessage msg)
{
// do not process bot messages or system messages
if (msg.Author.IsBot || msg.Source != MessageSource.User) return;
// only process this type of message
var message = msg as SocketUserMessage;
if (message == null) return;
// match the message if it starts with R2D2
var match = Regex.Match(message.Content, @"^\s*R2D2\s+", RegexOptions.IgnoreCase);
int? pos = null;
if (match.Success)
{
// this is an R2D2 command, everything after the match is the command text
pos = match.Length;
}
else if (message.Channel is IPrivateChannel)
{
// this is a command sent directly to the private channel of the bot,
// don't expect to start with R2D2 at all, just execute it
pos = 0;
}
if (pos.HasValue)
{
// this is a command, execute it
var context = new SocketCommandContext(_client, message);
await _commandService.ExecuteAsync(context, message.Content.Substring(pos.Value), _services);
}
else
{
// processing of messages that are not commands
if (_settings.BotEnabled)
{
// if the bot is enabled and people are talking about it, show an image and say "beep beep"
if (message.Content.Contains("R2D2",StringComparison.OrdinalIgnoreCase))
{
await message.Channel.SendFileAsync(_settings.RootPath + "/img/R2D2.gif", "Beep beep!", true);
}
}
}
}``````

This code will forward commands to the command service if message starts with R2D2, else, if bot is enabled, will send replies with the R2D2 picture and saying beep beep to messages that contain R2D2.

### Step 5 - handling command results

Command execution may end in one of three states:

• command is not recognized
• command has failed
• command has succeeded

Here is a CommandExecuted event handler that takes these into account:

``````private async Task CommandExecuted(Optional<CommandInfo> command, ICommandContext context, IResult result)
{
// if a command isn't found
if (!command.IsSpecified)
{
await context.Message.AddReactionAsync(new Emoji("🤨")); // eyebrow raised emoji
return;
}

// log failure to the console
if (!result.IsSuccess)
{
await Log(new LogMessage(LogSeverity.Error, nameof(CommandExecuted), \$"Error: {result.ErrorReason}"));
return;
}
// react to message
await context.Message.AddReactionAsync(new Emoji("🤖")); // robot emoji
}``````

Note that the command info object does not expose a result value, other than success and failure.

### Conclusion

This post has shown you how to create a Discord chat bot in .NET and add it to an ASP.Net Core web site as a hosted service. You may see the result by joining this blog's chat and giving commands to Tyr, the chat's bot:

• play
• fart
• use metric or imperial units in messages
• use Yahoo Messenger emoticons in messages
• whatever else I will add in it when I get in the mood :)

# Turning Quicksort on its head

For a more in depth exploration of the concept, read Towards generic high performance sorting algorithms

## Sorting

Consider QuickSort, an algorithm that uses a divide and conquer strategy to sort efficiently and the favourite in computer implementations.

It consists of three steps, applied recursively:

1. find a pivot value
2. reorder the input array so that all values smaller than the pivot are followed by values larger or equal to it (this is called Partitioning)
3. apply the algorithm to each part of the array, before and after the pivot

QuickSort is considered generic, meaning it can sort any type of item, assuming the user provides a comparison function between any two items. A comparison function has the same specific format: compare(item1,item2) returning -1, 0 or 1 depending on whether item1 is smaller, equal or larger than item2, respectively. This formalization of the function lends more credence to the idea that QuickSort is a generic sorting algorithm.

Multiple optimizations have been proposed for this algorithm, including using insertion sort for small enough array segments, different ways of choosing the pivot, etc., yet the biggest problem was always the optimal way in which to partition the data. The original algorithm chose the pivot as the last value in the input array and the average complexity was O(n log n), but worse case scenario was O(n^2), when the array was already sorted and the pivot was the largest value. Without extra information you can never find the optimal partitioning schema (which would be to choose the median value of all items in the array segment you are sorting).

But what if we turn QuickSort on its head? Instead of providing a formalized comparison function and fumbling to get the best partition, why not provide a partitioning function (from which a comparison function is trivial to obtain)? This would allow us to use the so called distribution based sorting algorithms (as opposed to comparison based ones) like Radix, BurstSort, etc, which have a complexity of O(n) in a generic way!

My proposal for a formal signature of a partitioning function is partitionKey(item,level) returning a byte (0-255) and the sorting algorithm would receive this function and a maximum level value as parameters.

Let's see a trivial example: an array of values [23,1,31,0,5,26,15] using a partition function that would return digits of the numbers. You would use it like sort(arr,partFunc,2) because the values are two digits numbers. Let's explore a naive Radix sorting:

• assign 256 buckets for each possible value of the partition function result and start at the maximum (least significant) level
• put each item in its bucket for the current level
• concatenate the buckets
• decrease the level and repeat the process

Concretely:

• level 1: 23 -> bucket 3, 1 -> 1, 31 -> 1, 0 -> 0, 5 -> 5, 26 -> 6, 15 -> 5 results in [0,1,31,5,15,6]
• level 0: 0 -> 0, 1 -> 0, 31 -> 3, 5 -> 0, 15 -> 1, 6 -> 0 results in [0,1,5,6,15,31]

Array sorted. Complexity is O(n * k) where k is 2 in this case and depends on the type of values we have, not on the number of items to be sorted!

More complex distribution sorting algorithms, like BurstSort, optimize their function by using a normal QuickSort in small enough buckets. But QuickSort still requires an item comparison function. Well, it is easy to infer: if partFunc(item1,0) is smaller or larger than partFunc(item2,0) then item1 is smaller or larger than item2. If the partition function values are equal, then increase the level and compare partFunc(item1,1) to partFunc(item2,1).

In short, any distribution sorting algorithm can be used in a generic way provided the user gives it a partitioning function with a formalized signature and a maximum level for its application.

Let's see some example partitioning functions for various data types:

• integers from 0 to N - maximum level is log256(N) and the partition function will return the bytes in the integer from the most significant to the least
• ex: 65534 (0xFFFE) would return 255 (0xFF) for level 0 and 254 (0xFE) for level 1. 26 would return 0 and 26 for the same levels.
• integers from -N to N - similarly, one could return 0 or 1 for level 0 if the number is negative or positive or return the bytes of the equivalent positive numbers from 0 to 2N
• strings that have a maximum length of N - maximum level would be N and the partition function would return the value of the character at the same position as the level
• ex: 'ABC' would return 65, 66 and 67 for levels 0,1,2.
• decimal or floating point or real values - more math intensive functions can be found, but a naive one would be to use a string partitioning function on the values turned to text with a fixed number of digits before and after the decimal separator.
• date and time - easy to turn these into integers, but also one could just return year, month, day, hour, minute, second, etc based on the level
• tuples of any of the types above - return the partition values for the first item, then the second and so on and add their maximum levels

One does not have to invent these functions, they would be provided to the user based on standard types in code factories. Yet even these code factories will be able to encode more information about the data to be sorted than mere comparison functions. Stuff like the minimum and maximum value can be computed by going through all the values in the array to be sorted, but why do it if the user already has this information, for example.

Assuming one cannot find a fixed length to the values to be sorted on, like real values or strings of any length, consider this type of sorting as a first step to order the array as much as possible, then using something like insertion or bubble sort on the result.

## Finding a value or computing distinct values

As an additional positive side effect, there are other processes on lists of items that are considered generic because they use a formalized form function as a parameter. Often found cases include finding the index of an item in a list equal to a given value (thus determining if the value exists in a list) and getting the distinct values from an array. They use an equality function as a parameter which is formalized as returning true or false. Of course, a comparison function could be used, depending on if its result is 0 or not, but a partitioning function can also be used to determine equality, if all of the bytes returned on all of the levels are equal.

But there is more. The format of the partition function can be used to create a hash set of the values, thus reducing the complexity of the search for a value from O(n) to O(log n) and that of getting distinct values from O(n^2) to O(n log n)!

In short, all operations on lists of items can be brought together and optimized by using the same format for the function that makes them "generic": that of a partitioning function.

## Conclusion

As you can see, I am rather proud of the concepts I've explained here. Preliminary tests in Javascript show a 20 fold improvement in performance for ten million items when using RadixSort over the default sort. I would really love feedback from someone who researches algorithms and can even test these assumptions under benchmark settings. Them being complex as they are, I will probably write multiple posts on the subject, trying to split it (partition it?) into easily digestible bits

The concept of using a generic partitioning function format for operations on collections is a theoretical one at the moment. I would love to collaborate with people to get this to production level code, perhaps taking into account advanced concepts like minimizing cache misses and parallelism, not only the theoretical complexity.

# Optimizing complex SQL queries and stored procedures

## Intro

There is a saying that the novice will write code that works, without thinking of anything else, the expert will come and rewrite that code according to good practices and the master will rewrite it so that it works again, thinking of everything. It applies particularly well to SQL. Sometimes good and well tried best practices fail in specific cases and one must guide themselves either by precise measurements of by narrow rules that take decades to learn.

If you ever wondered why some SQL queries are very slow or how to write complex SQL stored procedures without them reaching sentience and behaving unpredictably, this post might help. I am not a master myself, but I will share some quick and dirty ways of writing, then checking your SQL code.

## Some master rules

First of all, some debunking of best practices that make unreasonable assumptions at scale:

1. If you have to extract data based on many parameters, then add them as WHERE or ON clauses and the SQL engine will know how to handle it.

For small queries and for well designed databases, that is correct. The SQL server engine is attempting to create execution plans for these parameter combinations and reuse them in the future on other executions. However, when the number of parameters increases, the number of possible parameter combinations increases exponentially. The execution optimization should not take more than the execution itself, so the engine if just choosing one of the existing plans which appears more similar to the parameters given. Sometimes this results in an abysmal performance.

There are two solutions:

The quick and dirty one is to add OPTION (RECOMPILE) to the parameterized SELECT query. This will tell the engine to always ignore existing execution plans. With SQL 2016 there is a new feature called Query Store plus a graphical interface that explores execution plans, so one can choose which ones are good and which ones are bad. If you have the option, you might manually force an execution plan on specific queries, as well. But I don't recommend this because it is a brittle and nonintuitive solution. You need a DBA to make sure the associations are correct and maintained properly.

The better one, to my own surprise, is to use dynamic SQL. In other words, if you have 20 parameters to your stored procedure, with only some getting used at any time (think an Advanced Search page), create an SQL string only with the parameters that are set, then execute it.

My assumption has always been that the SQL engine will do this for me if I use queries like WHERE (@param IS NULL OR <some condition with @param>). I was disappointed to learn that it does not always do that. Be warned, though, that most of the time multiple query parameters are optimized by running several operations in parallel, which is best!

2. If you query on a column or another column, an OR clause will be optimal.

Think of something like this: You have a table with two account columns AccId and AccId2. You want to query a lot on an account parameter @accountId and you have added an index on each column.

At this time the more readable option, and for small queries readability is always preferable to performance improvement, is WHERE AccId=@accountId OR AccId2=@accountId. But how would the indexes be used here, in this OR clause? First the engine will have to find all entries with the correct AccId, then again find entries with the correct AccId2, but only the entries that have not been found in the first search.

First of all, SQL will not do this very well when the WHERE clause is very complex. Second of all, even if it did it perfectly, if you know there is no overlap, or you don't care or you can use a DISTINCT further on to eliminate duplicates, then it is more effective to have two SELECT queries, one for AccId and the other for AccId2 that you UNION ALL afterwards.

My assumption has always been that the SQL engine will do this automatically. I was quite astounded to hear it was not true. Also, I may be wrong, because different SQL engines and their multitude of versions, compounded with the vast array of configuration options for both engine and any database, behave quite differently. Remember the parallelism optimization, as well.

3. Temporary tables as slow, use table variables instead.

Now that is just simple logic, right? A temporary table uses disk while a table variable uses memory. The second has to be faster, right? In the vast majority of cases this will be true. It all depends (a verb used a lot in SQL circles) on what you do with it.

Using a temporary table might first of all be optimized by the engine to not use the disk at all. Second, temporary tables have statistics, while table variables do not. If you want the SQL engine to do its magic without your input, you might just have to use a temporary table.

4. A large query that does everything is better than small queries that I combine later on.

This is a more common misconception than the others. The optimizations the SQL engine does work best on smaller queries, as I've already discussed above, so if a large query can be split into two simpler ones, the engine will be more likely able to find the best way of executing each. However, this only applies if the two queries are completely independent. If they are related, the engine might find the perfect way of getting the data in a query that combines them all.

Again, it depends. One other scenario is when you try to DELETE or UPDATE a lot of rows. SQL is always "logging" the changes that it does on the off chance that the user cancels the query and whatever incomplete work has been done has to be undone. With large amounts of data, this results into large log files and slow performance. One common solution is to do it in batches, using UPDATE (TOP 10000) or something similar inside a WHILE loop. Note that while this solves the log performance issue, it adds a little bit of overhead for each executed UPDATE

5. If I have an index on a DATETIME column and I want to check the records in a certain day, I can use CAST or CONVERT.

That is just a bonus rule, but I've met the problem recently. The general rule is that you should never perform calculations on columns inside WHERE clauses. So instead of WHERE CAST(DateColumn as DATE)=@date use WHERE DateColumn>=@date AND DateColumn<DATEADD(DAY,1,@date). The calculation is done (once) on the parameters given to the query, not on every value of DateColumn. Also, indexes are now used.

## Optimizing queries for dummies

So how does one determine if one of these rules apply to their case? "Complex query" might mean anything. Executing a query multiple times results in very different results based on how the engine is caching the data or computing execution plans.

A lot of what I am going to say can be performed using SQL commands, as well. Someone might want to use direct commands inside their own tool to monitor and manage performance of SQL queries. But what I am going to show you uses the SQL Management Studio and, better still, not that horrid Execution Plan chart that often crashes SSMS and it is hard to visualize for anything that the most simple queries. Downside? You will need SQL Management Studio 2014 or higher.

There are two buttons in the SSMS menu. One is "Include Actual Execution Plan" which generates an ugly and sometimes broken chart of the execution. The other one is "Include Live Query Statistics" which seems to be doing the same, only in real time. However, the magic happens when both are enabled. In the Results tab you will get not only the query results, but also tabular data about the execution performance. It is amazingly useful, as you get a table per each intermediary query, for example if you have a stored procedure that executes several queries in a row, you get a table for each.

Even more importantly, it seems that using these options will start the execution without any cached data or execution plans. Running it several times gives consistent execution times.

In the LiveQuery tables, the values we are interested about are, in order of importance, EstimateIO, EstimateCPU and Rows.

EstimateIO is telling us how much of the disk was used. The disk is the slowest part of a computer, especially when multiple processes are running queries at the same time. Your objective is to minimize that value. Luckily, on the same row, we get data about the substatement that generated that row, which parameters were used, which index was used etc. This blog is not about how to fix every single scenario, but only on how to determine where the biggest problems lie.

EstimateCPU is saying how much processing power was used. Most of the time this is very small, as complex calculations should not be performed in queries anyway, but sometimes a large value here shows a fault in the design of the query.

Finally, Rows. It is best to minimize the value here, too, but it is not always possible. For example a COUNT(*) will show a Clustered Index Scan with Rows equal to the row count in the table. That doesn't cause any performance problems. However, if your query is supposed to get 100 rows and somewhere in the Live Query table there is a value of several millions, you might have used a join without the correct ON clause parameters or something like that.

## Demo

Let's see some examples of this. I have a Main table, with columns ID BIGINT, Random1 INT, Random2 NVARCHAR(100) and Random3 CHAR(10) with one million rows. Then an Ind table, with columns ID BIGINT, Qfr CHAR(4) and ValInd BIGINT with 10000 rows. The ID table is common with the Main table ID column and the Qfr column has only three possible values: AMT, QTY, Sum.

Here is a demo on how this would work:

``````DECLARE @r1 INT = 1300000
DECLARE @r2 NVARCHAR(100) = 'a'
DECLARE @r3 CHAR(10) = 'A'
DECLARE @qfr CHAR(4) = 'AMT'
DECLARE @val BIGINT = 500000

DECLARE @r1e INT = 1600000
DECLARE @r2e NVARCHAR(100) = 'z'
DECLARE @r3e CHAR(10)='Z'
DECLARE @vale BIGINT = 600000

SELECT *
FROM Main m
INNER JOIN Ind i
ON m.ID=i.ID
WHERE (@r1 IS NULL OR m.Random1>=@r1)
AND (@r2 IS NULL OR m.Random2>=@r2)
AND (@r3 IS NULL OR m.Random3>=@r3)
AND (@val IS NULL OR i.ValInd>=@val)
AND (@r1e IS NULL OR m.Random1<=@r1e)
AND (@r2e IS NULL OR m.Random2<=@r2e)
AND (@r3e IS NULL OR m.Random3<=@r3e)
AND (@vale IS NULL OR i.ValInd<=@vale)
AND (@qfr IS NULL OR i.Qfr=@qfr)``````

I have used 9 parameters, each with their own values, to limit the number of rows I get. The Live Query result is:

You can see that the EstimateIO values are non-zero only on the Clustered Index Scans, one for each table. Where is how the StmtText looks like: "|--Clustered Index Scan(OBJECT:([Test].[dbo].[Ind].[PK__Ind__DEBF89006F996CA8] AS [i]),  WHERE:(([@val] IS NULL OR [Test].[dbo].[Ind].[ValInd] as [i].[ValInd]>=[@val]) AND ([@vale] IS NULL OR [Test].[dbo].[Ind].[ValInd] as [i].[ValInd]<=[@vale]) AND ([@qfr] IS NULL OR [Test].[dbo].[Ind].[Qfr] as [i].[Qfr]=[@qfr])) ORDERED FORWARD)".

This is a silly case, but you can see that the @parameter IS NULL type of query condition has not been removed, even when parameter is clearly not null.

Let's change the values of the parameters:

``````DECLARE @r1 INT = 300000
DECLARE @r2 NVARCHAR(100) = NULL
DECLARE @r3 CHAR(10) = NULL
DECLARE @qfr CHAR(4) = NULL
DECLARE @val BIGINT = NULL

DECLARE @r1e INT = 600000
DECLARE @r2e NVARCHAR(100) = NULL
DECLARE @r3e CHAR(10)=NULL
DECLARE @vale BIGINT = NULL``````

Now the Live Query result is:

Same thing! 5.0 and 7.2

Now, let's do the same thing with dynamic SQL. It's a little more annoying, mostly because of the parameter syntax, but check it out:

``````DECLARE @sql NVARCHAR(Max)

DECLARE @r1 INT = 300000
DECLARE @r2 NVARCHAR(100) = NULL
DECLARE @r3 CHAR(10) = NULL
DECLARE @qfr CHAR(4) = NULL
DECLARE @val BIGINT = NULL

DECLARE @r1e INT = 600000
DECLARE @r2e NVARCHAR(100) = NULL
DECLARE @r3e CHAR(10)=NULL
DECLARE @vale BIGINT = NULL

SET @sql=N'
SELECT *
FROM Main m
INNER JOIN Ind i
ON m.ID=i.ID
WHERE 1=1 '
IF @r1 IS NOT NULL SET @sql+=' AND m.Random1>=@r1'
IF @r2 IS NOT NULL SET @sql+=' AND m.Random2>=@r2'
IF @r3 IS NOT NULL SET @sql+=' AND m.Random3>=@r3'
IF @val IS NOT NULL SET @sql+=' AND i.ValInd>=@val'
IF @r1e IS NOT NULL SET @sql+=' AND m.Random1<=@r1e'
IF @r2e IS NOT NULL SET @sql+=' AND m.Random2<=@r2e'
IF @r3e IS NOT NULL SET @sql+=' AND m.Random3<=@r3e'
IF @qfr IS NOT NULL SET @sql+=' AND i.Qfr=@qfr'
IF @vale IS NOT NULL SET @sql+=' AND i.ValInd<=@vale'

PRINT @sql

EXEC sp_executesql @sql,
N'@r1 INT, @r2 NVARCHAR(100), @r3 CHAR(10), @qfr CHAR(4),@val BIGINT,@r1e INT, @r2e NVARCHAR(100), @r3e CHAR(10),@vale BIGINT',
@r1,@r2,@r3,@qfr,@val,@r1e,@r2e,@r3e,@vale
``````

Now the Live Query results are:

At first glance we have not changed much. IO is still 5.0 and 7.2. Yet there are 3 less execution steps. There is no parallelism and the query has been executed in 5 seconds, not 6. The StmtText for the same thing is now: "|--Clustered Index Scan(OBJECT:([Test].[dbo].[Ind].[PK__Ind__DEBF89006F996CA8] AS [i]), ORDERED FORWARD)". The printed SQL command is:

``````SELECT *
FROM Main m
INNER JOIN Ind i
ON m.ID=i.ID
WHERE 1=1  AND m.Random1>=@r1 AND m.Random1<=@r1e``````

## Conclusion

Again, this is a silly example. But with some results anyway! In my work I have used this to get a stored procedure to work three to four times faster!

One can optimize usage of IO, CPU and Rows by adding indexes, by narrowing join conditions, by reducing the complexity of executed queries, eliminating temporary tables, partitioning existing tables, adding or removing hints, removing computation from queried columns and so many other possible methods, but they amount to nothing if you cannot measure the results of your changes.

By using Actual Execution Plan together with Live Query Statistics you get:

• consistent execution times and disk usage
• a clear measure of what went on with each subquery

BTW, you get the same effect if you use SET STATISTICS PROFILE ON before the query. Yet, I wrote this post with someone that doesn't want to go into extra SQL code in mind. Also, when calculating performance, it is recommended to add a DBCC FREEPROCCACHE line before execution OR add the option RECOMPILE to your query (this doesn't work on a stored procedure execution, you would have to change the SP queries to include RECOMPILE).

I wish I had some more interesting examples for you, guys, but screenshots from the workplace are not something I want to do and I don't do any complex SQL work at home. I hope this helps.

# I finally understood what a reducer is!

When I was looking at Javascript frameworks like Angular and ReactJS I kept running into these weird reducers that were used in state management mostly. It all felt so unnecessarily complicated, so I didn't look too closely into it. Today, reading some random post on dev.to, I found this simple and concise piece of code that explains it:

``````// simple to unit test this reducer
function maximum(max, num) { return Math.max(max, num); }

// read as: 'reduce to a maximum'
let numbers = [5, 10, 7, -1, 2, -8, -12];
let max = numbers.reduce(maximum);``````

Kudos to David for the code sample.

The reducer, in this case, is a function that can be fed to the reduce function, which is known to developers in Javascript and a few other languages, but which for .NET developers it's foreign. In LINQ, we have Aggregate!

``````// simple to unit test this Aggregator ( :) )
Func<int, int, int> maximum = (max, num) => Math.Max(max, num);

// read as: 'reduce to a maximum'
var numbers = new[] { 5, 10, 7, -1, 2, -8, -12 };
var max = numbers.Aggregate(maximum);``````

Of course, in C# Math.Max is already a reducer/Aggregator and can be used directly as a parameter to Aggregate.

I found a lot of situations where people used .reduce instead of a normal loop, which is why I almost never use Aggregate, but there are situations where this kind of syntax is very useful. One would be in functional programming or LINQ expressions that then get translated or optimized to something else before execution, like SQL code. (I don't know if Entity Framework translates Aggregate, though). Another would be where you have a bunch of reducers that can be used interchangeably.

# Towards generic high performance sorting algorithms

## Intro

I want to examine together with you various types of sort algorithms and the tricks they use to lower the magic O number. I reach the conclusion that high performance algorithms that are labeled as specific to a certain type of data can be made generic or that the generic algorithms aren't really that generic either. I end up proposing a new form of function that can be fed to a sorting function in order to reach better performance than the classic O(n*log(n)). Extra bonus: finding distinct values in a list.

## Sorting

But first, what is sorting? Given a list of items that can be compared to one another as lower or higher, return the list in the order from lowest to highest. Since an item can be any type of data record, to define a generic sorting algorithm we need to feed it the rules that make an item lower than another and that is called the comparison function. Let's try an example in Javascript:

``````  // random function from start to end inclusive
function rand(start, end) {
return parseInt(start + Math.random() * (end - start + 1));
}

// measure time taken by an action and output it in console
let perfKey = 0;
function calcPerf(action) {
const key = perfKey++;
performance.mark('start_' + key);
action();
performance.mark('end_' + key);
const measure = performance.measure('measure_' + key, 'start_' + key, 'end_' + key);
console.log('Action took ' + measure.duration);
}

// change this based on how powerful the computer is
const size = 10000000;
// the input is a list of size 'size' containing random values from 1 to 50000
const input = [];
for (let i = 0; i < size; i++)
input.push(rand(1, 50000));

// a comparison function between two items a and b
function comparisonFunction(a, b) {
if (a > b)
return 1;
if (a < b)
return -1;
return 0;
}

const output = [];
// copy input into output, then sort it using the comparison function
// same copying method will be used for future code
calcPerf(() => {
for (let i = 0; i < size; i++)
output.push(input[i]);
output.sort(comparisonFunction);
});
``````

It's not the crispest code in the world, but it's simple to understand:

• calcPerf is computing the time it takes for an action to take and logs it to the console
• start by creating a big array of random numbers as input
• the array in a result array and sorting it with the default sort function, to which we give the comparison function
• display the time it took for the operation.

This takes about 4500 milliseconds on my computer.

Focus on the comparison function. It takes two items and returns a number that is -1, 0 or 1 depending on whether the first item is smaller, equal or larger than the second. Now let's consider the sorting algorithm itself. How does it work?

A naive way to do it would be to find the smallest item in the list, move it to the first position in the array, then continue the process with the rest of the array. This would have a complexity of O(n2). If you don't know what the O complexity is, don't worry, it just provides an easy to spell approximation of how the amount of work would increase with the number of items in the input. In this case, 10 million records, squared, would lead to 100 trillion operations! That's not good.

Other algorithms are much better, bringing the complexity to O(n*log(n)), so assuming base 10, around 70 million operations. But how do they improve on this? Surely in order to sort all items you must compare them to each other. The explanation is that if a<b and b<c you do not need to compare a to c. And each algorithm tries to get to this in a different way.

However, the basic logic of sorting remains the same: compare all items with a subset of the other items.

## Partitioning

A very common and recommended sorting algorithm is QuickSort. I am not going to go through the entire history of sorting algorithms and what they do, you can check that out yourself, but I can focus on the important innovation that QuickSort added: partitioning. The first step in the algorithm is to choose a value out of the list of items, which the algorithm hopes it's as close as possible to the median value and is called a pivot, then arrange the items in two partitions: the ones smaller than the pivot and the ones larger than the pivot. Then it proceeds on doing the same to each partition until the partitions are small enough to be sorted by some other sort algorithm, like insertion sort (used by Chrome by default).

Let's try to do this manually in our code, just the very first run of the step, to see if it improves the execution time. Lucky for us, we know that the median is around 25000, as the input we generated contains random numbers from 1 to 50000. So let's copy the values from input into two output arrays, then sort each of them. The sorted result would be reading from the first array, then from the second!

``````  // two output arrays, one for numbers below 25000, the other for the rest
const output1 = [];
const output2 = [];
const pivot = 25000;

calcPerf(() => {
for (let i = 0; i < size; i++) {
const val = input[i];
if (comparisonFunction(val, pivot) < 0)
output1.push(val);
else
output2.push(val);
}
// sorting smaller arrays is cheaper
output1.sort(comparisonFunction);
output2.sort(comparisonFunction);
});
``````

Now, the performance is slightly better. If we do this several times, the time taken would get even lower. The partitioning of the array by an operation that is essentially O(n) (we just go once through the entire input array) reduces the comparisons that will be made in each partition. If we would use the naive sorting, partitioning would reduce nto n+(n/2)2+(n/2)2 (once for each partitioned half), thus n+n2/2. Each partitioning almost halves the number of operations!

So, how many times can we half the number of operations for? Imagine that we do this with an array of distinct values, from 1 to 10 million. In the end, we would get to partitions of just one element and that means we did a log2(n) number of operations and for each we added one n (the partitioning operation). That means that the total number of operations is... n*log(n). Each algorithm gets to this in a different way, but at the core of it there is some sort of partitioning, that b value that makes comparing a and c unnecessary.

Note that we treated the sort algorithm as "generic", meaning we fed it a comparison function between any two items, as if we didn't know how to compare numbers. That means we could have used any type of data as long as we knew the rule for comparison between items.

There are other types of sorting algorithms that only work on specific types of data, though. Some of them claim a complexity of O(n)! But before we get to them, let's make a short detour.

### Distinct values

Another useful operation with lists of items is finding the list of distinct items. From [1,2,2,3] we want to get [1,2,3]. To do this, we often use something called a trie, a tree-like data structure that is used for quickly finding if a value exists or not in a list. It's the thing used for autocorrect or finding a word in a dictionary. It has an O(log n) complexity in checking if an item exists. So in a list of 10 million items, it would take maybe 20 operations to find the item exists or not. That's amazing! You can see that what it does is partition the list down to the item level.

Unfortunately, this only works for numbers and strings and such primitive values. If we want to make it generic, we need to use a function that determines when two items are equal and then we use it to compare to all the other items we found as distinct so far. That makes using a trie impossible.

Let me give you an example: we take [1,1,2,3,3,4,5] and we use an externally provided equality function:

• create an empty output of distinct items
• take first item (1) and compare with existing distinct items (none)
• take next item (1) and compare with existing distinct items (1)
• item is found, so we do nothing
• ...
• we take the last item (5) and compare with existing items (1,2,3,4)

The number of operations that must be taken is the number of total items multiplied by the average number of distinct items. That means that for a list of already distinct values, the complexity if O(n2). Not good! It increases exponentially with the number of items. And we cannot use a trie unless we have some function that would provide us with a distinctive primitive value for an item. So instead of an equality function, a hashing function that would return a number or maybe a string.

However, given the knowledge we have so far, we can reduce the complexity of finding distinct items to O(n*log(n))! It's as simple as sorting the items, then going through the list and sending to output an item when different from the one before. One little problem here: we need a comparison function for sorting, not an equality one.

## So far

We looked into the basic operations of sorting and finding distinct values. To be generic, one has to be provided with a comparison function, the other with an equality function. However, if we would have a comparison function available, finding distinct generic items would become significantly less complex by using sorting. Sorting is better than exponential comparison because it uses partitioning as an optimization trick.

## Breaking the n*log(n) barrier

As I said above, there are algorithms that claim a much better performance than n*log(n). One of them is called RadixSort. BurstSort is a version of it, optimized for strings. CountSort is a similar algorithm, as well. The only problem with Radix type algorithms is that they only work on numbers or recursively on series of numbers. How do they do that? Well, since we know we have numbers to sort, we can use math to partition the lot of them, thus reducing the cost of the partitioning phase.

Let's look at our starting code. We know that we have numbers from 1 to 50000. We can find that out easily by going once through all of them and computing the minimum and maximum value. O(n). We can then partition the numbers by their value. BurstSort starts with a number of "buckets" or lists, then assigns numbers to the buckets based on their value (dividing the value to the number of buckets). If a bucket becomes too large, it is "burst" into another number of smaller buckets. In our case, we can use CountSort, which simply counts each occurrence of a value in an ordered array. Let's see some code:

``````  const output = [];
const buckets = [];
calcPerf(() => {
// for each possible value add a counter
for (let i = 1; i <= 50000; i++)
buckets.push(0);
// count all values
for (let i = 1; i <= size; i++) {
const val = input[i];
buckets[val - 1]++;
}
// create the output array of sorted values
for (let i = 1; i <= 50000; i++) {
const counter = buckets[i - 1];
for (let j = 0; j < counter; j++)
output.push(i);
}
});
``````

This does the following:

• create an array from 1 to 50000 containing zeros
• for each value in the input, increment the bucket for that value
• at the end just go through all of the buckets and output the value as many times as the value in the bucket shows

This algorithm generated a sorted output array in 160 milliseconds!

And of course, it is too good to be true. We used a lot of a priori knowledge:

• min/max values were already known
• the values were conveniently close together integers so we can use them as array indexes

I can already hear you sigh "Awwh, so I can't use it!". Do not despair yet!

The Radix algorithm, that is used only for numbers, is also used on strings. How? Well, a string is reducible to a list of numbers (characters) so one can recursively assign each string into a bucket based on the character value at a certain index. Note that we don't have to go through the entire string, the first few letters are enough to partition the list in small enough lists that can be cheaply sorted.

Do you see it yet?

## A generic partition function

What if we would not use an equality function or a comparison function or a hashing function as a parameter for our generic sort/distinct algorithm? What if we would use a partition function? This partition function would act like a multilevel hashing function returning values that can also be compared to each other. In other words, the generic partition function could look like this:

function partitionFunction(item, level) returning a byte

For strings it returns the numeric value of the character at position level or 0. For numbers it returns the high to low byte in the number. For object instances with multiple properties, it would return a byte for each level in each of the properties that we want to order by. Radix style buckets would use the known values from 0 to 255. The fact that the multilevel partitioning function is provided by the user means we can pack in it all the a priori knowledge we have, while keeping the sorting/distinct algorithm unchanged and thus, generic! The sorting will be called by providing two parameters: the partitioning function and the maximum level to which it should be called:

sort(input, partitioningFunction, maxLevel)

## A final example

Here is an implementation of a radix sorting algorithm that receives a multilevel partitioning function using our original input. Note that it is written so that it is easily read and not for performance:

``````  // will return a sorted array from the input array
// using the partitioning function up to maxLevel
let buckets = Array.from({length: 256}, () => []);
buckets[0] = input;
// reverse order, because level 0 should be the most significant
for (let level = maxLevel-1; level >=0; level--) {
let tempBuckets = Array.from({length: 256}, () => []);
for (let bucketIndex = 0; bucketIndex < buckets.length; bucketIndex++) {
const bucket = buckets[bucketIndex];
const bucketLength = bucket.length;
for (let bucketOffset = 0; bucketOffset < bucketLength; bucketOffset++) {
const val = bucket[bucketOffset];
const partByte = partitioningFunction(val, level);
tempBuckets[partByte].push(val);
}
}
buckets = tempBuckets;
}
const output = [].concat(...buckets);
return output;
}

// return value bytes, from the most significant to the least
// being <50000 the values are always 2 bytes
function partitioningFunction(item, level) {
if (level === 0) return item >> 8;
if (level === 1) return item & 255;
return 0;
}

let output3 = [];
calcPerf(() => {
});
``````

Want to know how long it took? 1300 milliseconds!

You can see how the same kind of logic can be used to find distinct values, without actually sorting, just by going through each byte from the partitioning function and using them as values in a trie, right?

## Conclusion

Here is how a generic multilevel partitioning function replaces comparison, equality and hashing functions with a single concept that is then used to get high performance from common data operations such as sorting and finding distinct values.

I will want to work on formalizing this and publishing it as a library or something like that, but until then, what do you think?

## Wait, there is more

There is a framework in which something similar is being used: SQL. It's the most common place where ORDER BY and DISTINCT are used. In SQL's case, we use an optimization method that uses indexes, which are also trie data structures storing the keys that we want to order or filter by. Gathering the data to fill a database index also has its complexity. In this case, we pre-partition once and we sort many. It's another way of reducing the cost of the partitioning

However, this is just a sub-type of the partition function that I am talking about, one that uses a precomputed data structure to reach its goal. The multilevel partition function concept I am describing here may be pure code or some other encoding of information we know out of hand before doing the operation.

Finally, the complexity. What is it? Well instead of O(n*log(n)) we get O(n*k), where k is the maximum level used in the partition function. This depends on the data, so it's not a constant, but it's the closest theoretical limit for sorting, closer to O(n) than the classic log version. I am not the best algorithm and data structure person, so if you have ideas about it and want to help me out, I would be grateful.

# The things I wanted to learn and what really happened

Four years ago this day I was blogging about a list of technologies that I wanted to learn or at least explore. It was a very ambitious list, with the idea that I might shame myself into studying at least part of it. Apparently, I am shameless. Anyway, this is a follow-up post on how it all went down.

## .NET Core: MVC, Entity Framework, etc.

I actually had the opportunity to use ASP.Net Core at my job. I didn't have a lot of it, though, so I just did the usual: start working on something, google the hell out of everything, make it work. I used .NET Core 1 and 2, moved to 3 and now they are just getting out 5. Was it useful? Well, actually no.

The state of the software industry is in constant flux and while new technologies pop up all the time, there is also a strong force moving the other way, mainly coming from the people that pay for software. Why invest in a new technology when the old one has been working fine for years? Yeah, don't answer that. My point is that while staying current with .NET Core was a boon, I didn't really see a lot of paying customers and employers begging me to do anything with it. In fact, I left my job working on a .NET Framework 4.7 automation project to work on an ASP.Net MVC 4 application written in Visual Basic. So I am learning things that are new to me, but really are very old.

All I am saying is that there is a paradox of job opportunities for technologies that you are supposed to know and be an expert in, but for which few have had the courage to actually pay before.

Bottom line: having been familiar with older technology that made .Net Core exist, it was kind of simple for me to get into it, but I am not an expert in it. The fact that software generally moved to the web and that server code is now as slim as it can be also undermines the value of being proficient in it. That is, until you really need to be.

## OData and OAuth

Simply said, nothing on this front. These are mostly related to frontend stuff, to new web facing applications that can be found at a public URL. I have not worked in this field basically since forever.

## Typescript, Angular

Oh, the promise of Typescript: a strongly typed language than compiles to Javascript and can be used to code in using static analysis tools, structured code, clear project boundaries! I had the opportunity to work with it. I did that in a much lesser degree than I should have. Yet the tiny contact with the new language left me disappointed. Because it is a superset of Javascript, it inherits most of its problems, can be easily hacked and the tooling chain is basically the Javascript one.

I commend Microsoft for even attempting to develop this language and I see that it has become popular indeed. I like C#. I like the vanilla Javascript of old. I dislike the weak hybrid that Typescript seems to be. I may be wrong. Things might evolve in very interesting directions with Typescript. I will wait until then.

And Angular has also grown in a completely different beast. I thought Angular 1 was great, then they came in with version 2 which was completely different. Now it's 9 or 10 or whatever. I liked the structured projects and how easily data change could be controlled from non-UI code. Yet did it all have to be this complex?

## Node.JS, npm, Javascript ES6+, Bower, etc.

I thought that since I like Javascript, I might enjoy Node.JS. Working with Typescript I had to have contact with npm and at least install Node on the computer. Yet I got more and more disillusioned. A complicated chain of tools that seem to be targeted to a concept of "app" rather than a website, for which you need a completely different set of tools, the numerous changes in the paradigm of Javascript and the systems using it and, frankly, a lack of a real reason for using Javascript when I can use C# made me stop trying to get in on the action on this front. And before I started to understand the Node ecosystem, Deno appeared.

Personally it feels that Javascript is growing too fast and uncontrolled, like a cancer. The language and tooling need to stabilize and by that I mean cut some of the fat away, not add new things. I loved some of the additions to the standard, like arrow functions, let/const and block scope, nullable and spread operators, iterators and generator functions, but then it just started to get more and more bloated, like trying to absorb every new concept in all of the other languages.

Bottom line: it is ironic that I am now working in an app that must work on Internet Explorer 11, so I cannot even use the new features of the Javascript standard. And yes, I hate that, but at the same time I can do the job just fine. Food for thought.

## Docker

What a great idea this Docker: just tell the system what kind of setup you need and it creates it for you. It can even create it for you from scratch every single time, in isolation, so you can run your software without having to install all that heavy crap I was just telling you about above. And yet I hated it with a vengeance from the moment I tried to use it. With a clear Linux vibe, it wouldn't even work right on Windows. The images for these pseudo-virtual machines were huge. The use was cumbersome. The tooling reminded me of the good old days when I installed a Slackware distro on any old computer, without X system, of course, and then compiled beta versions of every single software I wanted to use and repeated the process whenever I had an issue.

For a tool that is supposed to bring consistency and ease of use, it worked really inconsistently and was hard to manage. I understand that maybe all that would have gone away if I invested the effort of understanding the whole concept, but it really felt like going backward instead of forward. One day, perhaps, I will change my mind. As of now, I am just waiting for the person who reinvents Docker and makes it do what it has promised to do: ease my life.

## Parallel programming

Hey, this is actually a plus! I had a lot of opportunities to work with parallel programming, understanding the various pitfalls and going deep in the several types of parallel programming concepts in .NET. Am I a parallel programming guru now? No. But I went through the await/async phase and solved problems at all levels of parallelism. I don't like async/await. It is a good idea, it is great when you start a new project, but to add async/await in existing legacy code is a pain the butt. Inevitably you will want to execute an async method from a sync context, control the order or amount of parallel processing, forget to call an async method with await or trying to run one parallely by not calling it with await on purpose and finding that your scope has been disposed.

Bottom line: I don't have to like it to use async/await, but I feel (without actually knowing a perfect solution) the design of the feature could have been better. And now they added it to Javascript, too! Anyway, I am comfortable in the area now.

## HTML5, Responsive Design, LESS/SASS, ReactJS, new CSS, frontend

A lot of new stuff to learn for a frontend developer. And how glad I am that I am not it! Again I hit the reality of business requirements vs technology requirements. To learn front end development right now is a full job that you have to do alone, for free and in your spare time. Meanwhile, the projects you are working on are not on the cutting edge, no one wants to change them just to help you learn something and, again, these concepts are mostly for public facing web applications. Maybe even mobile development.

It does help to know all of this, but it is not a job that I want to do. Frontend is for the young. Can I say that? ReactJS, the few times I looked at it, appeared to me to be a sort of ASP for the browser, with that ugly JSX format that combines code and markup and that complicated state flow mechanism. Angular felt too stuffy and so on. Every time I look into frontend stuff it seems like software for the server side from twenty years before.

Bottom line: If any web framework appealed to me, that would be VueJS! And no one used it at any of my places of work. It seems a framework dedicated to staying simple and efficient. And while I want to know the concepts in all of this stuff, why would I go deep into this when I need a UI designer for anything with a shape? I will be waiting for the resurgence of simple frameworks using the new features of HTML99 and that do not require to learn an entire new ecosystem to make anything work.

## WPF, UWP

I remember I absolutely loved developing in WPF. The MVVM concept of separating the UI completely from the code controlling it, which is then completely separated from the data models used, was very nice. The complete customizability of the UI without changing the code at all was amazing. The performance, not so much. But did it matter? I thought not. Yet the web obliterated the tech from the development world. I believe it deserves a rebirth.

That being said, I have done nothing in this direction either. Shame on me.

## Deep learning, AI, machine learning

I really wanted to understand and use artificial intelligence, especially since I did my licence paper on neural networks. I love the new concepts, I've read some of the stuff they came up with and I've even been to some conferences on machine learning. Yet I have not yet found the project that could make it worthwhile. Machine learning and especially deep learning projects require huge amounts of data, while working on the underlying technology requires advanced knowledge of statistics and math, which I do not have.

It seems to me that I will always be the end user of AI rather than working on building it.

## Conclusion

So there you have it: I worked on .NET Core and parallelism and even microservices, dabbled in Typescript and Angular, created a full on LINQ implementation on Javascript and later Typescript. Meanwhile the world went more and more towards frontend and mobile development and Node and stuff like machine learning. And presently I am working for a bank, coding in VB and using jQuery.

Strangely, I don't really feel guilty about it. Instead I feel like I am still going on the path of discovery, only more of myself rather than of a particular technology. The more I wait, the more new stuff appears and makes whatever I was dreaming of learning obsolete. I even considered not being a developer anymore, maybe becoming an architect or application designer. When something new appears, the immediate instinct is to check it out, to see what novelties they come up with and to find the problems and then discover solutions, then blog about them. It's a wonderfully addictive game, but after playing it for a while, it just blurs into the same thing over and over again. It's not the technology that matters, but the problems that need fixing. As long as I have problems to fix and I can find the solution, I am happy.

I ask you, if you got to this point, what is the point of learning any technology, when you don't have the cause, the purpose, the problem that you have to solve? An HR person asked me recently why I keep working where I do. I answered: because they need me.

# Interview question: write a CSV exporter

## Intro

So the requirement is "Write a class that would export data into a CSV format". This would be different from "Write a CSV parser", which I think could be interesting, but not as wildly complex as this. The difference comes from the fact that a CSV parser brings a number of problems for the interviewed person to think about right away, but then it quickly dries up as a source for intelligent debate. A CSV exporter seems much simpler, because the developer controls the output, but it increases in complexity as the interview progresses.

This post is written from the viewpoint of the interviewer.

## Setup

First of all, you start with the most basic question: Do you know what CSV is? I was going to try this question out on a guy who came interviewing for senior developer and I was excited to see how it would go. He answered he didn't know what CSV was. Bummer! I was incredulous, but then I quickly found out he didn't know much else either. CSV is a text format for exporting a number of unidimensional records. The name comes from Comma Separated Values and might at first glance appear to be a tabular data format, an idea made even more credible by Excel being able to open and export .csv files. But it is not. As the name says, it has values separated by a comma. It might even be just one record. It might be containing multiple records of different types. In some cases, the separator for value and record are not even commas or newline.

It is important to see how the interviewee explains what CSV is, because it is a concept that looks deceivingly simple. Someone who first considers the complexity of the format before starting writing the code works very differently in a team than someone who throws themselves into the code, confident (or just unrealistically optimistic) that they would solve any problem down the line.

Some ideas to explore, although it pays off to not bring them up yourself:

• What data do you need to export: arrays, data tables, list of records?
• Are the records of the same type?
• Are there restrictions on the type of record?
• What separators will there be used? How to escape values that contain chosen separators?
• Do values have restrictions, like not containing separators?
• CSV header: do we support that? What does it mean in the context of different types of input?
• Text encoding, unicode, non-ASCII characters
• How to handle null values?
• Number and date formatting
• Is there an RFC or a specification document for the CSV export format?

## Implementation

In this particular interview I have chosen that the CSV exporter class will only support an input of `IEnumerable<T>` (this is .NET speak for a bunch of objects of the same type).

Give ample opportunities for questions from the person interviewed. This is not a speed test. It is important if the candidate considers by themselves issues like:

• are the object properties simple types? Like string, long, integer, decimal, double, float, datetime?
• since the requirement is any T, what about objects that are arrays, or self referencing, or having complex objects as properties?
• how to extract the values of any object (discussions about .NET reflection or Javascript object property discovery show a lot about the interviewee, especially if they start said discussions)

Go through the code with the candidate. This shows their ability to develop software. How will they name the class, what signature will they use for export method, how they structure the code and how readable it is.

At this stage you should have a pretty good idea if the candidate is intelligent, competent and how they handle a complex problem from requirement to implementation.

## Dig deeper

This is the time to ask the questions yourself and see how they react to new information, the knowledge that they should have asked themselves the same questions and the stress of changing their design:

• are comma and newline the only supported separators?
• are separators characters or strings?
• what if an exported value is a string containing a comma?
• do you support values containing newline?
• if you use quotes to hold a value containing commas and newlines, what happens if values contain quotes
• empty or null values. Any difference? How to export them? What if the object itself is null?
• how to handle the header of the CSV, where do you get the name of the properties?
• what if the record type is an array or IEnumerable?
• what will be the numeric and date formatting used for export?
• does the candidate know what text encoding is? Which one will they use and why?

How have the answers to these questions changed the design? Did the candidate redesign the work or held tight to the original idea and tried to fix everything as it comes?

At this point you should know how the person being interviewed responds to new information, even scope creep and, maybe most importantly, to stress. But we're not done, are we?

## Bring the pain

Bring up the concept of unit testing. If you are lucky, the candidate already brought it up. Either way, now it is time to:

• split the code into components: the reflection code, the export code, the file system code (if any).
• abstract components into interfaces in order to mock them in unit tests
• collect all the specifications gathered so far in order to cover all the cases
• ask the candidate to write one unit test

## Conclusion

A seemingly simple question will take you and the interview candidate through:

• finding out how the other person thinks
• specification gathering
• code design
• implementation
• technical knowledge in a multitude of directions
• development process
• separation of concerns, unit testing
• human interaction in a variety of circumstances
• determining how the candidate would fit in a team

Not bad for a one line question, right?

# The obvious mistakes you still make as a senior developer

I want to write this post to talk about the most common mistake I make as a software developer, even after almost 20 years of experience. And it's not code related. It's more human. But I would also like to hear what you think your biggest mistakes are that are not related to lack of experience. Don't be shy!

My mistake: assumptions.

I was assigned this bug recently and, wanting to do a fast job and impress people, I investigated the code, noticed a bug, fixed it, then immediately gave it in for review. I had reasons for doing that, because I was new and did not know the application well. The leads would tell me if they thought I did not find the problem. But, out of the goodness of my heart, you see, I've decided to test the fix as well. And I discovered that the feature was barely implemented. It was not a bug, it was a full fuck up.

What happened here? I assumed a certain quality of the code and expected, without any reasonable evidence, to find a small typo or a logic bug that would be solved by writing a few lines of code. Instead, I had to reimplement the whole thing as a fix, I pissed off the lead devs because they had enough on their plate and made the entire team wonder what I was doing there. I mean, I haven't even tested the fix!

Doing things fast means working on valid assumptions that allow you to cut corners. In a new environment, with a team you are not familiar with and a code base you don't understand, you do not have the luxury to make assumptions. Instead, do a good job first: investigate thoroughly, see what the reported problem is, find the actual problem (which may be different), come with an attack plan, implement it, then test that it had the desired result. Yes, it takes more time than to quickly compile the logic flow in your head and hope for the best, but in some places you don't get second chances at fixing things, teams are more formal, processes need to be followed. Optimism is also based on assumptions. Be a realist instead.

In order to win a game you need to know its rules. That applies both to development process and to local politics, which sometimes are more important than the work. Once you are a good player, you can start experimenting. The title "senior developer" is not given in a vacuum, but is relevant (or not) depending on the environment. Yes, you have experience, but you don't have *this* experience yet. In my desire to be efficient and fast I didn't do anything right and I couldn't believe I have been that stupid.

Now, how about you? What are your silly mistakes that you can't believe you are still making?

# Quirks in Javascript regular expressions

I am subscribed to the StackOverflow newsletter and most of the times the "top" questions there are really simple things that gain attention from a lot of people. Today I got one question that I would have thought has an obvious answer, but it did not.

The question was what does `"asdf".replace(/.*/g,"x")` return?

And the answer to the question "What does a regular expression replace of everything with x return?" is.... [Ba da bum!] "xx".

The technical answer is there in the StackOverflow question, but I am gonna walk you through some steps to get to understand this the... dumb way.

So, let's try variations on the same theme. What does `"asdf".matchAll(/.*/g)` return? Well, first of all, in Chrome, it returns a RegExpStringIterator, which is pretty cool, because it's already using the latest Javascript features and it is returning an iterator rather than an array. But we can just use `Array.from` on it to get an array of all matches: for "asdf" and for "".

That's a pretty clear giveaway. Since the regular expression is a global one, it will get a match, then the next one until there is nothing left. First match is "asdf" as expected, the next one is "", which is the rest of the string and which also matches .* Why is it, then, that it doesn't go into a stack overflow (no pun intended) and keep turning up empty strings? Again, it's an algorithm described in an RFC and you need a doctorate in computer science to read it. Well, it's not that complicated, but I did promise a dumb explanation.

And that is that after you get a match on an index, the index is incremented. First match is found at index 0, the next one at 4. There are no matches from index 5 on.

Other variations on this theme are `"asdf".matchAll(/.?/g)`, which will return "a","s","d","f","". You can't do "asdf".matchAll(/.*/) , you get a TypeError: undefineds called with a non-global RegExp argument error that really doesn't say much, but you can do "asdf".match(/.*/g) which returns just an array of strings, rather than more complex objects. You can also do

``````var reg = /.*/g;
console.log(reg.exec("asdf"),reg.exec("asdf"),reg.exec("asdf"),reg.exec("asdf"))``````

This more classic approach will return "asdf", "", "", "" and it would continue to return empty strings ad infinitum!

But how should one write a regular expression to get what you wanted to get, a replacement of everything with x? `/.+/g` would work, but it would not match an empty string. On the other hand, when was the last time you wanted to replace empty strings with anything?