Learning from React series:

  • Part 1 - why examining React is useful even if you won't end up using it
  • Part 2 - what Facebook wanted to do with React and how to get a grasp on it
  • Part 3 - what is Reactive Programming all about?
  • Part 4 - is React functional programming?
  • Part 5 - Typescript, for better and for worse
  • Part 6 (this one) - Single Page Applications are not where they wanted to be

We cannot discuss React without talking about Single Page Applications, even if one can make a React based web site that isn't a SPA and SPAs that don't use a framework or library. What are SPAs? Let's start with what they are not.

SPAs are not parallax background, infinite scroll pages where random flashy things jump at you from the top and bottom and the sides like in a bloody ghost train ride! If you ever considered doing that, this is my personal plea for you to stop. For the love of all that is decent, don't do it!

SPAs are desktop applications for the web. They attempt to push the responsive, high precision actions, high CPU usage and fancy graphics to the client while maintaining the core essentials on the server, like security and sensitive data, while trying to assert full control over the interface and execution flow. In case connectivity fails, the cached data allows the app to work just fine offline until you reconnect or you need uncached data. And with React (or Angular and others), SPAs encapsulate UI in components, just like Windows Forms.

You know who tried (and continues to try) to make Windows Forms on the web? Microsoft. They started with ASP.Net Web Forms, which turned into ASP.Net MVC, which turned into ASP.Net Web API for a while, then turned to Blazor. At their heart, all of these are attempts to develop web applications like one would desktop applications.

And when they tried to push server side development models to the web they failed. They might succeed in the future and I wish them all the luck, but I doubt Microsoft will make it without acknowledging the need to put web technologies first and give developers full and direct access to the browser resources.

Ironically, SPAs (and modern web development in general) put web technologies first to a degree that makes them take over functionality already existing in the browser, like location management, URL handling and rendering components, but ignore server technologies.

It is relevant to make the comparison between SPAs and desktop applications because no matter how much they change browsers to accommodate this programming style, there are fundamental differences between the web and local systems.

For one, the way people have traditionally been trained to work on the web is radically different from how modern web development is taught.

Remember Progressive Enhancement? It was all about serving as much of the client facing, relevant content to the browser first, then enhancing the page with Javascript and CSS. It started from the idea that Javascript is slow and might not be enabled. Imagine that in 2021! When first visiting a page you don't want to keep the users waiting for all the fancy stuff to load before they can do anything. And SEO, even if nowadays the search engine(s?) know how to execute Javascript to get the content as a user would, still cares a lot about the first load experience.

Purely client tools like React, Angular, Vue, etc cannot help with that. All they can do is optimize the Javascript render performance and hope for the best. There are solutions cropping up: check out SSR and ReactDomServer and React Server Components. Or Astro. Or even Blazor. The takeaway here is that a little bit of server might go a long way without compromising the purity of the browser based solution.

Remember jQuery and before? The whole idea back then was to access the DOM as a singular UI store and select or make changes to any element on the entire page. Styling works the same way. Remember CSS Zen Garden? You change one global CSS file and your website looks and feels completely different. Of course, that comes with horrid things like CSS rule precedence or !important [Shudder], yet treating the page as a landscape that one can explore and change at will is a specifically browser mindset. I wasn't even considering the possibility when I was doing Windows Forms.

In React, when I was thinking of a way to add help icons to existing controls via a small script, the React gurus told me to not break encapsulation. That was "not the way". Well, great, Mandalorian! That's how you work a lot more to get to the same thing we have done for years before your way was even invented! In the end I had to work out wrapper elements that I had to manually add to each form control I wanted to enhance.

In the same app I used Material Design components, which I thought only needed a theme to change the way they look and feel, only to learn that React controls have to be individually styled and that the theme itself controls very few things. Even if there is support for theming, if you want to significantly change the visuals and behaviour you will have to create your own controls that take what they need (much more than what Material UI controls do) from the theme provider.

A local desktop application is supposed to take most of the resources that are available to it. You can talk about multitasking all you want, but normal people focus on one complex application at a time. At its core a SPA is still a browser tab, using one thread. That means even with the great performance of React, you still get only one eighth (or something, based on the number of processors) from the total computer resources. There are ways of making an application use multiple threads, but that is not baked in React either. Check out Neo.js for an attempt to do just that.

You can't go too far in the other direction either. Web user experience is opening many tabs and switching from one to the other, refreshing and closing and opening others and even closing the browser with all the tabs open or restoring an entire group of bookmarks at once. And while we are at the subject of URLs and bookmarks, you will find that making a complex SPA consistently alter the address location so that a refresh or a bookmark gets you to the same place you were in is really difficult.

A local Windows app usually has access to a lot of the native resources of the computer. A browser is designed to be sandboxed from them. Moreover, some users don't have correct settings or complete access to those settings, like in corporate environments for example. You can use the browser APIs, but you can't fully rely on them. And a browser tab is subject to firewall rules and network issues, local policies, browser extensions and ad blockers, external ad providers and so on.

You may think I am taking things to an unreasonable extreme. You will tell me that the analogy to desktop apps breaks not despite, but because of all of the reasons above and thus SPAs are something else, something more light, more reusable, webbier, with no versioning issues and instant access and bookmarkable locations. You will tell me that SPAs are just normal web sites that work better, not complex applications. I will cede this point.

However! I submit that SPAs are just SPAs because that's all they could be. They tried to replace fully fledged native apps and failed. That's why React Native exists, starting as a way to do more performant apps for mobiles and now one can write even Windows applications with it.

Single Page Applications are great. I am sure they will become better and better with time until we will forget normal HTML pages exist and that servers can render and so on. But that's going in the wrong direction. Instead of trying to emulate desktop or native apps, SPAs should embrace their webbiness.

Is Javascript rendering bad? No. In fact it's just another type of text interpreted by the browser, just like HTML would be, but we can do better.
Is Javascript URL manipulation bad? No. It's the only way to alter the address location without round trips to the server, but sometimes we need the server. Perhaps selective loading of component resources and code as needed will help.
Is single threaded execution bad? No, but we are not restricted to it.
Is component encapsulation bad? Of course not, as long as we recognize that in the end it will be rendered in a browser that doesn't care about your encapsulation.
The only thing that I am still totally against is CSS in Javascript, although I am sure I haven't seen the best use of it yet.

React is good for SPAs and SPAs are good for React, but both concepts are trying too hard to take things into a very specific direction, one that is less and less about the browser and more about desktop-like components and control of the experience. Do I hate SPAs? No. But as they are now and seeing where they are going, I can't love them either. Let's learn from them, choose the good bits and discard the chaff.  

  Learning from React series:

  • Part 1 - why examining React is useful even if you won't end up using it
  • Part 2 - what Facebook wanted to do with React and how to get a grasp on it
  • Part 3 - what is Reactive Programming all about?
  • Part 4 - is React functional programming?
  • Part 5 (this one) - Typescript, for better and for worse
  • Part 6 - Single Page Applications are not where they wanted to be

  Typescript is a programming language developed by Microsoft. It is a superset of Javascript that allows a lot of type checking and manipulation, hence the name. React and Vue fully support it while Angular requires it. So what is the reason for the adoption of this new language? What are its advantages and disadvantages?

  First of all, what is it? I would start metaphorically, if you can forgive that. Imagine a vast jungle, grown organically since time immemorial, chaotic and wild. Many developers went in, but few have come out unscathed, some never to be seen again. That's Javascript for you. It was released in 1995 as a basic scripting language for browsers, but it was designed as so flexible and complete that it could be used as a programming language in any context with minor modifications. For a very long time tightly coupled with its (very inefficient) browser implementations, it was dismissed from being a proper programming language. But that ended pretty much when V8 was launched, a performant Javascript engine that could be used separately to run the language in whatever situation the developer wanted. With V8, Chrome was launched and soon enough Node.js, which ran Javascript on the server as a proper language.

  The worst and best feature of Javascript is flexibility. You can do pretty much whatever you want in it, as it is a dynamic language unencumbered by such silly things as encapsulation, classes, types and so on. So if you started in a structured way, you could do a lot, if not - like most people unfamiliar with the language - you created a mess that no one could understand, including yourself. So if Javascript is a jungle, Typescript is Duke Nukem coming to cut the trees, wall off vast swathes of forest and only allow a narrow path for life to exist. Only, on that narrow path you get the same chaotic and wild proliferation. A lot fewer software developers traverse the forest and come out with PTSD, but a lot more people go through than before and mistakes can and will be made.

  I guess what I am trying to say is that Typescript sometimes feels like a square peg forced into a round hole. It is not a bad language. In fact, it is amazing in some parts. The type system introduced by Microsoft acts like a kind of system of annotations that inform on what you are actually doing. Tools are aware now of the types of values you use, can optimize code, find errors, warn devs, autocomplete code, help with development, etc. And I am willing to bet that people working on the language are having the time of their lives, because it has to be fun to work on abstract computer science and getting paid, too.

  But what does that mean for the frontend industry? It means that people are getting pushed on that narrow jungle path, for better or for worse. As a small business, you will have to either accept a shitty website created by cheap Javascript and vanilla HTML cavemen or get a lot out of your pocket to hire people who spend time and effort to understand Typescript and some, if not most, of the frontend frameworks that are fashionable at the moment. As a large company you will get tectonic shifts in technology, leaving a large part of your workforce in limbo, while having to spend a lot on hiring and redesigning flows. As an industry, we become dependent on several companies that spend the effort of keeping their frameworks up to date and documented. 

  Let me give you some Typescript questions (that I will not answer) to test your knowledge:

  • can you tell me what all of these types are and how they differ from each other: undefined, null, any, unknown, never, void ?
  • how can you tell if a Typescript object is of a specific form (the equivalent of the .NET 'is' or 'as' functionality)?
  • what is the difference between a union of literal types and an enum?
  • what are and how can you use BigInt, ReadOnlyArray, Partial, NonNullable, Required?
  • what is the difference between a private member of a Typescript class and one starting with #?
  • do you know how to use unions in interpolated strings?
  • what is the difference between interface, type, class, type intersection, class expression and module?

 I could go on and on. On how the possibility of null is now something you have to declare explicitly, for example. I didn't (dare to) ask about type guards and how narrowing works and what conditional types are. And there are so many gotchas for developers coming from other languages, because the language features have been added by people who worked on C#, so they are kind of the same, but actually not. Type meaning and conversion is a large bit of confusing difference between Typescript and C#/Java. For example you can define a class and then cast some data to it, but you don't get what you expect:

class MyClass {
  public name: string='';
  public sayHello() { console.log(`Hello ${this.name}`); }
}

const my:MyClass = { name: 'Siderite' } as MyClass;
console.log(my); // { "name": "Siderite" }
console.log(typeof(my)); // "object"
console.log(my instanceof MyClass) // false
console.log(my.sayHello()); // ERR: my.sayHello is not a function 

There are still web sites dedicated to the inconsistencies of Javascript. Typescript doesn't solve these issues, it mostly hides them. I am sure it's fun to play with types, but is that the optimal solution for the problem at hand, mainly the many ways you can do Javascript wrong? I would argue no. It's fun to work in, but there is a clear dependency between Typescript and Javascript, which forced so many changes in Typescript from Javascript and the other way around, as they have to be kept in sync. All while Javascript needs to remain backwards compatible, too.

"But what about React? Weren't you talking about that, Siderite?"

Yes, I was. I only looked deeper into Typescript because I did this project in React. Before, I had used it with Angular and frankly I didn't feel the friction that I felt now. Angular is designed with Typescript in mind, the development experience is smoother. Angular also uses two directional bindings to propagate changes and that means less Typescript code. The only code you actually need to write is network API code, for which you have out of the box HTTP services, and some limited interface logic. React doesn't do that.

First of all, React has been designed within a kind of declarative/functional mindset, as I explained in previous chapters of this series. It focuses a lot on immutability and functions that are passed around and declaring what your expectations are. Typescript is fundamentally an imperative language. After forcing it through the round hole, the square peg now has to go through a triangular hole, too. The immutability forces one to use a lot of code for changes coming from the UI towards the Typescript logic.

Then, React is a library. It was designed as such and has less levers to force the developer in a direction or another. Even when following a clear development strategy, there are many of which to choose from, all tried and tested and valid, but very different from one another. The jungle was tamed, but now you must consider a multiverse of jungles, each with a different shape.

Finally, React started out in Javascript. Many documentation pages are still just about Javascript. New innovations in the field of React are developed and tested out independently, by various people with various skills and motivations. The learning curve is not steep, but the paths are many.

So in the end, Typescript is an interesting experiment in programming languages, one that will probably surprise me in the near future with ideas that can only be implemented using it. However it is not perfect and its dependency on Javascript is unfortunate, even if its inspiration was not. The purpose of the language was to guide and help developers mired in Javascript confusion, but using it with React goes against that very purpose, as React is still something relatively new and evolving wildly in all directions, so React doesn't help Typescript. Does Typescript help React? I would say yes. However I don't feel that it is enough in its current form. The friction between the two concepts is palpable and I dare any of you to prove me wrong.

It seems I've talked a lot about the problems of React rather than its benefits. I blamed it on things ranging from confusing and obsolete documentation to inconsistent goals of the library and underlying programming language. That's the way I work, focusing on problems so I can find one I can fix. In the next chapter I want to discuss about React in the wild and what are the good things people are saying about it. The most interesting question however, the one that I want to answer with this entire series, is how can we improve our work by adapting lessons learned, either from React to whatever we do or the other way around. What concrete ideas should we adopt from React and which we should condemn to the pit of failed concepts?

  Learning from React series:

  • Part 1 - why examining React is useful even if you won't end up using it
  • Part 2 - what Facebook wanted to do with React and how to get a grasp on it
  • Part 3 - what is Reactive Programming all about?
  • Part 4 (this one) - is React functional programming?
  • Part 5 - Typescript, for better and for worse
  • Part 6 - Single Page Applications are not where they wanted to be

  React was designed just when classes and modules were making their way into Javascript, so it made sense to use them. Developers that are not coming from the Javascript or dynamic languages world are used to the type safety and hierarchical structure that classes provide. And it also made sense from the standpoint of the product. If you want to encapsulate state, logic and presentation why not used existing functioning models like classes, components and so on.

  However, at the same time ideas like functions being first class citizens of programming languages and functional programming were making a comeback, mostly because of big data. That meant that lambdas (arrow functions) were popping up everywhere. If you are a C# developer, you already are familiar with them. Something like Func<int,int> func = (int x)=> x*2; represents a lambda function, which is the same as something written like private int f2(int x) { return x*2; }, yet lambda functions can be declared inside code blocks, can be implicitly cast to Expressions and manipulated and they are brilliant as method parameters. Check out the lambda version in C# compared to the function version in VB:

// C#
var items = allItems.Where(i=>!i.deleted);
// C# function body
var items = allItems.Where(i=>{
                             return !i.deleted
                           });
// VB
Dim items = allItems.Where(Function(i) Not i.deleted)
// VB function body
Dim items = allItems.Where(Function(i) 
			      Return Not i.deleted
			   End Function)

 Similarly, Javascript had only function syntax, even if functions were designed to be first class citizens of the language since its inception. Enter arrow functions in Javascript:

// before
var self = this;
var items = allItems.filter(function(i) {
  return self.validate(i);
});

// after
var items = allItems.filter(i=>this.validate(i));

Note how arrow functions don't have an internal 'this' so you don't need to bind functions or create self variables.

So at this point, React changed and instead of classes, they implemented "functional syntax" in React Hooks. Behind the scenes a component is still generated as a class which React uses and the old syntax is still valid. For example at this time there is no way to create an error boundary component using functional syntax. The result is a very nice simplification of the code:

// React classic (pardon the pun)
export class ShowCount extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      count: 0
    };
  }
  componentDidMount() {
    this.setState({
      count: this.props.count
    })
  }

  render() {
    return (
      <div> 
        <h1> Count : {this.state.count} </h1>
      </div>
    );
  }
}

// React Hooks
export function ShowCount(props) {
  const [count, setCount] = useState();

  useEffect(() => {
    setCount(props.count);
  }, [props.count]);

  return (
    <div>
      <h1> Count : {count} </h1>
    </div>
  );
}

// courtesy of https://blog.bitsrc.io/6-reasons-to-use-react-hooks-instead-of-classes-7e3ee745fe04

  But this does not just provide a better syntax, it also changes the way development is done. Inheritance is basically eliminated in favor of composition and people are starting to use the word "functional" in sentences uttered in the real world. And while the overall design of React to use unidirectional binding and immutable variables was there since inception, I do feel like this is just one more step towards a functional programming approach and the reason for so many functional purists popping up lately.

  What is functional programming, though? Wikipedia defines it as "a declarative programming paradigm in which function definitions are trees of expressions that map values to other values, rather than a sequence of imperative statements which update the running state of the program." Sound familiar?

  I will have you know that I have friends that have rebelled and gone to the other side, making applications (including UI) with F# and refusing to submit to the Galactic Imperative. After playing with React I can say that I understand why this approach has appeal. One declares what they need, ignore flow and constrain their efforts inside components that are more or less independent. A program looks and feels like a big function that uses other functions and to which you just provide inputs and out comes UI ready to use. If the same input is provided, the same output results. You can test it to perfection, you can infer what happens with an entire tree of such functions and make optimizations in the transpiler without changing the code. You can even use a diff algorithm on the output tree and just update what changed in the UI.

  But it is time to call bullshit. We've used functions that receive pure data on one side and output user interface on the other side since forever. They are called views. One could even argue that an API is a data provider and the application is the function that uses the data to output UI. You don't ignore flow, you move it up! You will still have to model the interactions between every piece of data you have and all the events that come in. One might even say the unforgivable and assert that React is just another Model-View thingie with the extra constraint that it will forcibly re-render a component when its input state changes.

  That is my main takeaway from React: the idea that forcing re-rendering of components forces the developer to move the state up, closer to where it should be. No one can store stuff in browser variables, in element attributes and data, because all of it will be lost on next render. That is good news, but also very bad news. Let me get you through an example:

  We have data that we need shown in a grid. Every row has an expand/collapse button that will show another grid under it, with details related to that row. The React way of doing things would take us through these steps:

  • create a component that represents the grid and receives an array as input
  • it will contain code that maps the array to a list of row components which receive each row as the input
  • the row component will render a button that will dispatch an expand event for the row when clicked
  • on click the row expanded state will be changed and the data for the row detail grid retrieved

  It sounds great, right? OK, where do you store the state of row expansion? How do we push it to the row component? Let's use a map/dictionary of row id and boolean, why don't we? Does that mean that when you expand/collapse a row only the boolean changes or the entire structure? What will get re-rendered? The row component in question or all the row components?

  What happens when we go to the next page in the grid and then go back? Should we return to the same row expansion states? Where should the scrollbar in the grid be? Should we keep that in the state as well and how do we push it to the grid component? Do row details grids have scroll? Doesn't the size of each component affect the scroll size, so how do we store the scroll position? What is the user resizes the browser or zooms in or out?

  What happens when we resize a grid column? Doesn't that mean that all row components need to be re-rendered? If yes, why? If no, why? What if you resize the column of a detail grid? Should all detail grids have the same resizing applied? How do you control which does what?

  Many grids I've seen are trying to store the expansion, the details, everything in the object sent as a parameter to the row. This seems reasonable until you realize that adding anything to the object changes it, so it should trigger a re-render. And then there is Typescript, which expects an object to keep to its type or else you need to do strange casts from something you know to "unknown", something that could be anything. That's another story, though.

  Suddenly, the encapsulation of components doesn't sound so great anymore. You have to keep count of everything, everywhere, and this data cannot be stored inside the component, but outside. Oh, yes, the component does take care of its own state, but you lose it when you change the input data. In fact, you don't have encapsulation in components, but in pairs of data (what React traditionally calls props) and component. And the props must change otherwise you have a useless component, therefore the data is not really immutable and the façade of functional programming collapses.

  There are ways of controlling when a component should update, but this is not a React tutorial, only a lessons learned blog post. Every complexity of interaction that you have ever had in a previous programming model is still there, only pushed up, where one can only hope it is completely decoupled from UI, to which you add every quirk and complexity coming from React itself. And did we really decouple UI or did we break it into pieces, moving the simplest and less relevant one out and keeping the messy and complex one that gave us headaches in the first place in? It feels to me like React is actually abstracting the browser from you, rather than decoupling it and letting the developer keep control of it.

  After just a month working in this field I cannot tell you that I understood everything and have all the answers, but my impression as of now is that React brings very interesting ideas to the table, yet there is still a lot of work to be done to refine them and maybe turn them into something else.

  Next time I will write about Typescript and how it helps (and hinders) React and maybe even Angular development. See you there!

 Learning from React series:

  • Part 1 - why examining React is useful even if you won't end up using it
  • Part 2 - what Facebook wanted to do with React and how to get a grasp on it
  • Part 3 (this one) - what is Reactive Programming all about?
  • Part 4 - is React functional programming?
  • Part 5 - Typescript, for better and for worse
  • Part 6 - Single Page Applications are not where they wanted to be

The name React is already declaring that it is used in reactive programming, but what is that? Wikipedia is defining it as "a declarative programming paradigm concerned with data streams and the propagation of change". It expands on that to say that it declares the relationship between elements and updates them when either change. You can easily imagine a graph of elements magically updating as any of them changes. However, the implementation details of that magic matter.

  In 2011 Microsoft revealed a free .Net library called Reactive Extensions, or ReactiveX or RX. It was based on a very interesting observation that the observer/observable patterns are the mirror images of iterator/iterable. When the iterator moves through an iterable, the observer reacts to events in the observable; one is imperative, the other reactive. The library was so popular that it was immediately adopted for a bunch of programming languages, including Javascript. It also allowed for operations traditionally used for arrays and collections to work with a similar syntax on observables. This is a great example of reactive programming because instead of deciding when to perform a data access (and having to check if it is possible and everything is in range and so on), the code would just wait for something to happen, for an event that provided data, then act on the data.

  One might argue that Verilog, a hardware description language, is also reactive, as it is based on actions being performed on certain events and it even uses non-blocking assignments, which are like declarations of state change which happen at the same time. Reminds me of the way React is implementing state management.

  Of course, reactive programming is also modern UI and when I say modern, I mean everything in the last twenty years. Code gets executed when elements in the user interface change state: on click, on change, on mouse move, on key press etc. That is why, the developers at Facebook argue, browser based UI programming should be reactive at the core. This is not new, it's something you might even be already very familiar with in other contexts. Code that is triggered by events is also called event-driven programming.

  But at the same time, others also claim their software is reactive. Microservices are now very fashionable. The concept revolves around organizing your product into completely independent modules that only have one external responsibility, which then one wires together via some sort of orchestrator. The biggest advantage of this is obviously separation of concerns, a classic divide and conquer strategy generalized over all software, but also the fact that you can independently test and deploy each microservice. You don't even have to shut down running ones or you can start multiple instances, perhaps with multiple versions and in different locations. This is also distributed programming. The way the communication between microservices is done is usually via some sort of message queue, like Rabbit MQ, but I am working on a really old software, written like 15 years ago, which uses IBM MQ to communicate between different portions of the software - let's call them macroservices :) Well, this is supposed to be reactive programming, too, because the microservices are reacting to the messages arriving on the queue and/or sent from others.

  The observer pattern is old, it's one of the patterns in the original design patterns book Design Patterns: Elements of Reusable Object-Oriented Software, which started the software design pattern craze which rages on even now. Anybody who ever used it extensively in their software can (and many do) claim that they did reactive programming. Then there is something called the actor model (which will probably confuse your Google if you search for it), which is actually a mathematical concept and originated in 1973! Implementations of actors are eerily similar to the microservices concept from above.

  And speaking of events, there is another pattern that is focusing on declaring the flow of changes from a given state, given an event. It's called a state machine. It also boasts separation of concerns because you only care about what happens in any state in case of an event. You can visualize all the possible flows in a state machine, too, as names arrows from any state to another, given that such a transition is defined. The implementation of the state machine engine is irrelevant as long as it enables these state transitions as defined by the developer. 

  Everything above, and probably some other concepts that are named differently but kind of mean the same thing, is reactive programming. Let me give you another example: a method or a software function. Can one say it is reactive? After all, it only executes code when you call it! Couldn't we say that the method reacts to an event that contains the parameters the method needs? What about Javascript, which is designed to be single threaded and where each piece of code is executed based on a queue of operations? Isn't it a reactive programming language using an event bus to determine which actions to perform?

  And that's the rub. The concept of reactivity is subjective and generally irrelevant. The only thing that changes and matters is the implementation of the event transport mechanism and the handling of state.

  In a traditional imperative program we take for granted that the execution of methods will be at the moment of the call and that all methods on that thread will be executed one after the other and that setting a value in memory is atomic and can be read immediately after by any other piece of code and you can even lock that value so it is only read by one entity at a time. Now imagine that you are writing the same program, only we can't make the assumptions above. Calling methods can result in their code getting executed at an arbitrary time or maybe not at all. Whatever you change in a method is only available to that method and there is no way for another method to read the values from another. The result? Your code will take a lot of care to maintain state locally and will start to look more like a state machine, modelling transitions rather than synchronous flows. The order of operations will also be ensured by consuming and emitting the right sort of events. Permanent and/or shared storage will become the responsibility of some of the modules and the idea of "setting data" will become awkward. Keeping these modules in sync will become the greatest hurdle.

  That's all it is! By eliminating assumptions about how your code is executed, the result is something more robust, more generic, more compartmentalized. Is it the golden hammer that will solve all problems? Of course it isn't. We've seen how the concepts at the core of reactive programming have been there since forever. If that was the best way, everybody would already be working like that. The biggest problems of this kind of thinking are resource duplication, as everybody has to keep all the data they use locally, and synchronization, as one cannot assume there exists any source of absolute truth that can be accessed by all at the same time. Debugging the system also becomes a bit complicated.

  In 2014 a bunch of people created something called "The Reactive Manifesto" and anyone can sign it. In it, they define a reactive system as:

  • Responsive - responds quickly
  • Resilient - remains responsive in case of failure
  • Elastic - remains responsive regardless of workload
  • Message Driven - async message passing as the boundary between components

  As you can see, reactive mostly means responsive for them and the concept is more one of organization management than a specific set of programming languages and tools. Note that, in fact, you can create a system that communicates via async message passing between components on the client, on the server, between client and server and so on. There can be multiple types of technology and different stacks. As long as they can react to asynchronous messages, they can be integrated into such a system. 

  This post has reached already a big size and I haven't even touched on functional programming and how it tries to solve... well, everything. I will do that in the next chapter. I have to say that I find the concept of a programming language that eliminates global variable scope and public fields and introduces a delay and a random order of execution of methods or properties from other classes fascinating. Imagine testing and debugging that, then moving the working code to production, where the delay is removed. You will also see that a lot of the ideas above influence how React development is done and perhaps you will understand purists telling everybody how things are not correct until you implement this or that in a certain way. Till next time!

Learning from React series:

  • Part 1 - why examining React is useful even if you won't end up using it
  • Part 2 (this one) - what Facebook wanted to do with React and how to get a grasp on it
  • Part 3 - what is Reactive Programming all about?
  • Part 4 - is React functional programming?
  • Part 5 - Typescript, for better and for worse
  • Part 6 - Single Page Applications are not where they wanted to be

In order to understand React we must consider what were the advantages that Facebook found in the concept when they created the library. There a numerous presentations and articles that they pushed to explain it, but it kind of distills to this:

  • mutation of values complicates flows in complex applications
  • traditional Model-View patterns promote mutation through two directional data binding
  • solution for mutation:
    • use unidirectional data binding and re-render views as the data changes
    • send inputs as events that a higher level entity will interpret
  • solution for slow browser render overhead:
    • code is organized in smaller and smaller components that depend on a few values in the overall data
    • there is an intermediate layer of rendering between React and the actual browser DOM called a Virtual DOM, which only sends to the browser the changes that renders affected

But wait, you will say! How do the values change if they are immutable? The point here is that your components have an immutable internal state. The values that are passed to the components can still change, but on a higher level. They declare a label that have a text and for the label the text never changes. When the text changes in the logic of the application, a new label instance with a new text is rendered. The advantage of this is that the declaration of the render for a component defines the way the component will always render. It's declarative of your intent of how that component should look. They wanted to replace mutation with what they called reconciliation between the previous state and the current state. They almost treated web development like game development or remote access software.

So when you look at a JSX syntax and you see Javascript mixed with HTML and switching back and forth, that is different from the ASPX spaghetti code where server logic was mixed up with UI declarations. JSX is less an XML flavor and more a Javascript flavor. In fact, it is transpiled to Javascript code when executed and the changes to the UI are imperative commands that tell it how to change based on how the render code affected the component. But for the developer it's just a story of how the component should look given some data.

React was released in May 2013 and it went through some transformations on the way. Internally it probably started as an object oriented approach which was cumbersome and fought with the overall design of Javascript. They fixed that by the time of the release by using the JSX syntax, but still with code looking like

var component = React.createClass({ render: function() { return <something/>; } });

And further down the line they came up with React hooks, which move further from the class oriented approach and more towards a functional one with stuff like

const component = (props) => (<something />);

which of course is lighter still. Note that Javascript changed during that time, allowing for more stuff to happen. And now they added Typescript support, so you get something like

const Component = (props:{text:string})=>(<something />);

This evolution of the library is also one of the biggest obstacles against learning how to use it as in your Goggle search you might find solutions and problems from any time in the history of the library. You will find the perfect tool for what you wanted, but all the examples are in Javascript and in Typescript it works differently or the answers refer to previous versions of absolutely everything and you don't know which of them is the one that should apply to your task. Or even worse, some package that you use and it made your life so easy conflicts with the one you just want to use because they were never meant to work together.

As opposed to Angular, which was designed as a framework, to constrain and control what you do, React is a library. And it evolves not only through the efforts of Facebook, but also of a very active community. People would add React to their existing project rather than rewriting it from scratch. This means they will have used any of the various versions of React as well as any of the various versions of all the packages that they use in conjunction with it. And these packages are wild! Imagine a developer coming from the world of WPF or vanilla web design with HTML and CSS as the primary focus and Javascript an after thought or from Angular. They will work with React, but use it in a way they are more familiar with. That means that even if the library has a backing philosophy, there is little it can do to force you to use it in that way.

Let's take just one example: MobX, a package that takes existing data objects and wraps them in proxies that notify of changes. With React you use it as you would a ViewModel in WPF. Instead of sending an event called CLICK from a click event and handling it in some state machine looking code you can just modify the "immutable" property of the data you were sent and behind the scene a setter is executed which lets everybody know that it has changed and so it triggers a re-render which will take the value of the actual property from the behind-the-scene getter. You get your way, React kind of gets its way.

To recap:

  • components live in a tree, rather than a graph, and they all render based on the properties they are passed
  • components declare how they should look like
  • any input from the components is sent up as an event to which the system "reacts"
  • slow browser rendering is replaced with a virtual representation of the UI and updated with specific commands to reflect only the changes from one state to the other
  • React is an evolving beast with many heads and finding your own way of doing things lies at the root of any successful development effort

While exploring development with this library, developers have found various tools and concepts useful to integrate. As we usually do, they started to generalize these attempts and come up with some new principles or get to the root of the underlying flow. I want to spend the next part of the series discussing some of these more general or architectural ideas and try to see what they are worth.

Learning from React series:

  • Part 1 (this one) - why examining React is useful even if you won't end up using it
  • Part 2 - what Facebook wanted to do with React and how to get a grasp on it
  • Part 3 - what is Reactive Programming all about?
  • Part 4 - is React functional programming?
  • Part 5 - Typescript, for better and for worse
  • Part 6 - Single Page Applications are not where they wanted to be

  A billion years ago, Microsoft was trying to push a web development model that simulated Windows Forms development: ASP.Net Web Forms. It featured several ideas:

  • component based design (input fields were a component, you could bundle two together into another component, the page was a component, etc)
  • each component was rendering itself
  • components were defined using both HTML-like language, Javascript, CSS and server .Net code bundled together, sometimes in the same file
  • the rendering of the component was done on the server side and pushed to the client
  • data changes or queries came from the client to the server via event messages
  • partial rendering was possible using UpdatePanels, which were a wrapper over ajax calls that called for partial content
    • at the time many juniors were putting the entire page into an UpdatePanel and said they were doing AJAX while senior devs smugly told them how bad that was and that it shouldn't be done. I agreed with the senior devs, but I really disliked their uninformed condescending attitude, so I created a method of diffing the content sent previously and the new content and sending only the difference. This minimized the amount of data sent via the network about a hundred times. 

  Sound familiar? Because for me, learning React made me think of that almost immediately. React features:

  • component based design
  • each component renders itself
  • components are defined by bundling together HTML, Javascript/Typescript and CSS
  • the rendering of the component is done in the virtual DOM and pushed to the actual browser DOM
  • data changes or queries are sent from the browser to the React code via event messages
  • partial rendering is built in the system, by diffing the existing render tree with a newly generated one from data changes

  At first look, an old guy like me would say "been there, done that. It's bad design and it will soon go away". But I was also motivated enough at the time of ASP.Net Forms to look into it, even under the hood, and understand the thing. To say it was badly designed would be stupid. It worked for many years and powered (and still does) thousands of big applications. The reason why Forms failed is because better ideas came along, not because it was a bad idea when it was created.

  Let's look a little bit on what made Forms become obsolete: the MVC pattern, more specifically implemented by Ruby developers and taking the world by storm and ending up being adopted by Microsoft, too. But Model View Controller wasn't a new pattern, it has been used forever on desktop applications, so why was it such a blow to Forms? It was a lot of fashion elitism, but also that MVC molded itself better on web applications:

  • a clear separation of concerns: data, logic and display
  • the ability to push the display more towards the client, which was new but becoming increasingly easy in browsers
  • a clear separation of programming languages: HTML in html files, Javascript in .js files, server code in .cs files
  • full (and simple) control over how things were rendered, displayed and sent to the server

  Yet in the case of React, things are going in the opposite direction, going from MVC applications with clearly separated language files to these .jsx or .tsx files that contain javascript, html and even css in the same file, mixed together into components. Is that bad? Not really. It kind of looks bad, but the entire work is done on the client. There is no expensive interface between a server and a browser that would make the model, otherwise used successfully for decades in desktop applications, ineffective. It is basically Windows Forms in the browser, with some new ideas thrown in. As for the combination of languages in a single syntax, it can be abused, just like any technology, but it can also be done right: with state, logic and UI represented by different files and areas of the application. Yes, you need script to hide or show something based on data, but that script is part of the UI and different from the script used to represent logic. Just the language is the same. 

  "Is Siderite joining the React camp, then? Switching sides, going frontend and doing nasty magic with Javascript and designing web pages?" people will ask. A reasonable question, considering most of my close friends still think React is for people who can't code or too young to remember the .aspx hell. The answer is no! Yet just as with the UpdatePanel seniors that couldn't stop for a second to look a bit deeper into an idea and see how it could be done and then cruelly dismissing curious juniors, thinking that React can't teach you anything is just dumb.

  In this series I will explore not only React ideas, but also the underlying principles. React is just an example of reactive programming, which has been in use, even if less popular, for decades. It is now making a comeback because of microservices, another fashionable fad that has had implementations since 1990, but no one gave them the time of day. Ideas of data immutability are coming from functional programming, which is also making a comeback as it works great with big data. So why not try this thing out, iron out the kinks and learn what they did right?

  Many people, including myself, automatically think of "the client is always right" when talking capitalism, but that's not correct. In fact, capitalism is defined as "an economic and political system in which a country's trade and industry are controlled by private owners for profit, rather than by the state"; profit is the only goal or driver of the system, even for different systems, which might be controlled by the state, for example. Every social or legislative characteristic of a particular political system is just a patch that tries to fix from outside what is obviously a ridiculously simplistic concept.

  Even so, we still cling to the idea that we are "served" by companies, like we are some sort of aristocrats being catered for by legions of servants. There is a logic to that, as when a company stops catering to their clientele, they are supposed to lose it. But that's just an illusion. In fact, this only happens if some very specific conditions are met:

  1. the client is aware of the bad service, meaning:
    • they know what they're supposed to get
    • they know what they are getting
  2. there is an alternative to the service, meaning:
    • there is at least another company able to meet the requirements
    • the company is accessible to the client
      • the cost of access is reasonable
      • the client is allowed to access the competitor
    • the client knows the competitor exists
  3. the client wants to get better service, meaning:
    • there isn't too small of a difference between various services (subjective perception that all companies are the same)
    • the effort of changing the service is not too large (subjective perception of effort)
    • the emotions of the client do not bind them to the company (subjective perception of the company)

  Now, all of the points above require not only effort, but persistent effort. One needs to get the necessary knowledge and then keeping up to date, avoiding misdirection and obfuscation, then make decisions and then take action. But even so...

  A company is usually a client of other companies. In my job I am usually hired by companies who do management of people for other companies that need software, with any number of intermediaries and internal chains of command inside each. One might think that because the end client is paying for the entire chain, we should all care about what they want. But we do not! And here are a few of the reasons:

  1. the client does NOT know what they want, therefore they will never be aware of what they're supposed to get
  2. the client has spent considerable effort in creating a relationship between them and the company that provides them with the software, therefore the effort of changing to another provider is quite forbidding
  3. the client has spent considerable effort in creating a relationship with just one company, because they don't want to handle ANY of the responsibilities that company is supposed to handle, therefore they are not in contact with competitors
  4. the chain of command inside the client is made up of people who have personal connections with the company in question, so changing to another provider would be in their detriment, therefore the client is not allowed to change to another provider
  5. the company itself is not providing a better service because it doesn't have to, for the same profit; their competitors think alike, so there is absolutely no reason to change
  6. there is an intermediary system to measure productivity: instead of getting paid for value, the company gets paid for hours worked, for example
  7. a smaller nimbler company might rise to provide the same service for less money or a better service for the same amount, but there is point 4. as well as the perception that smaller companies are riskier
  8. a company might want to change, but be unable to because it depends on other companies or it lacks the necessary competency
  9. one might be competent inside of a company, but if the company is big enough, they get promoted to other posts until they reach a position were they are not competent enough to be promoted
  10. clients themselves prefer to cut costs than get better service

  This happens at every level from the top - where you feel you might be, to the bottom - where you actually are. And guess what? When you realize the same effort and care that you expect from others should be coming from you, you quickly find reasons not to provide any of them:

  1. my job is boring, why should I do better?
  2. my boss is an ass, why should I do better?
  3. my clients are idiots, why should I do better?
  4. I never meet my clients, that someone else's job, why should I do better?
  5. whenever I tried to change something, someone shut me down because they are better positioned than me, why should I do better?
  6. I am getting paid by the hour, so working faster means less money, why should I do better?
  7. most of the money from the client remains with the entire chain of people over me and I get the scraps, why should I do better?
  8. my job is not important for me, it's just a means to support my actual life, why should I do better?

  This may appear as a pyramid of interests where you are just a cog in the machine or whatever, but it is not, it's a full circle. You get what you give. The client and the company and the least competent employee is always you. There are no other species of creatures that fill any of these positions. Your boss is human, your employee is human, your client is human. In Romania there is the saying that you're stealing your own hat.

  But it's OK. You can't do any better, so why should you do better? You are only human, so why be more human? You are a tiny cog in the machine so why grow larger? Do whatever you want, I don't care.

  When I was a high school kid it was fashionable among the street thugs of Bucharest to use the insult "slave". I don't believe this was related to the history of slavery of the gipsy people in Romania's past, which was the ancestry of many influential such thugs, as it was something borrowed from the Americans, where that issue is much more aggravating. If I am correct, then the very use of the term denotes the way Romanians relate to other people, especially those who they perceive superior. Of course, if I am wrong, then I am guilty of the same thing, so QED, I guess?

  After the Romanian Revolution against Communism (note the big R we use for that event, being the only real change we ever affected as a people) people from different countries came to assess the opportunities presented by a newly opened territory. One report that I saw with my own eyes was from a Jewish lawyer who said just that: Romania is filled with highly educated people who distrust their own government, laws and look poorly on local products and people. Instead, they revere national brands they never actually had any real recent contact with like the U.S. and West Germany, preferring ideas and things imported from there to things they could get or make locally. He concluded that it was a good place to invest in, since the quality of the local human resource was high and their expectations were low.

  The Revolution was in 1989, 32 years ago, but the mentality is still mostly there, as even the people who teach the children of today are still of the generation that lived through that era. It is funny to discuss education with a Romanian, they all complain about how bad it is related to how it was, because children aren't fed the same amount of unprocessed information that was the mainstay of the Communist education. They are rarely complaining about the lack of technological advancement in schools or of skills that are useful in real life or about how children are not taught how to be passionate and happy. Even so, they don't do anything at all to change anything. The only measure of control that parents have is to which school to pay people for their kids to get into and, more recently, to which expensive private school to send their kids in order to give them what they see as the proper education.

  The heroes of the Romanians are not the successful entrepreneurs. The media rarely mentions them and then it is mostly because they were paid for by those people. In case an average Romanian hears about a successful Romanian businessman, it is assumed they had connections with the people who ruled the country during the Communist era. Since they started with money and/or influence, their success is surely undeserved. Funny enough, people like doctors are not heroes either, because in Romania people usually have to bribe medical personnel to get any of the treatments that are theoretically free. Even when they go to private clinics the instinct is to pay extra to appease the doctors who, in their natural state, would try to kill them. This is, of course, caused by a systemic underfunding of medicine and by the endemic corruption which sees any funds misused or embezzled. Because of this, a medic who saves lives - even when they refuse to accept any extra money, is not a hero, but just incompetent or parasitic. Heroes are not the people who left the country to get abused as cheap labor abroad either. People who went to Spain to pick strawberries or to Greece to pick olives are just poor uneducated people who are clearly poor stock, perhaps even gipsy people. They deserve no respect. Same for prostitutes, who are universally despised as shaming the country and at the same time praised for being amongst the most beautiful of prostitutes.

  There was once a news report about people going to pick strawberries on TV and the mother of a girl going to Spain explained how she taught her daughter to conform to whatever they say there, to not antagonize the boss. Just work hard and make money and do whatever he says. This was her mother!

  Police people are not heroic, either. They don't protect the citizens, they enforce unreasonable rules. Politicians are not heroes, they are either the absolute evil or the person who is opposing the absolute evil, which makes them slightly less evil. The few situations where political fervor is so high that someone becomes close to being considered heroic, their efforts pale in comparison with the expectations put on them, which of course proves they were evil all along.

  No, the heroes of the Romanians are the hard working people who work for other people. Great theoreticians, the army of Romanian software developers that left the country to work for multinational corporations, any doctor who would be despised in the country is a hero when working abroad, engineers of all kinds, any white collar job, even people working in construction - which is somehow seen as a clean job. With the caveat that they must not be gipsy or poor stock, they surely left the country to thieve and steal and they are shameful to us all. If people "from the developed countries" (this is a phrase still very much in use in relation to the Western countries) praise a hard working Romanian, the entire country breaths a collective awwh of pride, like dogs petted on their heads for being good boys. "You see?", they say, "Now they see what we are made of!".

  One obvious exception is any managerial job. If you are a manager, you actually do little to nothing, you are an oppressor, not a hard working individual. Even if you got there through your own efforts (and didn't rely on the people you knew or money you had) you have a dirty job that deserves no respect. Unless you are related to the person who judges the situation, in which case you are the pride of the family. 

  The biggest heroes are of course great athletes, especially if they are part of an international team. If a Romanian is part of a of German football team, the entire country believes he would be the reason why Germans are good at football. Of course, people doing great things in Romanian sports are usually seen as part of the endemic corruption in Romanian sports. They still get the status of heroes, but they are always on trial and any mistake will be fatal to their reputation. An interesting exception is Simona Halep, which is the greatest Romanian hero of all, because she goes to international Tennis competitions and wins some of them against the best international tennis players in the world! The fact that she is reasonably good looking doesn't hurt either. She is of the people for the people.

  There are great Romanian writers, for example, but we only value the dead ones. We learn in school about great Eminescu, or Rebreanu, or Blaga. The more poetic, the better. The more obtuse the better. And in order to verify if children really read these big greats, we ask them to write commentaries. And kids just read existing commentaries and summarize them. With the advent of Google and AI, it's impossible to not automate this, since it requires the lowest level of human inventivity. Current writers are poor unknown quantities, always shorted when negotiating with local publishers. And no one reads books anymore anyway, what are they doing? There are great Romanian actors and filmmakers, but few people actually bother to watch Romanian productions. We occasionally see films winning some international prize somewhere and it's usually some drab and depressing drama about people being treated as slaves and behaving like one.

  All these point to a mentality that can only be called "slave mentality". Anything Romanian is poor quality, therefore we export raw material and import the very things made from it. As an example, we export a lot of apples and import a lot of apple juice. The opportunities for Romanians are to either find a good job in Romania or - much better - a good job abroad. To have a company, make your own money, employ other people, is still seen as something dirty. Romanians are not educated on how to make, keep or invest money, instead they are instructed on which jobs are better paid. When I was a child, I knew that I should go for being a medic or a lawyer. Now kids are probably taught to try to go into software. No parent would ever tell a kid to think freely, pursue their dreams and try to make something of themselves through their own strength unless it first starts with being employed somewhere. Surely, dreams can wait, they say, all sad and depressed.

  Funny enough, people who somehow get to be business owners, employers or managers many times behave like slave owners. This is also slave mentality: slaves don't dream of being free, but of becoming masters themselves. This sadly also encourages Romanian employees to feel that anything Romanian sucks. In my career the most unprofessional, choleric, petty and unethical employers were Romanians. With small exceptions that I will attribute to mental illness and not national culture.

  Even when we go on holidays, the biggest complaint we have is "it was nice, but there were too many Romanians there". When we were most complaining about how poor our country was, right after the big R, we were the country importing most expensive cars and high end smartphones.

  This is a sad sad cycle, Romania is and will continue to be a nation of employees. Nothing in our culture impresses the importance of self actualization, of generating and defending our own values, of pushing to get ahead and then actually pulling people with you. Nothing teaches us to band together under those values and fight for them. From all of the cultures we've had contact with, which mostly invaded our country and enslaved us for most of our history, we only learned to slave or enslave others.

  We had and we still have a lot of potential, smart people, hard working people, but is that relevant without a set of values? I personally feel that the Romanians abroad impress their employers, but also their employees or their subordinates and their friends, not by being hard working, but by being open and by not being complete assholes, by being kind and ethical. It hurts me to see how American we have become, partially because of the malevolent influence of Russia, but instead of going for the American ideal declarations of freedom and equality, we emulate the American actions of selfish individuality and status driven inequality. They also pushed this idea that somehow you are either a Capitalist or a Communist, master or slave. This just in case you are not a Fascist or a Terrorist, which makes you gipsy or poor stock.

  Let's abandon this obsolete thinking of masters and slaves and instead think of the others as simply people. Your employer is your partner not your owner, people from other nations are still just people, dreaming is still OK and when you reach the top, give yourself a pat on the back and help others up. Find something you care about and do it. There is nothing on the other end. Every moment is not the first day of your life, but the very last. Make it worthwhile.

  Yesterday I saw one of those things that make me exclaim "Only in Romania!". There was a pharmacy and in front of it there was an old lady waiting her turn to enter, because there can't be too many people inside. Next to her, like a few feet away, there was a maskless man puffing away on a cigarette, his smoky exhalations reaching the masked old woman as she was trying to protect herself and the people in the pharmacy.

  And it made me more attentive to people around me. Bands of maskless high schoolers, with the loud talking, fake swagger and hysterical laughter typical of their age and need for attention were happily walking on the street, not a care in the world, their bravado, as idiotic as it was, clear on their faces. Then groups of parents, masks on their chin, smoking and talking to each other as they supervise their children playing, blocking access ways where people have to squeeze between them in order to pass. I know those people, too. They already believe that they have been exposed as much as they could be, that trying to protect themselves when their children literally exchange spit and snot is futile. Then the drivers, happy that they don't have to wear masks inside their own car, but pretending to forget to use them even when they have to get out to buy something or to get home. Meanwhile, the police has found some specific sweet spots where they hunt for people not doing their duty regarding the pandemic, ignoring all others. And of course, the noseholes. People who wear their masks because it's mandatory, but only on their mouths, because they breathe through their noses.

  And yes, all of these are selfish assholes, but that's not my point now. These people are contributing to the rise of infections, but only a few weeks ago, when the rates were plummeting, they were careful with the masks, the distance, everything. The first and simplest hypothesis on why this happens is that as people stop wearing masks, infections start rising, and it surely does explain a lot, but it's not enough. These people could have relaxed weeks ago. Why now?

  My theory is that as schools opened, the number of people not wearing masks on the streets increased. Whether due to parent fatigue (let's call it that) or idiotic bravado or because some of them are too small to wear masks, children have skewed the proportion of people being responsible. And like a switch being flipped, other have started copying the same behavior. It's not even something conscious. I have difficulty believing people are actually reasoning that if a four year old doesn't wear mask, they should stop, too. It's a social phenomenon, a monkey see, monkey do thing. This has nothing to do with charts or numbers or public announcements which clearly show that we have now reversed our trend and the percentage of sick people is rising again. Another subconscious trigger is that more light is coming from the sun as we head towards spring. People attach darkness with crisis and light with things getting better. And yes, again, without thinking about it.

  This means several things. One of them is that communication of the facts doesn't function as it should. This is on the media and their panic inducing way of reporting things. They only discuss the bad things, the shocking things, not the good ones. Numbers don't matter for people who have lived in fear for months. At one point or another they are going to break and act out. The solution is to reward people when things are going well, to inspire rather than scare people off. It would have been easy to understand and feel what is going on if, after weeks of smiling news anchors congratulation the population for their efforts, they would have turned sad and announced the trend is reversing. But they don't do this and that makes them responsible.

  Second thing is that statistically people are barely aware of their existence. Like automatons they live their lives and copy each other's behaviors without a single thought. If prompted, they will find rationalizations for their actions after the fact, then feel resentment towards the person that forced them to make the effort. There is no solution for this, I am afraid. It falls on the authorities to be aware of this and act preventively to ensure a healthy proportion of people looking like they are doing the right thing. Note: looking like. This can also mean good propaganda or spin. Who would have thought that I would ever advocate for that?

  Finally, this cannot be turned into another cat and mouse game, where the authorities are punishing people when they catch them, so people do bad things when the police is not there. This is not a carrot and stick situation. The ingrained distrust of authorities and petty selfishness we learned during the Communist era makes traditional measures from the government fail miserably or even backfire. This is the most difficult thing yet: change the ways the system works in order to reward people for their good work. This is anathema to Romanian life and politics, but it must be done. Social responsibility needs to be bred back into our national DNA. Otherwise we will remain a nation of assholes like the guy in the image.

  One can take a container in which there is water and keep pouring oil in and after a time there will be more oil than water. That's because oil is hydrophobic, it "fears water" in a direct translation of the word. You can then say that the percentage of oil is higher than the percentage of water, that there is more oil in the container. Skin color in a population doesn't work like that, no matter how phobic some people are. Instead of water and oil, it's more like paint. One can take a container in which there is white paint and keep pouring black, red, yellow and brown paint in, but from a very early stage, that paint is no longer white.

  I keep finding these statistics about which part of the world is going to have Whites in a minority after a while. Any statistic counting people by color of skin is purist in nature and, as we should know by now, the quest for purity begets violence. The numbers are irrelevant if the basis of these statistics is conceptually wrong. In a true openly diverse population, white skin color should disappear really quick. The only chance for it to exist is if people with white skin would not mingle with people of any other color.

  What is a White person? Someone who has white skin? Someone who has European ancestry? Someone who has no ancestry that is not European? Are Jews white? How about coptic Egyptians? Some Asians are really white, too. There is no argument that uses the concept of White which is not directly dependant on the idea of racial purity. And then there is Non-White. A few days ago someone was noting that it feels weird to use the term Latino, considering how many different countries and interests are represented by the people labeled as such. So how can anyone meaningfully use a term like Non-White, which groups together Black people, Mexicans, Chinese, Indians, Eskimos and Native-Americans, among many others? Two "African-American" people of identical skin color may be as different as someone can imagine: one a many generations American with slave ancestors, the other a middle-class African recently arrived in the US.

  What I am saying is that the most politically correct terms, used (and imposed) by proponents (and arbiters) of racial justice and equality, are as purist as they could be. The only argument that one can possibly bring here is that purism is somehow different and distinct from racism. This is absurd. One can be a purist and not be racist, but not the other way around. In fact, when people are trying to limit your freedom of expression because some of your words or concepts may be offensive, they are in fact fighting for the purity of ideas, one that is not marred by a specific idea of purity that they are against. These are similar patterns, so similar in fact that I can barely see a difference. No wonder this kind of thinking has taken root most in a country where a part of its founders were called Puritans!

  So how about we change the rhetoric to something that does not imply segregation or a quest for purity or a war on something or cancelling other people or creating safe spaces or hating something that is other? And the phrase above is not ironic, since I am not proposing we fight against this kind of ideas, only that we acknowledge their roots and that we come up with new ones. Let us just grow in different directions, rather than apart.

  I've read today this CNN article: 'Star Trek: Discovery' to introduce history-making non-binary and transgender characters. And it got me thinking on what this means for the Star Trek universe. It means absolutely nothing. Star Trek has had people turned into other species, duplicated, merged, their genetic code altered and fixed, made young and old. It has had species with no gender, multiple genders and various philosophies. It has interspecies relationships, including sexual.

  Star Trek has tackled intolerance many times, usually by showing the Federation crew having contact with an alien species that does the same things we do today, in caricature. It tackled race intolerance, from Kirk's kiss with Uhura to the episode with the species with black on one side and white on the other discriminating the people who had their colors the other way around. It tackled gender discrimination in multiple situations. It tackled sex change and identity change with the Trill. It featured multi sex civilisations. The happy tolerance train seems to stop with anything related to using inorganic technology with the human body, but no one is perfect and Janeway was awful with everybody.

  A person who is biologically a man yet desires to be treated as a woman would be normal for Star Trek. It would be inconsequential. If they go the way of the oppressed member of another culture that they meet, they will not solve anything, they will just have another weird alien around, which defeats the purpose. If they go with a non-binary crewmember they should not acknowledge the fact except in passing. Yes, habituate the public with the concept, let them see it in a series and get used to it, but the people in Star Trek should already have passed that point. Hell, they could go with a person who changes their sex every one in a while, to spice things up.

  What I would do is have a character who is clearly of a different sex than the gender they identify with and someone badgering them to have a proper sex change while they refuse. Now that would be a Star Trek worthy dilemma. You want to make it spicy? Have them go to the doctor and change their race instead, behave like a black person while wearing the high tech equivalent of blackface. What? Are you refusing someone the ownership of their identity?

  I really doubt they will go that way, though. Instead they will find some way of bringing the subject up again and again and again and throw it in our faces like a conflict that has to be resolved. In the bright and hopeful future, there should be no conflict about it! This CBS announcement should not have existed. You want to put some transgender people in, OK, put them in. It's not a boasting point, is it? The announcement basically does the opposite of what they claim to do: "Oh, look, we put non binary people in our series! How quaint! Hurrah! Only we do it, come watch the freak show!".

  Please, writers, please please please, don't just write stories then change the gender or race of characters because it's en vogue. Stop it with the gender swapping, which is the creative equivalent of copy and paste. Write with the story in mind, with the context, with the characters as they would normally behave. Don't add characters after you've thought of the story just to make them diverse either. Just write stories with characters that make sense! You don't know people from that demographic? Find one, spend time with them, then adjust your characters accordingly. I am so tired of tiny female action heroes, flamboyant and loud gays and the wise old lesbian. How come no one finds those offensive? It's like someone said "OK, we will have shitty black and female and non-cis characters for now. When people get used to them, we will actually have them do something and be realistic and perhaps in 2245 we'll even have them be sympathetic".

  They tried the woke way from the very beginning in Discovery, with the Stamets/Culber gay couple. They kept showing them kissing and washing their teeth together and other stuff like that, when it made little difference to the story. Most people on Star Trek are written as single, for some weird reason that makes no sense, unless their relationship furthers the story. Riker and Troi could be the exception, though, yet even they were not kissy kissy on the bridge all the time. I never understood that couple. Dax and Worf made more sense, for crying out loud! And remember Starfleet is a military organization. You may put women and men and trans and aliens and robots together in a crew, but their role is to do their job. Their sex, their gender even less, makes no difference.

  Gene Roddenberry was a dreamer of better futures, where all of our idiotic problems have been left behind and reason prevailed, but even he imagined a third World War leading to humanity changing its ways as a start. Star Trek has always analysed the present from the viewpoint of an idyllic future, a way of looking back that is inherently rational: "Imagine the future you want, then wonder what would people from that time think of you". It's brilliant! Don't break that to bring stupid into the future. To tackle present social issues you have to first be a Trekkie, already there in the exalted future, before you consider the dark ages of the 21st century with a fresh perspective.

  For a more in depth exploration of the concept, read Towards generic high performance sorting algorithms

Sorting

  Consider QuickSort, an algorithm that uses a divide and conquer strategy to sort efficiently and the favourite in computer implementations.

  It consists of three steps, applied recursively:

  1. find a pivot value
  2. reorder the input array so that all values smaller than the pivot are followed by values larger or equal to it (this is called Partitioning)
  3. apply the algorithm to each part of the array, before and after the pivot

  QuickSort is considered generic, meaning it can sort any type of item, assuming the user provides a comparison function between any two items. A comparison function has the same specific format: compare(item1,item2) returning -1, 0 or 1 depending on whether item1 is smaller, equal or larger than item2, respectively. This formalization of the function lends more credence to the idea that QuickSort is a generic sorting algorithm.

  Multiple optimizations have been proposed for this algorithm, including using insertion sort for small enough array segments, different ways of choosing the pivot, etc., yet the biggest problem was always the optimal way in which to partition the data. The original algorithm chose the pivot as the last value in the input array and the average complexity was O(n log n), but worse case scenario was O(n^2), when the array was already sorted and the pivot was the largest value. Without extra information you can never find the optimal partitioning schema (which would be to choose the median value of all items in the array segment you are sorting).

  But what if we turn QuickSort on its head? Instead of providing a formalized comparison function and fumbling to get the best partition, why not provide a partitioning function (from which a comparison function is trivial to obtain)? This would allow us to use the so called distribution based sorting algorithms (as opposed to comparison based ones) like Radix, BurstSort, etc, which have a complexity of O(n) in a generic way!

  My proposal for a formal signature of a partitioning function is partitionKey(item,level) returning a byte (0-255) and the sorting algorithm would receive this function and a maximum level value as parameters.

  Let's see a trivial example: an array of values [23,1,31,0,5,26,15] using a partition function that would return digits of the numbers. You would use it like sort(arr,partFunc,2) because the values are two digits numbers. Let's explore a naive Radix sorting:

  • assign 256 buckets for each possible value of the partition function result and start at the maximum (least significant) level
  • put each item in its bucket for the current level
  • concatenate the buckets
  • decrease the level and repeat the process

Concretely:

  • level 1: 23 -> bucket 3, 1 -> 1, 31 -> 1, 0 -> 0, 5 -> 5, 26 -> 6, 15 -> 5 results in [0,1,31,5,15,6]
  • level 0: 0 -> 0, 1 -> 0, 31 -> 3, 5 -> 0, 15 -> 1, 6 -> 0 results in [0,1,5,6,15,31]

Array sorted. Complexity is O(n * k) where k is 2 in this case and depends on the type of values we have, not on the number of items to be sorted!

  More complex distribution sorting algorithms, like BurstSort, optimize their function by using a normal QuickSort in small enough buckets. But QuickSort still requires an item comparison function. Well, it is easy to infer: if partFunc(item1,0) is smaller or larger than partFunc(item2,0) then item1 is smaller or larger than item2. If the partition function values are equal, then increase the level and compare partFunc(item1,1) to partFunc(item2,1).

  In short, any distribution sorting algorithm can be used in a generic way provided the user gives it a partitioning function with a formalized signature and a maximum level for its application.

  Let's see some example partitioning functions for various data types:

  • integers from 0 to N - maximum level is log256(N) and the partition function will return the bytes in the integer from the most significant to the least
    • ex: 65534 (0xFFFE) would return 255 (0xFF) for level 0 and 254 (0xFE) for level 1. 26 would return 0 and 26 for the same levels.
  • integers from -N to N - similarly, one could return 0 or 1 for level 0 if the number is negative or positive or return the bytes of the equivalent positive numbers from 0 to 2N 
  • strings that have a maximum length of N - maximum level would be N and the partition function would return the value of the character at the same position as the level
    • ex: 'ABC' would return 65, 66 and 67 for levels 0,1,2.
  • decimal or floating point or real values - more math intensive functions can be found, but a naive one would be to use a string partitioning function on the values turned to text with a fixed number of digits before and after the decimal separator.
  • date and time - easy to turn these into integers, but also one could just return year, month, day, hour, minute, second, etc based on the level
  • tuples of any of the types above - return the partition values for the first item, then the second and so on and add their maximum levels

  One does not have to invent these functions, they would be provided to the user based on standard types in code factories. Yet even these code factories will be able to encode more information about the data to be sorted than mere comparison functions. Stuff like the minimum and maximum value can be computed by going through all the values in the array to be sorted, but why do it if the user already has this information, for example.

  Assuming one cannot find a fixed length to the values to be sorted on, like real values or strings of any length, consider this type of sorting as a first step to order the array as much as possible, then using something like insertion or bubble sort on the result.

Finding a value or computing distinct values

  As an additional positive side effect, there are other processes on lists of items that are considered generic because they use a formalized form function as a parameter. Often found cases include finding the index of an item in a list equal to a given value (thus determining if the value exists in a list) and getting the distinct values from an array. They use an equality function as a parameter which is formalized as returning true or false. Of course, a comparison function could be used, depending on if its result is 0 or not, but a partitioning function can also be used to determine equality, if all of the bytes returned on all of the levels are equal.

  But there is more. The format of the partition function can be used to create a hash set of the values, thus reducing the complexity of the search for a value from O(n) to O(log n) and that of getting distinct values from O(n^2) to O(n log n)!

  In short, all operations on lists of items can be brought together and optimized by using the same format for the function that makes them "generic": that of a partitioning function.

Conclusion

  As you can see, I am rather proud of the concepts I've explained here. Preliminary tests in Javascript show a 20 fold improvement in performance for ten million items when using RadixSort over the default sort. I would really love feedback from someone who researches algorithms and can even test these assumptions under benchmark settings. Them being complex as they are, I will probably write multiple posts on the subject, trying to split it (partition it?) into easily digestible bits

 The concept of using a generic partitioning function format for operations on collections is a theoretical one at the moment. I would love to collaborate with people to get this to production level code, perhaps taking into account advanced concepts like minimizing cache misses and parallelism, not only the theoretical complexity.

 More info and details at Towards generic high performance sorting algorithms

Intro

  I want to examine together with you various types of sort algorithms and the tricks they use to lower the magic O number. I reach the conclusion that high performance algorithms that are labeled as specific to a certain type of data can be made generic or that the generic algorithms aren't really that generic either. I end up proposing a new form of function that can be fed to a sorting function in order to reach better performance than the classic O(n*log(n)). Extra bonus: finding distinct values in a list.

Sorting

  But first, what is sorting? Given a list of items that can be compared to one another as lower or higher, return the list in the order from lowest to highest. Since an item can be any type of data record, to define a generic sorting algorithm we need to feed it the rules that make an item lower than another and that is called the comparison function. Let's try an example in Javascript:

  // random function from start to end inclusive
  function rand(start, end) {
    return parseInt(start + Math.random() * (end - start + 1));
  }
  
  // measure time taken by an action and output it in console
  let perfKey = 0;
  function calcPerf(action) {
    const key = perfKey++;
    performance.mark('start_' + key);
    action();
    performance.mark('end_' + key);
    const measure = performance.measure('measure_' + key, 'start_' + key, 'end_' + key);
    console.log('Action took ' + measure.duration);
  }
  
  // change this based on how powerful the computer is
  const size = 10000000;
  // the input is a list of size 'size' containing random values from 1 to 50000
  const input = [];
  for (let i = 0; i < size; i++)
    input.push(rand(1, 50000));
  
  // a comparison function between two items a and b
  function comparisonFunction(a, b) {
    if (a > b)
      return 1;
    if (a < b)
      return -1;
    return 0;
  }
  
  const output = [];
  // copy input into output, then sort it using the comparison function
  // same copying method will be used for future code
  calcPerf(() => {
    for (let i = 0; i < size; i++)
      output.push(input[i]);
    output.sort(comparisonFunction);
  });

  It's not the crispest code in the world, but it's simple to understand:

  • calcPerf is computing the time it takes for an action to take and logs it to the console
  • start by creating a big array of random numbers as input
  • the array in a result array and sorting it with the default sort function, to which we give the comparison function
  • display the time it took for the operation.

  This takes about 4500 milliseconds on my computer.

  Focus on the comparison function. It takes two items and returns a number that is -1, 0 or 1 depending on whether the first item is smaller, equal or larger than the second. Now let's consider the sorting algorithm itself. How does it work?

  A naive way to do it would be to find the smallest item in the list, move it to the first position in the array, then continue the process with the rest of the array. This would have a complexity of O(n2). If you don't know what the O complexity is, don't worry, it just provides an easy to spell approximation of how the amount of work would increase with the number of items in the input. In this case, 10 million records, squared, would lead to 100 trillion operations! That's not good.

  Other algorithms are much better, bringing the complexity to O(n*log(n)), so assuming base 10, around 70 million operations. But how do they improve on this? Surely in order to sort all items you must compare them to each other. The explanation is that if a<b and b<c you do not need to compare a to c. And each algorithm tries to get to this in a different way.

  However, the basic logic of sorting remains the same: compare all items with a subset of the other items.

Partitioning

  A very common and recommended sorting algorithm is QuickSort. I am not going to go through the entire history of sorting algorithms and what they do, you can check that out yourself, but I can focus on the important innovation that QuickSort added: partitioning. The first step in the algorithm is to choose a value out of the list of items, which the algorithm hopes it's as close as possible to the median value and is called a pivot, then arrange the items in two partitions: the ones smaller than the pivot and the ones larger than the pivot. Then it proceeds on doing the same to each partition until the partitions are small enough to be sorted by some other sort algorithm, like insertion sort (used by Chrome by default).

  Let's try to do this manually in our code, just the very first run of the step, to see if it improves the execution time. Lucky for us, we know that the median is around 25000, as the input we generated contains random numbers from 1 to 50000. So let's copy the values from input into two output arrays, then sort each of them. The sorted result would be reading from the first array, then from the second!

  // two output arrays, one for numbers below 25000, the other for the rest
  const output1 = [];
  const output2 = [];
  const pivot = 25000;
  
  calcPerf(() => {
    for (let i = 0; i < size; i++) {
      const val = input[i];
      if (comparisonFunction(val, pivot) < 0)
        output1.push(val);
      else
        output2.push(val);
    }
    // sorting smaller arrays is cheaper
    output1.sort(comparisonFunction);
    output2.sort(comparisonFunction);
  });

  Now, the performance is slightly better. If we do this several times, the time taken would get even lower. The partitioning of the array by an operation that is essentially O(n) (we just go once through the entire input array) reduces the comparisons that will be made in each partition. If we would use the naive sorting, partitioning would reduce nto n+(n/2)2+(n/2)2 (once for each partitioned half), thus n+n2/2. Each partitioning almost halves the number of operations!

  So, how many times can we half the number of operations for? Imagine that we do this with an array of distinct values, from 1 to 10 million. In the end, we would get to partitions of just one element and that means we did a log2(n) number of operations and for each we added one n (the partitioning operation). That means that the total number of operations is... n*log(n). Each algorithm gets to this in a different way, but at the core of it there is some sort of partitioning, that b value that makes comparing a and c unnecessary.

  Note that we treated the sort algorithm as "generic", meaning we fed it a comparison function between any two items, as if we didn't know how to compare numbers. That means we could have used any type of data as long as we knew the rule for comparison between items.

  There are other types of sorting algorithms that only work on specific types of data, though. Some of them claim a complexity of O(n)! But before we get to them, let's make a short detour.

Distinct values

  Another useful operation with lists of items is finding the list of distinct items. From [1,2,2,3] we want to get [1,2,3]. To do this, we often use something called a trie, a tree-like data structure that is used for quickly finding if a value exists or not in a list. It's the thing used for autocorrect or finding a word in a dictionary. It has an O(log n) complexity in checking if an item exists. So in a list of 10 million items, it would take maybe 20 operations to find the item exists or not. That's amazing! You can see that what it does is partition the list down to the item level.

  Unfortunately, this only works for numbers and strings and such primitive values. If we want to make it generic, we need to use a function that determines when two items are equal and then we use it to compare to all the other items we found as distinct so far. That makes using a trie impossible.

  Let me give you an example: we take [1,1,2,3,3,4,5] and we use an externally provided equality function:

  • create an empty output of distinct items
  • take first item (1) and compare with existing distinct items (none)
  • item is not found, so we add it to output
  • take next item (1) and compare with existing distinct items (1)
  • item is found, so we do nothing
  • ...
  • we take the last item (5) and compare with existing items (1,2,3,4)
  • item is not found, so we add it to the output

  The number of operations that must be taken is the number of total items multiplied by the average number of distinct items. That means that for a list of already distinct values, the complexity if O(n2). Not good! It increases exponentially with the number of items. And we cannot use a trie unless we have some function that would provide us with a distinctive primitive value for an item. So instead of an equality function, a hashing function that would return a number or maybe a string.

  However, given the knowledge we have so far, we can reduce the complexity of finding distinct items to O(n*log(n))! It's as simple as sorting the items, then going through the list and sending to output an item when different from the one before. One little problem here: we need a comparison function for sorting, not an equality one.

So far

  We looked into the basic operations of sorting and finding distinct values. To be generic, one has to be provided with a comparison function, the other with an equality function. However, if we would have a comparison function available, finding distinct generic items would become significantly less complex by using sorting. Sorting is better than exponential comparison because it uses partitioning as an optimization trick.

Breaking the n*log(n) barrier

  As I said above, there are algorithms that claim a much better performance than n*log(n). One of them is called RadixSort. BurstSort is a version of it, optimized for strings. CountSort is a similar algorithm, as well. The only problem with Radix type algorithms is that they only work on numbers or recursively on series of numbers. How do they do that? Well, since we know we have numbers to sort, we can use math to partition the lot of them, thus reducing the cost of the partitioning phase.

  Let's look at our starting code. We know that we have numbers from 1 to 50000. We can find that out easily by going once through all of them and computing the minimum and maximum value. O(n). We can then partition the numbers by their value. BurstSort starts with a number of "buckets" or lists, then assigns numbers to the buckets based on their value (dividing the value to the number of buckets). If a bucket becomes too large, it is "burst" into another number of smaller buckets. In our case, we can use CountSort, which simply counts each occurrence of a value in an ordered array. Let's see some code:

  const output = [];
  const buckets = [];
  calcPerf(() => {
    // for each possible value add a counter
    for (let i = 1; i <= 50000; i++)
      buckets.push(0);
    // count all values
    for (let i = 1; i <= size; i++) {
      const val = input[i];
      buckets[val - 1]++;
    }
    // create the output array of sorted values
    for (let i = 1; i <= 50000; i++) {
      const counter = buckets[i - 1];
      for (let j = 0; j < counter; j++)
        output.push(i);
    }
  });

  This does the following:

  • create an array from 1 to 50000 containing zeros
  • for each value in the input, increment the bucket for that value
  • at the end just go through all of the buckets and output the value as many times as the value in the bucket shows

  This algorithm generated a sorted output array in 160 milliseconds!

  And of course, it is too good to be true. We used a lot of a priori knowledge:

  • min/max values were already known
  • the values were conveniently close together integers so we can use them as array indexes

  I can already hear you sigh "Awwh, so I can't use it!". Do not despair yet!

  The Radix algorithm, that is used only for numbers, is also used on strings. How? Well, a string is reducible to a list of numbers (characters) so one can recursively assign each string into a bucket based on the character value at a certain index. Note that we don't have to go through the entire string, the first few letters are enough to partition the list in small enough lists that can be cheaply sorted.

  Do you see it yet?

A generic partition function

  What if we would not use an equality function or a comparison function or a hashing function as a parameter for our generic sort/distinct algorithm? What if we would use a partition function? This partition function would act like a multilevel hashing function returning values that can also be compared to each other. In other words, the generic partition function could look like this:

  function partitionFunction(item, level) returning a byte

  For strings it returns the numeric value of the character at position level or 0. For numbers it returns the high to low byte in the number. For object instances with multiple properties, it would return a byte for each level in each of the properties that we want to order by. Radix style buckets would use the known values from 0 to 255. The fact that the multilevel partitioning function is provided by the user means we can pack in it all the a priori knowledge we have, while keeping the sorting/distinct algorithm unchanged and thus, generic! The sorting will be called by providing two parameters: the partitioning function and the maximum level to which it should be called:

  sort(input, partitioningFunction, maxLevel)

A final example

  Here is an implementation of a radix sorting algorithm that receives a multilevel partitioning function using our original input. Note that it is written so that it is easily read and not for performance:

  // will return a sorted array from the input array
  // using the partitioning function up to maxLevel
  function radixSort(input, partitioningFunction, maxLevel) {
    let buckets = Array.from({length: 256}, () => []);
    buckets[0] = input;
    // reverse order, because level 0 should be the most significant
    for (let level = maxLevel-1; level >=0; level--) {
      let tempBuckets = Array.from({length: 256}, () => []);
      for (let bucketIndex = 0; bucketIndex < buckets.length; bucketIndex++) {
        const bucket = buckets[bucketIndex];
        const bucketLength = bucket.length;
        for (let bucketOffset = 0; bucketOffset < bucketLength; bucketOffset++) {
          const val = bucket[bucketOffset];
          const partByte = partitioningFunction(val, level);
          tempBuckets[partByte].push(val);
        }
      }
      buckets = tempBuckets;
    }
    const output = [].concat(...buckets);
    return output;
  }

  // return value bytes, from the most significant to the least
  // being <50000 the values are always 2 bytes  
  function partitioningFunction(item, level) {
    if (level === 0) return item >> 8;
    if (level === 1) return item & 255;
    return 0;
  }
  
  let output3 = [];
  calcPerf(() => {
    output3 = radixSort(input, partitioningFunction, 2);
  });

Want to know how long it took? 1300 milliseconds!

You can see how the same kind of logic can be used to find distinct values, without actually sorting, just by going through each byte from the partitioning function and using them as values in a trie, right?

Conclusion

Here is how a generic multilevel partitioning function replaces comparison, equality and hashing functions with a single concept that is then used to get high performance from common data operations such as sorting and finding distinct values.

I will want to work on formalizing this and publishing it as a library or something like that, but until then, what do you think?

Wait, there is more

There is a framework in which something similar is being used: SQL. It's the most common place where ORDER BY and DISTINCT are used. In SQL's case, we use an optimization method that uses indexes, which are also trie data structures storing the keys that we want to order or filter by. Gathering the data to fill a database index also has its complexity. In this case, we pre-partition once and we sort many. It's another way of reducing the cost of the partitioning

However, this is just a sub-type of the partition function that I am talking about, one that uses a precomputed data structure to reach its goal. The multilevel partition function concept I am describing here may be pure code or some other encoding of information we know out of hand before doing the operation.

Finally, the complexity. What is it? Well instead of O(n*log(n)) we get O(n*k), where k is the maximum level used in the partition function. This depends on the data, so it's not a constant, but it's the closest theoretical limit for sorting, closer to O(n) than the classic log version. I am not the best algorithm and data structure person, so if you have ideas about it and want to help me out, I would be grateful.  

Why this article should never have been written

  It's a bit too early to do this. I am sure no one in their right mind would want to display any non-positive words about George Floyd at this moment for fear of reprisals. But I feel like that is exactly what must be done. If Floyd was an innocent victim, a hero that overcame his background only to be brought down, literally, by the heavy boot of law enforcement, then what chance do normal people have?

  I didn't want to be writing about this. I think that the entire thing has become grotesque and the only thing that could now bring these Whites and Blacks together, corrupt police and gangstas, racists and anti-racists... is me! I am sure that if I were to enter the argument, all these people angrily hating each other would come together in trying to murder me. Because while I understand both sides, I can't fully endorse any of them. Racists are just dumb. What the hell does race matter in anything? But I do understand anti-anti-racists, the ones that hate being put together with assholes because of the color of their skin. Anti-racism protesters are dumb. Well, maybe. I am sure all of this protesting is finally going to have an impact, and this is good, but have you seen some of these people? Manicaly jumping on toppled down statues and roaring in megaphones about how great they are because they oppose evil. In this whole discussion, again, normal people are left out. They are boring, they don't clump together, they don't stand out, each one has their own slightly different opinion. Well, this is mine.

The gentle giant saint versus the black monster

  Something happened today that pushed me to write this post. I saw a Facebook post that detailed the criminal record of George Floyd. Cocaine dealing, two armed robberies, one which held him back four years, addiction and, when he was arrested, metamfetamine and fentanyl in his blood and the incriminating fake twenty dollar bill. Was it true? It is a very important question to ask, because many of these things are complete bullshit. So I googled it. And the result was actually worse: almost nothing!

  There are just two websites that actually advertise Floyd's criminal record: Great Game India - self titled "Journal on Geopolitics and International Relations" and sporting articles like "Coronavirus Bioweapon - How China Stole Coronavirus From Canada and Weaponized It" and "How A Pornstar & Sci-Fi Writer Influenced WHO Policies On Hydroxychloroquine With Fake Data" - and The Courier Daily, which seems legit. Funny though, when you search for "George Floyd criminal record" you get Game India first and not The Daily Mail, which is linked in their article and who actually did the research and published the court documents attesting to that. They are fifth on the search page. More, during the writing of this blog post, the Courier Daily link disappeared from my Google search and Game India was demoted to second place, with a "gentle giant" story on top instead.

  Either way, other than Game India, no other news outlet even phrases the title as to indicate George had been a criminal. The few who tackle the subject: The Star, The Daily Mail itself and even The Courier Daily, just portray the man as a flawed individual who nevertheless was set to change, found religion and even moved to Minneapolis to escape his past. And I agree with this viewpoint, because as far as I can see, the armed robbery had been many years before and the man had changed, in both behavior and intent. But hiding this doesn't really help. The Daily Mail article was published on the 26th of May, one day after Floyd's death, and the information therein is either not discussed or spun into a "gentle giant" narrative. He was a bouncer before the Coronavirus hit and he lost his job. George the gentle bouncer?

  One thing is certain, when you search for George's criminal record, it's more likely you get to articles about the criminal records of the arresting officers or Mark Wahlberg's hate crimes than what you actually searched for.

How did George die and why it doesn't matter

  But there is more. How did George die? You would say that having a knee on their throat while they gasp for air saying "I can't breathe" would be enough. But it's not. Several different reports say different things. The first one preliminarily portrays Floyd as a very sick man: coronary artery disease, hypertensive heart disease, even Covid-19. There were "no physical findings that support a diagnosis of traumatic asphyxia or strangulation", but instead they diagnosed it as a heart failure under stress and multiple intoxicants. Finally, two days later, the report admits "a cardiopulmonary arrest while being restrained" by officers who had subjected Floyd to "neck compression". But Floyd's family would not have that, so they commissioned their own autopsy. The result? Floyd died from "asphyxia due to compression of the neck", affecting "blood flow and oxygen going into the brain", and also from "compression of the back, which interferes with breathing". The medical examiner said Floyd had no underlying medical problem that caused or contributed to his death.

  So which was it? It doesn't really matter. He died with a knee on his neck, which should never happen to anyone, and both reports admit it was a homicide. But ignoring all of these other data points doesn't help. People just vilify the policeman and raise George to saintly status. You want to solve something? Start with the truth. All of it. Now both sides have the ammunition required to never change their minds.

  I have not found any article that makes a definitive claim on which report is the good one, if any. They all lean on believing the second, because it fits, but if the first one was a complete fabrication, why wasn't anyone charged with it?

Wikipedia v. Facebook

  So of course I would find about Floyd's criminal past from Facebook. It makes so much sense. It is a pool of hateful bile and rank outrage that brings ugly right up to the surface. But this time it pointed me towards an interesting (albeit short) investigation. Without it, I would have swallowed up the entire completely innocent victim narrative that is pushed on every media outlet. So, once in a blue moon, even Facebook is good for something.

  As you may have noticed above, I took some information from Wikipedia, which has an entire article dedicated to George Floyd's death. It is there where the information about his two medical autopsies is also published. On George Floyd's page, his early life consists of: basketball, football, his family calling him a gentle giant. Then he customized cars, did some rap and was an informal community leader. Only then did he get arrested a few times then put in jail for five years. He was charged in 2007, went to jail in 2009 and was released on 2013. It's just six years and it does not define a man, but try to say that to a police officer who has just read the fact sheet on his cruiser's terminal and has to arrest a 1.93m tall intoxicated man.

  And you may want to read the entire chain of events. The police didn't just put him on the ground, they talked to him, they put him in their car, they fought, they pulled him out, all while being filmed and surrounded by a crowd.

You will never gonna get it

  How much of this is truth and how much of it is spin? You will never know. There are so many people that have to justify their own shit using carefully chosen bits and pieces from this that there will never be a truthful image of who George Floyd was and what happened to him. He is now more than a man and also much less: he is a symbol, rallying people to cry out against injustice, as they see it. The greatest thing George Floyd ever did was die and after that he stopped being human. How sad is that?

  In truth, he was a flawed man. He was not perfect. He was your everyman. A policeman casually killing him while getting filmed doing it hurts on so many levels because that could be you. That was you or your friend or your relative somewhere. But no, they had to make it about being black and being the gentle giant and being killed by the bad psycho cop and his cronies. It sounds like a Hollywood movie because it is scripted as one. You can be certain that at this point several documentaries and movies are in the works about this. And when you'll see it, a big time celebrity will be interpreting Floyd and the cop will be played by that actor who plays psychos in every other movie because he has that face. Once you go black, you can never go back.

  I am not defending anyone here. As I said in the beginning, I am on nobody's side in this. I just hope no one will knee me or my friends to death while everybody films it down.

The world has spoken

  I find it amazing that the protests in Minneapolis have spread to the entire world. It makes me hope that they will slowly turn into protests about things that matter even more than the color of one's skin, like our responsibility as a community to carefully choose our guardians, like having to think for ourselves if something is right or wrong and maybe doing something about it. George Floyd was killed slowly, over nine minutes, while people stood around and filmed it. Not just the other officers, but civilian bystanders, too.

  There were people who did something. At one point a witness said: "You got him down. Let him breathe." Another pointed out that Floyd was bleeding from the nose. Another told the officers that Floyd was "not even resisting arrest right now". Yet another said "Get him off the ground ... You could have put him in the car by now. He's not resisting arrest or nothing. You're enjoying it. Look at you. Your body language explains it." But that's all they did. Wikipedia calls them "witnesses", but you have to wonder: what skin color were they? Were they afraid they would be next and that's why all they could was beg for George's life as he slowly died? Or did they believe the story that American TV has fed them for decades, that cops are ultimately good people who break the rules in order to protect the innocent? Or maybe a more recent belief had taken hold: that filming injustice makes you a hero and it's more than enough.

  The world has spoken. Racism must go, police brutality must go. Let's not replace them by carefully crafted fantasies, though. Let's see the truth as it is so we can make it better.

2020 is great so far

  I am not being sarcastic. After a virus that punched presidents of the free world and dictators alike in the nose, that made people question their fake feelings of safety and forced them to act, here comes this age of protesting how things are. We have been shaken awake. Will we fall asleep again? I am sure we will, but some things will have changed by then. And the year is not yet over.

  I want to write this post to talk about the most common mistake I make as a software developer, even after almost 20 years of experience. And it's not code related. It's more human. But I would also like to hear what you think your biggest mistakes are that are not related to lack of experience. Don't be shy!

  My mistake: assumptions.

  I was assigned this bug recently and, wanting to do a fast job and impress people, I investigated the code, noticed a bug, fixed it, then immediately gave it in for review. I had reasons for doing that, because I was new and did not know the application well. The leads would tell me if they thought I did not find the problem. But, out of the goodness of my heart, you see, I've decided to test the fix as well. And I discovered that the feature was barely implemented. It was not a bug, it was a full fuck up.

  What happened here? I assumed a certain quality of the code and expected, without any reasonable evidence, to find a small typo or a logic bug that would be solved by writing a few lines of code. Instead, I had to reimplement the whole thing as a fix, I pissed off the lead devs because they had enough on their plate and made the entire team wonder what I was doing there. I mean, I haven't even tested the fix!

  Doing things fast means working on valid assumptions that allow you to cut corners. In a new environment, with a team you are not familiar with and a code base you don't understand, you do not have the luxury to make assumptions. Instead, do a good job first: investigate thoroughly, see what the reported problem is, find the actual problem (which may be different), come with an attack plan, implement it, then test that it had the desired result. Yes, it takes more time than to quickly compile the logic flow in your head and hope for the best, but in some places you don't get second chances at fixing things, teams are more formal, processes need to be followed. Optimism is also based on assumptions. Be a realist instead.

  In order to win a game you need to know its rules. That applies both to development process and to local politics, which sometimes are more important than the work. Once you are a good player, you can start experimenting. The title "senior developer" is not given in a vacuum, but is relevant (or not) depending on the environment. Yes, you have experience, but you don't have *this* experience yet. In my desire to be efficient and fast I didn't do anything right and I couldn't believe I have been that stupid.

  Now, how about you? What are your silly mistakes that you can't believe you are still making?