Random Rant: The Persistence of Vertical Scaling Practices in High Level Language Development

It is not 1994, even the most underpowered machine has more than 4GB of storage spinning at 5400 rpm, gigabytes of RAM no megabytes, and at least one processor running in the gigahertz range. Networks hum along at megabytes per second instead of kilobytes, and while it is beneficial to understand how to work in environments that are resource scarce. The context in which many of us work assumes that languages like Ruby, Python, PHP, and Perl are being chosen because productivity is paramount.

Before my detractors can begin to pick, I'm not advocating wasteful use of computing resources, in Python learn how to pop your lists and use generators, check for memory bloat in your runtime, and configure your database connectivity to make fewer rather than more calls over the network. But keep the readability and maintainability of your code in the front of your mind at all times. Don't put listeners on database events that in the background set attributes on your persisted types. Use stable state constructors with documentation, don't extend built in types because you want to access the attributes of your custom types like a hash, and implement some obscure access pattern from Cobol. Don't, don't, don't... I will hunt you down and make you wear a dunce cap for a week while you recite "Keep It Simple Stupid" over and over again.

One of the most zen signs I have ever seen on a cubical wall was "It has to work before it can be optimized." Another saying that I like, "If you write code at your cleverest, often you are not smart enough to debug it." There is a new sheriff in town and his name is Convention Over Configuration, and the sheriff's deputy is Horizontal Scaling, we are moving past the age of spending months working on code efficiency to improve performance vertically. There are still contexts where scrutinizing code efficiency is important; kernel programming, rich client development of applications that handle content which requires high throughput process such as audio and video, language development/compilers, but web development with Python or one of the other high level languages is not one of the context where vertical scaling is appropriate in most situations. What is far more important in development with high level languages is well documented code that is implemented in stable reusable packages, and the separation of concern between application layers using patterns like MVC.

Everyone has their own style, I'm sure there are web development groups that are very successful writing vertically optimized Python for web applications, I just don't want to work for any of them. My philosophy is work myself out of my job, create reusable and maintainable solutions that extended and scale easily, so that I can move on to the next project. If I wanted to write the most efficient algorithm for iterating over a struct I would be a kernel programmer.

Update: When I wrote this I didn't realize, but somehow subliminally knew that the Python/Pyramid web application I'm working on was written by a kernel developer. I made the realization today, I've hit somewhat of a wall and as I'm staring at the code hoping some kind of understanding will pop into my head. I suddenly realized where I had seen this style of coding before. There is very little OOP, direct manipulation of type members is chosen over encapsulation in methods, there are no constructors, and the few methods that do exist have complex interfaces. The method invocation depends on the presence of other optional arguments' values, where the values needed by the required arguments changes. The methods also have different side effects depending on the state of the application and can behave erratically. Suddenly it all hit me, I did a search on the other developer's name and all these hits for kernel development posts came back...

I don't think I'm too out of line by stating that kernel development is one of the areas in software engineering where modern development paradigms have made little impact. You are just beginning to hear about test driven development in C, and for the most part terseness of code is chosen over readability and maintainability. I'm not too proud to say that I'm probably not qualified to work on this application, I spend most of my time walking through the code with PDB examining the state of the application, but quite often the issue lies in another part of the application lost in the tangle of types coded to behave like structs and listeners that alter the state during built in method calls.