Engineering Hives and Silos


The story goes…

Business X has been using a software package for the last 20+ years to perform a critical, maybe even the most critical, function of the business.

20+ years ago this package used to be an off the shelf, install on an office PC, double-click to launch in Windows type of application. It probably looked rubbish but had a menu with 3+ levels to navigate dozens of screens with 20+ fields each to manipulate. There were only a few people that could learn and knew how to use it for some very specific tasks.

10+ years ago the vendor of that package found it easier – meaning no install, no update, no office PC – to put the same software on a bunch of servers and let users access it remotely (RDP).

Today many of these applications are still running businesses, mostly because they are cheap or because over time they became a monopoly.

Innovation at these vendors, and the businesses using them, would have stopped, if it wasn’t for a new movement.
Reluctantly, the vendors slowly added a new type of access – API – to their packages.
These dumb API-s would often look like an XML version of the screens they represent.

Business X realised they have a new option.
Rather than paying for an expensive 20+ years old application running remotely they can pay a lower fee for accessing the API.

Projects are in abundance making the transition – under the banner of Digital Transformation – from remote desktop Windows applications to web based, API led solutions.

There are so many ways these transformation projects go up in smoke.
One is where business X insists on rebuilding the same screen flow from the old desktop application.

Is this Digital Transformation? No.
Maybe modernisation? Hardly.
It is simply cost avoidance.

Engineering Hives and Silos

Tech Stack Bingo

Tech Stack Bingo is when a business has a portfolio of, seemingly random, tech they acquired (sold to) over the years.
As the projects roll in, the needs mark a match with an existing tech.
The closer the match the better. They win if no tech lingers on without a project and nothing comes in needing a new tech – BINGO.

Some sorry businesses try to cheat their way by forcing some close enough tech on the project.

This is where disaster strikes. It happens more often than one would expect, for example:

  • somebody bought the tech and now need to use it, cannot lose face (or the job)
  • everything is done with the one and only tech the organisation has some skills in (nail and hammer)

Engineering Hives and Silos

Do Not Expect Strategic When Asking for Tactical

The story goes…

The program – has lofty goals for transforming the way to do business, to catch up with challenger businesses (have not kept up with customer expectations).

The first few projects – identified and added to the project portfolio plan for the year.
As for the rest – there is no budget (only planning for the year), or no appetite (the sponsor may not be around next year).

The very first project – it must respond to the immediate problem (after all that’s how the program got the buy-in).
It must do that quickly – 1 month – therefore looking for a tactical solution (calling it MVP to make it sound less desperate).
The chosen technology/product must be future-proof and fit the strategic direction (cannot justify spend on throw away).

6 months later…

  • Why isn’t the first project complete? It was supposed to be quick.
  • Where is the value add from using this technology/product? It is supposed to be the strategic choice for all projects to come.

What has just happened?…

If tactical is aligned with strategic, it would not be called tactical, it would be called strategic.
The strategic choice product/technology may not be able to demonstrate value add for the entire program in a tactical (also the first) project.
The same features and functions for building strategic capabilities rarely, if ever, meet the criteria and constraints of legacy in a tactical build.

Engineering Hives and Silos

Not Meeting in the Middle

The person holding the P&L and sponsoring the project needs new, improved capabilities for the business to create value. A suitable IT solution and project could deliver the needs.
There is no interest in architecture, performance, documentation, choice of platform. It just needs to happen.

The team of individual contributors in delivery just want to get their code into production as fast as possible. This is particularly true for contractors and offshore delivery centres with economical rates measured on the speed of delivery.
They too have little interest in architecture, performance, documentation, platform.

Folks in the middle who are tasked to make it all sustainable – so the next project does not cost an arm and a leg, and does not take a lifetime to complete – are in an impossible position.
They do care about architecture, performance, documentation, platform, and more.

Many businesses have been practicing decades of waterfall shaped delivery. The projects are typically sized between 6 to 12 months, while parent programs can run much longer. A sizeable work upfront on requirements is rapidly followed by delivering, churning out code up to final delivery into production.
Meanwhile, governance bodies and design authorities are desperately trying to wedge themselves in between stages to provide some oversight and much needed guidance.

Now that everyone and their dog is pitching to be agile, a torrent of anti-patterns emerge from waterfalling into sprints expedited by doing away with plan, design and documentation.
Practically, entirely removing the opportunity for oversight and to work out sustainable solutions.

A worthy read on the struggles of business-IT alignment: Don’t become an Enterprise/IT Architect… by Gerben Wierda

Engineering Hives and Silos

Kicking the can down the road

Technical debt sneaks up on organisations and their systems. It is inevitable.
The ability of dealing with, managing, remediating it differentiates successful businesses from the rest.

Technical debt builds up over time. It could take years for it to amount to significance. If unattended, it reaches a time when the fixing costs are higher than the return from it.
The system still produces value at this point, it has not gone bankrupt. There are even workarounds to some of the issues.
However, innovation, new feature and product introduction is not sustainable anymore – nobody wants to touch a house of cards system, or pay the premium for the overhead of dealing with the mess.

The system enters the zone of – kicking the can down the road.

Occasionally, a large(ish) investment into the existing system might be justified if it generates enough value – as opposed to replacing it entirely, which would be too expensive.

In most cases, just enough fixes are applied to keep the lights on and to keep sweating the asset for the rest of its operational life.

Minimise the effect of bad debt and extend the productive lifespan of a system by

  • Looking out for the technical debt indicators
  • Putting strategies in place for managing technical debt
Engineering Hives and Silos

Multi-Tenant Systems and Rivers

The common metaphor for multi-tenant systems is the block of flats where tenants share the building and its services (water, gar, electricity, security, etc) but each tenant may organise and furnish their flat differently.

The part – that is often left out – is where the landlord limits each tenant how much change they can make to the flats, how many guests they can invite, how much of the services they can use. These limits what make multi-tenant buildings viable by protecting the infrastructure for everyone’s benefit.

SaaS and PaaS and IaaS are good examples of shared environments where the multi-tenancy metaphor works well.
The service providers – landlords – manage these environments closely, and enforce the limits strongly. This way everybody benefits the same way and providers can maintain their service levels (SLAs).

This metaphor struggles when transitioning to the next level of abstraction – sharing resources within one tenancy.
Multiple projects, new features, updates, fixes from multiple sources draining resources within the same tenancy.

The river metaphor for a shared environment is more suitable1

Each project and feature is an industrial installation – a factory plant, a power station, or any other build directly connected to the water ways.
Well behaving projects will keep the environment clean and liveable for everybody.
Projects with weak or no governance will pollute the waterways and ruin it for themselves and everybody else. Those downstream will have to work extra hard to clean up and maintain it for the rest.

Why does the river metaphor work better?
Because the societal and policy issues better resonate with the equivalent governance issues. It emphasises the negative effects of badly managed changes as opposed to showcasing the positive effects of a well managed environment.
It highlights the need for better and stronger governance in the absence of hard limits.

1Inspired by Seth Godin’s podcast episode: The river of time