We were in an IBM MVS environment, where an application tended to be made up of a large number of moving parts, all of which needed to be properly coordinated to be sure that the system would continue to function. We'd have Cobol and PL/1 programs, usually from a number of separate source files, IMS database description elements, display screen formats, Job Control Language and so forth.

These applications basically ran the company -- stock controls, parts ordering, you get the picture. Getting such implementations right first time was important. I had more than a few late nights and callouts when things went wrong, and the odd one or two overnight shifts when large and extra-important implementations were happening.

Some of these might have had upwards of 500 things to be compiled, linked, assembled, or just syntax checked along the way, and a single failure would abort the whole process. While this process might have seemed onerous and complex, it actually made life a lot easier for us, because it defined a set of rules as to what could and could not be done, and because it all went through a testing environment, helped to ensure that components didn't get forgotten, and steps didn't get overlooked.

And crucially, it took the developers themselves out of the nuts and bolts of getting things into production, and helped to ensure that they didn't have any sort of privileged access to the production environment.

A big Notes application
Fast forward to the Lotus Notes world of 2009. Notes applications these days often have upwards of 500 moving parts -- design elements -- in them, and many serious applications consist of much more than one Notes database.

Indeed, one application that I spent some years working with consisted of best part of 10 databases -- one having over 1,500 design elements, and there were over 100,000 lines of LotusScript in the whole application. And worse, it had to be distributed to multiple, non-replica production platforms in a coordinated manner, properly signed, agents scheduled to run, and so on.

And, just like quite a number of Notes applications these days, it was and is truly mission-critical to the organisation that owns it -- in this case, with potential life and death downstream consequences, if things were to go horribly wrong.

This all meant that we had to take extreme care about production implementations. We created template sets on CDs, and passed them to a test organisation, who tested. We then passed more templates to the production managers, who signed them, distributed the templates, and managed design refreshes and other actions required, and checked that it had all happened properly. The idea was that the development team didn't do such tasks themselves and could not be in a position where they could.