As the list of things that we support has grown, the number of servers has grown, and our documentation overhead has grown. Most of this documentation is in a relatively well-maintained mediawiki instance, but there are various parts of this solution that aren’t ideal. Namely that most of the documentation that we need to be able to generate for inspection on a moment’s notice is in manually generated lists, and is usually also pretty out of date. The need to document things like ‘who is served by, owns or knows the most about this application or server?’ has increased at the same time as the categories for each of those have broadened and blurred into one another. A good example is that two years ago, we supported two home-grown web applications deployed on one server. This year, we support upwards of ten web applications in four or five different frameworks deployed on three servers with a separate database server… and that’s just stuff that’s home-grown, not things we run in Drupal or Plone or some other framework.

The out of date problem has been partially solved by a quick run through the database with an out-of-date template and some snarky emails to the responsible administrators, but a longer-term solution that made it easy to extract the data needed to be put into place. Looking through the available mediawiki plugins revealed the Semantic extension.

The general gist of the plugin is that the data you need to extract is already in the document and just needs to be tagged as such. In our environment, especially, there’s tons of places where lists need to be generated — list of servers supported by a person, list of servers running which operating system, etc. so on so forth.

The nice thing about the Semantic Mediawiki extension is that it’s freeform. We have some servers that are maintained by one person, some servers where it’s preferable to contact one person over another, and some servers that are collectively maintained and used by as large as six people. The nature of semantic tagging means that we can put as many [[maintainer::kkatzke]] tags in as we want to, and then place them in the document in ways that are readable to a human as well as a computer.

The inline queries make it possible to generate lots of interesting lists right off the bat. I put a template in everyone’s user page that used the {{PAGENAME}} variable as an element of the inline query, and we can suddenly see what servers a person is responsible just by going to their User: page. By using the Special:Ask page to rite a query of the pages tagged with Category:server and display ?maintainer, we can extract the current maintainer information for either a report or just to know who to bust out of bed at midnight.

Now all I need is a way to keep the admins to keep it up to date. Since this approach will be easier than the current manual list templates, that should be about half of it… for the rest, I’m looking into crontab-activated cattle prods that will activate one week before campus IT reports are due.

Secret Driven-Development Acronyms

Proposed Subversion commands

With both, see the comments for extra user-added LOLz.

Seems people are already spooling up the laugh tracks for April Fools day.

I wish someone at work had read this before we implemented Zope/Plone. Sure, it’s nice to be able to train stakeholders in one system and have them maintain content on multiple sites, but the cost in systems support alone has outweighed anything we may have gained. I think we’re seeing a resurgence in statically oriented content management systems, aka “Site Compilers,” e.g. WhisperCMS, Webby, Nanoc, and RubyFrontier … the latter three via Daring Fireball.

• A few posts from Jason Santa Maria – A New Day, and the five month later Progress Report. Jason just broke things off after four and a half years with Happy Cog, but is still working on A List Apart.
• In the same vein (and author), Jason’s article from 24 Magazine on Making Modular Layout Systems for the web.
• If you’re going to go through all that work, it makes sense to use the magic of sIFR to get the typography correct.

Given my limited experience (only eight years of working and one failed startup), typing up this disagreement feels a bit sophomoric. On the other hand, the entire focus of my career has been on setting up systems to manage constraints in a seamless fashion. In fact, I specifically work to establish controls in places where they can be automated and seen as goals instead of constraints. Attitude is important when you’re dealing with artists.

Defining an “Artist”

For the sake of this discussion, I’m going to define an artist as someone whose purpose for doing something is entirely in the heart, and whose compensation for performing at task is either entirely or primarily intangible.

Yes, you’ll find artists at all levels of corporations and organizations without regard for size or mission. You know you’re dealing with artists when you see them put that extra little bit in to make a project successful — and then they go home and continue to build out their knowledge and their body of work, and they’ll blog / tweet / contribute to open-source projects, and otherwise make their work indistinguishable from their life.

Organizations of Artists

Let’s take a step back and look at it from an entirely corporate point of view.

As companies grow they invariably get more such checks, either in response to disasters they’ve suffered, or (probably more often) by hiring people from bigger companies who bring with them customs for protecting against new types of disasters.

I would replace “grow” with “age” in the previous paragraph. Companies can stay the same size but integrate ideas from all over — conferences, employees that read blogs, new hires, training, regulation requirements — much the way that bacteria transfer genes that increase antibiotic resistance between neighboring bacteria. The best, most subtle ideas spread in a somewhat viral nature.

What do organizations made up of artists have in common? Generally, they create a lot of emotion. Look at Apple’s branding and marketing. It’s all emotional.

Placing Controls in the Real World

Case in point: I do volunteer work with the Impact Animal Foundation. Volunteer work with a noble cause like a no-kill, 100% foster, limited intake adoption program can leave people feeling good, but it can be extremely exhausting. Burnout is a huge problem for both the artists in a corporation and volunteers in an emotionally involved charity.

To keep people from getting burnt out, the organization has some pretty strict rules on what kind of animals they’ll take in, how many they’ll take in, how many shifts a week someone can work … all common sense stuff. They also have a process for making sure a home is right for an animal that involves people trained in animal and humor behavior analysis, a home visit, “play dates” with any other animals in the house, and a contract that specifies that if the owner can’t take care of the animal, that it comes back to our rescue and not a kill shelter.

See those bolded words? Those are the controls. Just as Paul pointed out, all of these policies also come from a situation when either the organization, an individual volunteer, or an animal was hurt in some way. They may cause the organization to not adopt an animal even though there’s a willing and loving home available. On the other hand: They ensure that the organization will continue to be successful even with the emotionally charged environment that it has to operate in.

Making Controls into Goals

The primary goal of the animal rescue is always met:

1. volunteers go home feeling good at the end of the day
2. every single one of the animals that the organization adopts out are happy, safe, and loved for the rest of their lives

Could we help more animals if we didn’t limit who is taken in, or were more free with adoption procedures? Yes. Could we always know, beyond a shadow of a doubt that the animals were in good homes? No. We’d lose fosters right and left because they’d hear horror stories of where their beloved temporary family members ended up. Could we always be sure that we weren’t stressing our network of fostering volunteers if we took in every animal that came our way? No, we’d lose fosters left and right as we placed unfair demands on them. The organization is based on it’s fosters. Without fosters, there is no animal rescue. The controls on animal intake and placement preserve the organization.

In the software world, a company’s based on it’s customers. Releasing broken code too quickly will send customers away. Releasing updates too late will allow competitors to get in the door — and will send customers away. Without customers, there is no company. Controls on software releases exist to preserve the organization.

If you’re doing it right, your artists won’t see constraints, they’ll see goals. The goal of the animal rescue is to take in the animals that are in great need (i.e. will be put down if they are not taken in) and are adoptable without an inordinate amount of work. The goal of the software company is to release software with no bugs on time. These goals serve controls, but end up serving the customer. The benchmark of a successful adoption placement is a completed application, a good home visit, and a happy union. The benchmark of a successful software release is passed tests and timely delivery. These are challenges suitable for artists.

Artists are competitive in ways that you wouldn’t otherwise suspect. Challenges add spice to what might otherwise become boring and routine.

Keeping Goals from Slowing You Down

At big companies, software has to go through various approvals before it can be launched. And the cost of doing this can be enormous—in fact, discontinuous. I was talking recently to a group of three programmers whose startup had been acquired a few years before by a big company. When they’d been independent, they could release changes instantly. Now, they said, the absolute fastest they could get code released on the production servers was two weeks.

This didn’t merely make them less productive. It made them hate working for the acquirer.

Yeah. Two weeks is ridiculous. If testing takes that long, well, you’re doing it wrong. No testing at all (immediate release) can also be a case of you’re doing it wrong — if you make a change that destroys production data, it could take you weeks to untangle … or worse, it could cause the permanent loss of that customer’s particular bit of data. Changes to production do need to be made carefully and tested. When you’re in “private beta” or “alpha” stages, customers may not expect to have their data be ‘permanent’… but once you reach a public beta release (especially if it’s backed by or been acquired by a large public company!), you’re held to a much higher standard.

Any place where procedures limit the artists from releasing needs to be evaluated. The only constraints on the artists should be completed and met BY the artists. Release testing is a good example. There was no release process when I arrived at my current job. Putting one in place was a headache until the (very passionate) developers finally “got it” that they could meet all the testing goals themselves, and they get the same sense of accomplishment by meeting them that they do from releasing an update in the first place.

Our developers can complete every stage in the production release process themselves except for the final release.

1. Write Code, commit to Subversion.
2. Write (and pass) Unit Tests.
3. Write and pass Selenium tests
4. Tag a release in Subversion.
5. Do a “mock rollout” and write out any special instructions, such as database structure changes, for sysadmins.

When the release needs to go live, the Sysadmins quickly execute the above tasks. The data gets merged, the tests get run, and it gets rolled out. If the artists have done their job well, a change can take less than five minutes to go from development, through stage, and to live.

Everyone Ships

In our organization, it’s not the artists (developers) that ship — everyone works together to ship. The artists can clear it all the way through themselves. There’s a check/balance with the sysadmins. Ultimate responsibility lies with the managers. For a web development shop, we’re pretty large — six developers, two sysadmins, and about twenty to thirty projects or platforms.

I’m sure the same type of thing happens at Apple. The focus is on the artist, but everyone else is there to support the artist. In the case of the new MacBook design, a design engineer probably came up with the idea of using a monocoque frame. Did he submit the design to the purchasing department for them to find manufacturers? No, he probably either developed himself or went out and found a manufacturer capable of developing the processes they eventually used. Or he took it to his boss, who took it possibly up to Steve himself, and as a company Apple went and did it. Artists create, leaders lead, everyone ships.

Changes? They’re easy to sell, because they’re cheap. Processes that are streamlined so that they can be “pre-cleared” or “fast tracked” through? They’re efficient. Any restrictions? They make sense, and they’ve been taken to the leaders for clearance or confirmation. You can easily release in a few minutes even with controls in place. Letting artists ship does not mean throwing out your controls.