Some Vexing Glitches

In preparation for pre-launch (alphas) of the Dynamic Forms tool at NearState, we've been trying to get a web-site thrown together which can host the basic content we need. Gabi designed a nice logo for us, and the intent has always been to animate the logo on the web-site. After some extended R&D :P I finally managed to get some decent animation going with cleaned-up SVG exported from Illustrator.

As with the main products, we use Chrome during development, and cross-browser test on Firefox, Safari and IE (and Opera if we're feeling energetic). SVG has always rendered fairly well, although it had some nasty glitches in IE <8. In fact we have a detection for those browsers and use a static image instead. 

However, since ~Chrome 18 (I think it was) we now get a wierd flicker when animated elements cross over SVG text elements. <sigh /> And that means more hoking around in the markup to try and fix it. 

I know there's the option to switch to Canvas but is that really the best option? SVG still looks better on older browsers, and it's so much closer to what Gabi works with as a designer. Canvas is trendy, so I suppose that means the browser vendors will watch their implementations more closely. 

Innovation. Pah. It's all fine until you're doing it at 2am. Unless - sometimes at about 2:38am when magic happens. 

gbL.jsMop >> github

Well, I've finished initial work on the message passing framework we're using at nearstate for our F1 forms project. (http://www.nearstate.com#f1dynamic).

The framework is called gbL.jsMop (i.e. message-oriented programming).

More here: http://goofballlogic.github.com/gbL.jsMop/

Ted Dziuba is not cancer

But I still don't know if I really like him.

From his infamous post on node js, I proceeded to many other entertaining and flowery articles. Worth a read even if you end up thinking he's probably just another obnoxious American prick (which wouldn't be very fair of you if you've never met the guy).

One post that has stuck in my mind is one about process and software development methodology (and the virtue of almost-not-having one).  I think he may have a point here. I know he howls and whines like a starving dog in heat but he's right - much of this corporate bs "standups", "backlog lists" etc. really does not work very well. It might work ok in an idealised environment, but I've yet to see e.g. a Scrummaster run "proper" scrum, or even a variant which preserved any of its virtues in a corporate environment. At least in my time in London... It doesn't matter if it's a startup, a bank or an educational firm, large or small, they all end up relying on meaningless process because it lends structure, but that structure itself has no further meaning and uses up large amounts of time and emotional energy.

The same is true of mountains of unit tests and associated mountains of endlessly injected services into classes that would be small, simple classes in the absence of trying to shoe-horn TDD into the mix. Don't get me wrong - I love a bit of TDD, and I feel safer with some of my projects knowing it's there - but not usually in corporate projects. In those situations, you have a huge solution drowning in endless layers of redirection, specialised handling and most importantly - endless mapping of one layer's entities to another layer's entities - all of which, I suppose, should be TDD'd. So you write reams of tests for mapping entities (yes I know about automapper) which represent logic which shouldn't be there in the first place. (yes I know about BDD). All the unit testing in the world doesn't compensate for a lack of good architecture, and a lack of understanding of the virtues of small, autonomous systems communicating with messages (as opposed to a large stack of crud).

Why do businesses keep building these crap systems? Is it just a lack of bravery to separate things into meaningful discrete units? Are we that afraid of deployment? Is it too much to ask that "the website" isn't just one thing (or two if you count the database)?

Well, I'm not up for it any more. I will be asking about architecture in all future interviews for work I take on. What I want to know is - "how do you divide your systems up into meaningful subsystems?" "how to those subsystems communicate with each other?"

What flavour of compromise?

So, dear Bob, I think the answer is fairly simple. If you can’t build it all and you can’t buy it all, you simply have to find a compromise. The only question is what flavour that compromise should be.

The problem with many compromises is that they occur in the wrong way – as the accidental outcome of a failed effort to either buy all or build all.

To make choices which are not accidental requires that we open our eyes and accept the reality of the need to compromise, and on this point both developers and architects alike struggle to get real. Without the ability to put our minds in neutral and to analyse the situation from a point of rest, there is a huge chance that we will assume a silver-bullet approach to technology can work. Technologists get excited about technology because it’s powerful. We understand the secret potential inside these machines and we yearn to see that potential realised in the lives of the people we work with. Any whispered message offering the hope of quick realisation (rather than the more usual drawn-out frustration) is instantly compelling.

So perhaps the better path is to see that for any given problem, the likely solution is a combination of buy and build. Not the horrific buy-then-customise option which is so often the way things work, but instead, analyse the problem in terms of complexity, buy the complexity, and build the rest of it.

If you’re in the business of producing cars, buy the engine and the chassis, and possibly the fiddly electronic bits, then make the rest of it.

There’s a reason behind this apparently arbitrary designation. It comes down to knowing what our core values are as producers. If I am a car producer, the equipment involved in the manufacturing of an engine and/or chassis may be quite distinct from that of manufacturing the seats and body. In order to concentrate excellence, focus on doing one or two things really well, and outsource the rest. That way you can continue to add true value without diluting the excellence of the other components.

This can be applied using a workflow as follows:

1. For any given build, when you break the build down into its component parts, break them down into components which are of roughly equal impact to the end-user. Or, to state it another way, each component you identify should be of roughly equal business-value.
2. Decide on an aspect of the project which usually is the most painful (for most people this will be either "effort” or “complexity”, but it may also be an aspect which is simply outside of your core expertise (e.g. “user interface”))
3. For each component, assign a pain rating (3 – high, 2 – medium, 1 – low, 0 – none)
4. Try to buy components for anything 3 or 2, try to build anything 1 or 0

Bob's dilemma

I know what Bob means. BizTalk is, no doubt, a huge powerful beast which I know can be a really useful tool for traditional EAI. I suppose where it falls down is as a backbone for your entire service architecture. I don't think it was ever architected for that purpose. I think what's happening is that Bob is starting to think of things from a business-as-usual perspective. It's hard for me as a developer to think that way, but I can understand where he's coming from.

The dream of business technology architects is of this world of smooth-running, purchased off-the-shelf systems which they know will work (or be fixed) because the internals of those systems is Someone Else's Problem. As the teacher once said:

An SEP is something we can't see, or don't see, or our brain doesn't let us see, because we think that it's somebody else's problem.... The brain just edits it out, it's like a blind spot. If you look at it directly you won't see it unless you know precisely what it is. Your only hope is to catch it by surprise out of the corner of your eye.

The technology involved in making something properly invisible is so mind-bogglingly complex that 999,999,999 times out of a billion it's simpler just to take the thing away and do without it....... The "Somebody Else's Problem field" is much simpler, more effective, and "can be run for over a hundred years on a single torch battery."

This is because it relies on people's natural predisposition not to see anything they don't want to, weren't expecting, or can't explain.

I don't think this is all about wanting to avoid responsibility either. And it's not really about control. Yes, it would make things simpler to be able to control everything, but it's more than that... There's actually a sense of embarrassment or disbelief about how hard it is to get past the basics. I think these architects are really longing to do something wonderful with their technology, but they just keep getting hung up on all the detailed problems that come their way. Not least among these is the politics of dealing with all the people involved. It requires a certain amount of trusting the unreliable developers to create a custom built system and that just seems like a poor way to powerfully execute on the task at hand – it’s got way too many points of failure. Technology should be simple, sleek, solid – not flimsy and byzantine and likely to blow up if you look at it the wrong way. Technology should be built by Buddhists…

The other thing is – and an architect knows this better than most – often it’s not what inside that counts. Good solutions exist which have an amazingly sophisticated and beautiful interface, with some one inside peddling a bicycle to make it all happen. I think architects instinctively feel that in order to get a really satisfying solution, a sacrifice is required, which is to live in the knowledge that inside, everything is held together with tape….

So I understand – I really do get what he’s saying. But I suspect that he’s still way off base here. There is no way I can look at BizTalk and say, “yep – that’s the ticket to a successful enterprise. Just run everything through that box and everything will be ok.”

So what advice to give? I suspect another post is in order. The problem is pretty well described, but I still don’t see the answer. I understand why “just build it all yourself” isn’t a viable option, and I can see why “just buy it all” also doesn’t work.

And it gets worse!

Because I didn't blog on Tuesday, December 13, 2011

So I just got off the phone with Bob! He's worried that his soul is destined for damnation because he found himself thinking maybe Biztalk would be a good idea.

I paraphrase:

"I remember thinking when I first worked with Biztalk that it was the biggest pile of s**t since I tried to implement .NET SSO using Tiovli. And now, I find myself thinking.. hmm, yeah, we could use Biztalk here - avoid having to do some custom development. What the hell is going on? I know Biztalk is rubbish for anything nuanced, and yet, in this new position, it suddenly makes sense. But... I'm sure I'm missing something. It's like my soul has been corrupted."

Mwhahahaaahahaaa....

But seriously that's not good. I need to have a hard think about this. How to save Bob's soul?

What happens when a developer becomes a manager?

Because I didn’t blog on Monday, December 12, 2011

A friend of mine has recently moved into a new position at work (insurance) where there is little day-to-day development, and more focus on communication with the business about strategy and e.g. PoC work for frameworks and vendor platforms.

He says that he can already feel an internal change happening. Things that he observed with horror in his immediate manager he now finds himself doing with increasing frequency.

So I asked him a bit more about his immediate manager

He said that his manager was recently made the director for all of IT for all of Europe in their section of the company. It’s not a huge company, but it’s still quite a big role, and this manager is moving directly out of a architect role (having shortly before moved out of a senior development role). He says that he used to see his manager analysing things with a “wrong-headed” approach. “He just doesn’t seem to engage with the idea of a technology on a level which makes sense. When he looks at an idea, he’s thinking about it in the most horribly over-simplistic terms. He’ll make judgments about things which are based on completely flawed understandings of the basic concept.”

To which I replied, “But, isn’t that just a difference in perspective?”

“Yes, that’s what I’m saying – I am starting to see that attitude in myself now and it's due to the perspective. I can tell that I’m thinking about components in larger blocks, and that things like efficiency and uniformity is becoming more important to me than the technical detail because those things seem like a waste of time if I can just buy off the shelf. But I remember that this was a big problem when I saw how my manager responded to ideas in the past.

If I remember correctly, I perceived my manager making decisions that would produce what I thought of as very sub-standard outcomes (in terms of client experience and technical soundness). I was being asked to be a developer when, in reality, there was no real inclination on the part of the manager to develop anything. For him, development was the most painful, risky option, and now that I am moving into that kind of role, I understand exactly how he felt. But it still bothers me that I am betraying the reaction I had.

Perhaps the problem is that we just don’t make a clear enough distinction between an off-the-shelf product and the decision to develop a custom solution (or the boundary between both). It makes me feel even more strongly about the dangers of the “middle road” which is to buy an off-the-shelf product and then customise it to the business’ needs.”

My next question: “So, can ever see yourself justifying the development of custom solutions again?”

“To be honest, even if an off-the-shelf component is less than perfect, that now seems preferable to me than custom development. I think that there’s a role for development in companies, but it should be considered very, very carefully. I think that there truly are gaps in the market, where you still have to build it yourself. The main problem is that I think we under estimate the cost of developing custom solutions to the level of maintainability and operationalization required for it to be the right choice. I believe that you should develop components which are less than perfect only when you can prove to yourself that there is time and capacity to do an really, really solid job of the development effort.”