I don't know — frankly I think it is because application software engineering suffers in comparison with the advances in computer technology, or any technology where the goal is smaller, faster, better, cheaper ways of doing the same thing. Application developers are being asked to do new things in fundamentally different ways, to be visionaries, standing on hills with shifting sands, expected to completely rewrite process models for things that have "always been done this way" for decades.

I purchased multiple copies of Frederick P. Brooks' The Mythical Man Month 2nd edition — the one with the essay "No Silver Bullet" in it — to toss at over-optimistic werewolvesmanagers who howledwandered in my direction during my 31+-year career.

The ultimate solution by management was to reduce the apparent investment in applications programming by forcibly shrinking the centralized IT budget and firing most of the expert programmers.

The coding was still done. Only it was done under the table by amateurs — and "amateurs" in the worst sense of the word.

Oh, well. I've got a decent pension out of it.

Every up and coming young pup easily picks up the general idea, from a quick look around, that the state of software development is a dreadful mess. Then they become slightly familiar with their first cool technology -- language, development environment, methodology, what have you -- and get to see it work well in a limited, controlled environment. It takes either more experience (that they don't have yet) or more insight than most people possess to realize that the world in general is more complicated than the one case they had, and their spectacular success with their magic bullet technology will not be universally reproducible.

It's just human nature. Once a human has seen something work at least once and not yet seen it fail, they have to be much smarter than the average human not to assume that it will work every time.

The people that believe in the magic bullet probably don't know computers very well. Or do they? Sun touted Java as its magic bullet: "write once, run anywhere." Well, not quite.

There is a magic bullet, actually. The real problem is that that aiming it so that it gets the hard drive, CPU AND manager is hellaciously difficult.

There's no "magic bullet" in the sense of a procedure that can be imposed that will make things automatically work better. But there are two words that have a lot of magic in them if properly applied:

Document.

Test.

You need a decade, maybe two, to really have the perspective to understand that the zeal and promises of the new always tend to overpromise (except the very rare occasional new thing that actually *is* revolutionary). The same applies to all fields... finance, politics, law enforcement, the sciences, even the various arts like painting and fashion. The short lived ones are fads, the longer ones trends.

The decade is probably necessary, but it is not sufficient. I'm dealing with this sort of thing at the office lately (i.e. more so than usual). People with 2, 3, and even 4 decades of programming experience are swooning at some 'new' concepts. I, however, am mostly appalled.

Oh GOD yes.

Unfortunately, it's been my experience that the same type of people who don't document or test their code also believe in magic bullets. I don't know whether these people are lazy or just inexperienced.

Signs of that tend to be statements such as, "this code is self-documenting", "I'll test the code as I write it!", or my personal favorite, "I'll run this profiler and optimize all of my code!".

BTW, I didn't know you were also a programmer. We have to chat at the next Dorsai Thing. :-)

Jabberwocky has it pretty close. The problem itself is so large and complex that one can't understand it without many years getting your hands dirty with real work. Instead, non-progremmers grasp at a simplification, treat it as truth, and then proceed into the swamp without a clue what dirt is.

There have been significant improvements in programming methods over the years. Block-structured languages. Structured coding. Modularization. Interpreted languages. Objects. RDBMS. Client-server. The unified method. Extreme programming. IDEs. Each has been the magic bullet of their generation, and each has brought some actual improvements to the landscape. (IMHO objects are especially oversold, but that could be an over-exposure to o-o bigots the last few years.) And each solution, when applied to the appropriate problem space, shows real benefits. The key word is, of course, 'appropriate.'

In the real world, some of these actually work. In fact, most of them actually work. But none addresses the entire problem space, in spite of the claims of their proponents.

There's also a second effect. When there have been real improvements in software development, the field rushes ahead and addresses more and more complex problems until the improvements in productivity have been eaten up by the increased complexity of the problems addressed.

And it's understnading the complexity of the problems that is the biggest bottleneck. I can't recall how many different times I've been asked to do a 'little' job of automating a given task - only to find out the task was signficantly more complex than anyone understood.

I'm good at automating system administration. Really, really, good. But that's because I've been doing system administration for almost 30 years, and am a decent programmer to boot. Not a great programmer, merely an adequate programmer. But I understand the problem space really really well. That means my solutions actually work. It also means that sometimes I step back and say "that can't be automated." Or it can't be automated in the way that someone thinks it can, or to the degree that they desire.

A tool is just a tool. The hard part is using it.

You're totally right that the real problem is the exploding complexity of programming in general. Several generations of new technology and methodologies have come along, each making the tough problems of the previous generation so much more tractable that they do seem to be magic bullets, but each time, a new generation of problems evolves to overwhelm us programmers.

This evolution in complexity is the reason that programming has gone from something I loved and considered myself really good at in the 1980s to something I do because I have no other marketable skills and don't like very much today. Twenty years ago I had to deal with problems that I could understand, and the time it took me to understand the problem and craft a good solution was well within what people expected. Today, to be a good programmer you need to be able to dive into a million line code base, understand how to locate other people's solutions to something like the problem at hand in the code base (found in huge code libraries and on the web), and shoehorn those solutions into the code base in far less time than it takes me to even find and start to understand the third party solutions, much less how to glue them in without rewriting either the third party solution or the code base. I'm damn good at writing new code to solve problems; I'm lousy (or at least a lot less good) at adapting someone else's solution to almost my problem without rewriting it.