/dev/spout

What is Loose Coupling?

2008-06-16T01:03:50-05:00

I see many people throwing the term "loose coupling" around without understanding it. Many use the term without reference. Worse, there seems to be a sophomoric misinterpretation of it which can be summarized as "tight coupling is always bad".

Despite the fact that speak about loose coupling, there are precious few good descriptions of exactly what it is. Fortunately, Wikipedia (as of the writing of this blog) gets the definition right, defining loose coupling as "an approach where integration interfaces are developed with minimal assumptions between the sending/receiving parties, thus reducing the risk that a change in one application/module will force a change in another application/module". The words I emphasize are "minimal" and "risk". The word minimal cannot be replaced with "no". Also, loose coupling is is about mitigating risks of change, and you must not gloss over this. Even Wikipedia could explain these points more.

To really understand loose coupling, we must first acknowledge that as designers, we are essentially gambling on the types of changes our system will be asked to adapt to in the future. We form a hypothesis about cost and likelihood of changes to various design elements, and we split them into two classes: stable or fluid. We expect that the stable elements are unlikely enough to change over time that as an economics question we're better off assuming they won't, while the fluid elements we draw the opposite conclusion. Sometimes designers make these hypothesis explicitly, while others make them implicitly. Different designers can look at the same problem and draw different hypotheses about change, as predicting future requirements is not an exact science. Once we write code based on our hypothesis, we often say that the fluid elements are implementation details, while the stable elements are part of the design contract or interface.

Here's my definition of "loose coupling". Concrete integrated elements are "loosely coupled" with respect to a hypothesis about the risks of change, when the stable elements under the hypothesis are relied on by the interface design contracts, so that the hypothesized changeable elements are expressed only in the implementation details. We typically introduce tight coupling of stable elements purposefully so as to tolerate specific kinds of fluidity among the implementation details. The notion that tight coupling is always bad, is nonsense. Tight coupling to a few things that won't change is generally good.

To illustrate our definitions, let's consider an example. What does coding to interfaces in Java achieve? We introduce an interface so that we can switch out the implementation behavior. We are relying on the methods of the interface and their arguments and return types not changing. If we expect to actually have multiple implementations, this is clearly a good idea. If we never get a second implementation, we just engaged in over engineering, as we spent time creating two artifacts (the interface and it's lone implementation) for a benefit that never materialized.

Consider another example: when we use a SOAP web service interface, we couple to the HTTP protocal and XML standards, the SOAP and WSDL standards and schemas, as well as a homegrown XML Schema for our domain, and the operations and faults defined in the WSDL. In return, for all this tight coupling, we can remove any dependence on any platform or language specific constructs, let alone any specific implementation classes. It's usually a pretty good bet that things like HTTP, XML, SOAP, and WSDL change very little over time, though I look forward to WSDL 2.0. It's predecessor was a little over-engineered, offering too much flexibility at the expense of complexity.

Every time we create a solution, we gamble. Over time, our change hypothesis is repeatedly tested as our code is maintained and implements new requirements. We can compared, with the benefit of hindsight, the kinds of changes we've actually performed with how we predicted. There are four possibilities...

predicted change, observed change. Good.
predicted change, observed stable. This is the "over-engineering" mistake
predicted stable, observed stable. Good.
predicted stable, observed change. This is the "tight coupling" mistake.

There are costs associated with each mistake, and when you place your bets you have to weigh the likelihood and the expense of the above possibilities. The cost of having to make changes across tightly coupled elements are well known. The change is invasive, it may take a lot of work to determine everything in the dependency chain, it's easy to miss dependencies, it's harder to assure quality, and deployment are harder, etc... . On the other hand, the costs of over-engineering are bad too: code bloat, complexity, slower development, etc... .

Let's examine ordinary class encapsulation using the approach of this article. Encapsulation is a loose coupling technique applied to class design in an object oriented language. Classes define a public contract to their collaborators via their publicly declared methods. Regardless of whether the class implements a formal interface or is directly called by client code, the method arguments and return types will be depended upon. In fact, using an interface INCREASES the coupling to these, as the interface itself, all of it's implementations, and any client who called one of these methods (which is a larger set because of the abstraction) all must be modified if the method signature changes. We should weigh the odds and impacts and choose to expose methods with arguments and return types where your hypothesis is they are stable. So if we expect to always have Customers who have names that are Strings, by all means have customer.getName() return a String. On the other hand, if this might change, introduce a CustomerName object and/or interface to encapsulate the implementation details of names and expose this rather than String. Is this approach always better? No, it depends on your assessment of the future, and you simply can't make universally sweeping statements that one is better than the other.

As I mentioned, different designers can look at the same problem and assess the risks differently. There are two styles that seem to emerge, and which camp you are in seems to depend on how you weigh the pain of complexity against the pain of refactoring. Some people will not introduce loose coupling until it's clear there is a need for changes to implementation details. These people say things like "the simplest thing that works" and "you aren't going to need it". Others take their best guess based on the available knowledge and their experience and place their bets, sometimes introducing abstractions early. I liken these to the different styles of poker player: some people play tight and some play loose. Neither is "correct" in any objective sense. We've all seen bad examples of taking either too far: the guy who uses several design patterns to avoid relying on the fact that customer.name is a String, or the guy who simply refuses to code to interfaces and has methods that return HashMap instead of Map and so on. Either extreme can be bad. Hopefully now that we understand what loose coupling really is, the reader won't make either mistake.

How REST will neuter Web Services

2008-05-13T01:24:23-05:00

I normally like to blog about things that are, and leave tea leave reading to others. Not today. I've been thinking more about a specific example of solving a problem in the RESTful way, and the solution seems like one that is likely to play out in many different arenas. So here's my attempt to predict the what will happen as RESTful thinking penetrates into some of the harder aspects of software engineerings...

Basically, I think REST is going to neuter web services. Not destroy - just neuter. Instead, the Great Irony is that as RESTful patterns for solving problems of enterprise development are discovered, used, and socialized, the net effect will be to unleash a great revival of custom, not-HTTP, non-RESTful protocols. What!?! you say. Read on.

The particular problem I referred to above is a basic one in event driven architectures. The problem statement is this: How do you create a channel for messages where messages are queued, but consumers have unique ownership of messages they dequeue. This is basically the asynchronous form of a message based load balancer, used to drive (among many other things) highly horizontally scalable "master/worker" solutions. James Strachan has a RESTful solution to the queue problem. I've studied it. I recommend you study it, and read the thread at rest-discuss.

This blog is not about the solution per se, but more about who's looking for such solutions, and what they are going to do with them. ActiveMQ is a framework that created a use case specific protocol (Openwire) for messaging that works really well. They want to add RESTful HTTP to their stack of messaging protocols, which also includes STOMP and XMPP, and hopefully one day soon, AMQP. It makes a lot of sense why they would want to enable HTTP as a transport for messaging, but notice that the effect is to make it easier to use ActiveMQ, which is inevitably going to get Openwire solutions in place, for one simple reason: IT'S A LOT BETTER AT MESSAGING.

There's a really excellent comment in the rest-discuss thread about REST, compared to custom protocols, by Alan Dean. He's just collecting comments from Roy Fielding. The first notes that REST's statelessness requirement involves a tradeoff against custom protocols that may degrade performance, since custom protocols can reduce networking costs by leaving data on the server. The third quote notes that abiding by the uniform interface also involves a tradeoff because generic interface methods generally won't be as efficient as those tailored to a specific problem.

Now think of all those WS-* specs out there. Each one was somebody trying to solve a problem that comes up in complex systems. Instead of having another REST vs SOAP debate, how about a different question: why web services at all? Why NOT problem specific protocols like Openwire or AMQP? The standard answer is that these involve sacrifices in interoperability. Unless clients also speak Openwire, they can't get your information. While many clients may choose to do so, those that don't are excluded. This is not good.

But this is why REST is so appealing. It's the ultimate least common denominator solution. EVERYBODY speaks HTTP. It's even really good at some things, if you use it in the RESTful way. And if you leverage "hypermedia as the engine of application state" and idempotency, even hard things are possible (like building concurrency safe queues). But look at WHO is going to solve the hard problems. It's going to be custom protocol creators who are trying to overcome the interoperability objection to their "best" solution. It's guys like James Strachan who want to get ActiveMQ out there for messaging. Instead of a bunch of WS-* goop, REST will see its solutions come more as solution patterns that interact with a specific implementation toolkit that solves the problem for you. These implementations will also have custom protocols tailored to the problem. I predict most people will use these, secure in the knowledge that somebody who can't speak the language can always drop down to the super-portable RESTful API.

In fact, if you use languages that have good client libraries, why wouldn't you use Openwire over REST for it? Especially when the framework provider is going to hand you juicy client libraries to go with server binaries that "just work". The guys who have to use the RESTful mechanisms are the ones that write in languages that don't get big communities creating client libraries for every protocol under the sun. Your Erlang, Lua, or whatever client may have to interact via RESTful HTTP with the server. System admins and monitoring systems will poke at production servers to make sure they're alive using the RESTful APIs. Generic things will be done via REST.

So this is why I say REST will neuter web services. The super accessibility REST provides (better than SOAP solutions) is one half of a tradeoff. The other side is solving the problem with a highly tailored protocol solution that performs and scales. Folks that want the latter will bundle the former with it as a hedge. The end result, I predict, will be that more problems will be solved using tailored solutions which previously would have been unpalatable because they aren't universally accessible. SOAP won't beat either solution, but it won't lose to REST, it will lose to custom protocols that aren't truly viable until they add a RESTful interoperability API.

RESTful Service Descriptions

2008-04-13T02:48:59-05:00

Over several blogs recently I've looked at RESTful approaches to enterprise integration, somewhat critically. Despite my conclusion that REST inside the firewall is almost crippled by the lack of service descriptions, I do in fact believe that solutions are imminent. Moreover, once these solutions exist and are socialized, I believe it will be game changing within the enterprise. The solutions will not come from the REST community, who (mostly) stubbornly refuse to admit there is an issue. The solution will come anyway, so let's look at where it stands.

Let's start with what a service description must accomplish in a resource oriented enterprise setting. Let's suppose that we stick some RESTful services inside the enterprise ecosystem of a typical enterprise. What do we need to actually describe about them. Well, several things:

Where the URI "entry points" are. Where do you start? If you have resources representing a customer, where EXACTLY are they. If I want to look up customers, do I start at http://myco.com/customers or at http://myco.com/accounts? If you think this is silly because somebody could just tell you, then I say two things: first, even a small company probably has 2000 different entities of interest. URI's are never supposed to change in a perfect world. In the real world, they change all the time, especially for dev and test environments inside the firewall. On the internet you probably use a search engine, read blogs, or get emailed a link. Google doesn't typically index inside the firewall, so this doesn't help. The other options don't scale.
Metadata about parameters OK, so you know where to go to ask for a filtered set of your favorite entity. What parameters can you put into the URL, which are mandatory, what are their datatypes, and what are the allowable values? Once you know the entry point above, you can do a GET request and get back some nice document with links galore to things of interest (as in "hypermedia as the engine of application state"). Great, how do you specify the start date and end date? Do you have to specify the account name? If so, what's the parameter called? Can you specify the maximum record size to return? etc...
What media types are used Can I get purchase orders in JSON? XML? as an ATOM feed? Pretty simple question. You can get this via HTTP HEAD and/or by setting the accepts request header. Sounds like we don't need this in the service description, right. Well, you often have to do gap analysis before you have physical access to a service. You need to figure out things like: When they come on line with purchase orders next month, will they support JSON or do we need to submit a change request? If you have their latest service description, you know the answer. If that document changes, you can ask questions about why things where added or removed because you have something tangible to compare to.
Which HTTP operations can you access In theory, this is what HTTP OPTIONS should answer. In reality, who knows what the answer to HTTP OPTIONS looks like. It should probably be the relevent service description document. So rather than replacing it, HTTP OPTIONS becomes a way of retrieving it.
What validity constraints exist on the media types If you are using XML, what schema, DTD, relax-NG description applies? This is a big open issue. For the same reasons as above telling me to "get one and look" doesn't work. More fundamentally, you cannot infer a global constraint by looking at instances that happen to obey it. This problem has been mentioned by several people: stu, Stefan Tilkov, James Strachan, and Dan Diephouse

The spec that seems to be able to deliver on all this without adding a bunch of extra useless weight seems to be WADL. Here's a nice quick example of WADL as used in Jersey (the JSR-311 reference implementation) to give you a sense of it. It's difficult to see how anyone could object to this: it's simple, lightweight, does the above and little else, and is comprehensible. Your eyes will not melt if you try to read it or its spec. Jersey returns the WADL doc when you do an HTTP OPTIONS on the resource. That's sweet! I'm really hoping this gets more attention. I note WADL is coming out of the Java community, not the REST community.

Another exciting development is to clothe your RESTful web services in the Atom Publishing Protocol (AtomPub). AtomPub is a RESTful application protocol for CRUD operatoins on resources that produce Atom Feeds. To see the power, look at how ActiveMQ is proposing to implement a RESTful Queue. This is a nontrivial problem. If you don't see why, follow the link to the discussion at rest-discuss and read the thread.

The use of AtomPub and Atom directly for publishing to web services (as opposed to the typical use case of HTML based blogs) is an idea that's been getting a lot more attention lately. Unfortunately, as James Strachan points out, it doesn't let you easily associate an XML schema when you use the application+xml media type. But it could, with a minor modification. Maybe this problem should be solved at the HTTP level, by introducing the URI to appropriate metadata (such as the XML schema) as a media type parameter. Either way, it's possible that we'll see an application protocol (like AtomPub) wrapping up RESTful web services and filling the gaps within it's constructs.

Finally, there's WSDL 2.0. which will also support REST. At the moment WADL's simplicity and implementation lead seem to make it more interesting to me, but WSDL 2.0 will have to be supported by all the next gen SOAP stacks, so we're likely to see these going dual architectural style and supporting REST too.

Tilkov on Doubts About REST

2008-03-24T03:36:22-05:00

In a recent InfoQ piece Stefan Tilkov attempts to address doubts about REST. I offer my reactions to his 10 items.

REST may be usable for CRUD, but not for “real” business logic
There is no formal contract/no description language
Who would actually want to expose so much of their application’s implementation internals
REST works with HTTP only, it’s not transport protocol independent
There is no practical, clear & consistent guidance on how to design RESTful applications
REST does not support transactions
REST is unreliable
No pub/sub support
No asynchronous interactions
Lack of tools

In summary, I still think REST contracts need to mature (but it looks like progress is starting). Once this happens, standards and tooling can deal with the rest. It's not clear how REST can avoid the WS-* debacle that SOAP went through. But many of the ten doubts really have been dispelled.

What is Software Architecture?

2008-03-24T00:56:54-05:00

In reacting to my recent blog discussing the Uniform Interface as used in ROA via its reliance on REST, Roy Fielding posted a blog On software architecture. In it, Fielding states his opinion that people "don’t understand the differences between software architecture and implementation, let alone between architectural styles and software architecture." I'm not sure if that was directed at me specifically or not (I suspect it was), but I think it's unreasonable to introduce nuanced abstractions and then complain if people don't adopt them. Like any term of art, the meaning of "software architecture" is open to debate and individuals can propose their definitions and the community of practitioners will socialize what they find useful. But since the point was raised, I'm going to chime in and ask the question "what is software architecture" and I'm going to sketch my own answer. I don't see much value in abstractions above this, as there is too much vagueness at the this level alone.

My job title has "software architect" in it, so I do think I'm entitled to an opinion on what such terms mean. In an enterprise, an architect spends his time deciding what aspects of software and system design should be standardized across the organization and what things should be left to the judgment of individual implementations. Our goal, like that of anyone else in the business, is to deliver value to the company as efficiently as possible. When we decide to standardize, we have to spell out what is and isn't allowed, and we need to defend the value proposition associated with making rules over tolerating divergent solutions. I view an architecture as a set of principles, where a principle is a function that maps the space of designs into a binary space of "yes/encouraged/allowed/true/conformant/good" vs "no/discouraged/rejected/false/nonconformant/bad". When I use or hear discussion of architectural "constraints", I think in terms of such functions.

A business considers an architecture to be "good" if it leads to lower IT costs over time, balancing well between actually shipping working solution with a shorter-is-better look at development time. Since this is evaluated "over time" you have to include maintainability, adaptability, and so on. What makes an architecture "good" is dynamically changing: it depends on the habits and knowledge of the individuals in the organization, and it depends on the extent to which consensus has emerged. These things aren't perfectly knowable, and architectural leadership is as much about communicating and guiding as it is about deciding. I think my concepts here of software architecture align a lot with those of Martin Fowler, as described in his essay Who Needs an Architect?.

There is a classic debate between academics and applied practitioners as to the value of standalone theory. I'm an unapologetic applied practitioner. When we introduce nuances like the difference bewteen "architectural style" and "software architecture", I ask "who are you trying to communicate to?" When most people don't understand the distinction I wonder whose fault Dr. Fielding thinks that is and what we should do about it. At a minimum, I think it's fair to ask "why introduce a second level of abstraction?" Can't I reformulate a REST equivalent as a set of architectural principles in the sense I give above and take any proposed concrete design and say which, if any, of those principles it violates? Isn't the real problem here that we named the abstraction without having a few concrete architectures to justify it?

The use of language as a means to develop common understanding is an important step in the evolution of software architecture. Common understanding hopefully leads to communication about common experiences, and this in turn leads to consensus on what is or isn't a best practice worth standardizing upon. When I adopt language, I do so with this end in mind. I tried to sidestep the double abstraction by writing about ROA and not really about REST other than how it's used within ROA. I did this because over the past year or so, there has been an ongoing discussion influenced by Sam Ruby's book RESTful Web Services. Part of my job is to critically examine all sides of archictural trends. I, along with my fellow architects, need to guide my company on how and when to adopt architectural principles. In order to do that, I need a name for the set of practices we should consider adopting. All I can say is that continued confusion over terminology and concepts is not a good selling point for REST or ROA or whatever I should call the thing we could adopt is.

I note the Roy appears to lump me in as an "SOA advocate". I suppose it looks that way, but I'd like to make it clear that this is "for now" only. Again, what's "best" is a influenced heavily by human factors such as understanding, consensus, industry support, and so on. I'm no SOAP fanboy: there's a whole lot to dislike about SOAP and the lack of simpler solutions for SOA wastes endless sums of money across the IT industry. In my view the race is on to see if SOAP can simplify before some particular REST approach matures enough to displace it. I would not be surprised at all if specs like WADL or WSDL 2.0 help move a RESTful web services approach forward. But lots of REST advocates seem to have a philosophical opposition to these and my message to them is unambiguous: within the enterprise, REST won't matter without these. If nothing changes REST will continue be used to solve the problems of web communities. Solutions like Atom where a critical mass of people with a common problem collaborate on standards and tools will be the only arena where REST impacts. Inside the enterprise where programmers toil one or two at a time under deadlines to integrate the backends of systems in environments where politics exist, SOAP will be what people like me conclude we need to use. Again, I said "if nothing changes". I hope it will, and when it does you'll read about it here. Dr. Fielding reads this as "criticisms of [his] work". That's a glass half-empty view. A gap is an opportunity to improve, and closing this one would make it more useful in the space I happen to work in. That space is a big one, but it's not the only one. I wouldn't be discussing REST at all if it didn't solve some problem well. Please note that my Atom feed for this blog works great and is joyously RESTful.

Finally, I'd like to point out that Dr. Fielding essentially restated my main point when he says "REST constraints do not constrain Web architecture — they constrain RESTful architectures (including those found within the Web architecture) that voluntarily wish to be so constrained." It seems really odd to recognize the value of a set of voluntary constraints without immediately asking how to declare that you have volunteered, so that interacting parties can rely upon it. This of course, is exactly what a contract is. It would be nicer still, and downright useful, if there was an objective way to test that a site that declared itself to have some particular RESTful architecture actually did. In other words, its also useful to audit compliance to the contract.

ROA vs SOA: The Uniform Interface

2008-03-20T22:58:13-05:00

Here's part 3 in my series comparing Service Oriented and Resource Oriented architectures. If follows part 1: Comparing ROA and SOA and part 2: ROA and SOA and Service Contracts. This article will focus on the second major difference between ROA and SOA from part 1: REST's uniform interface vs SOAP's rich operation sets (the other two being contracts and data/metadata differentiation). REST relies on the HTTP uniform interface, and prefers to do so with minimal overloading (ie don't put requested operations as parameters), whereas SOAP provides a rich set of domain specific operations, and requires an invocation POST to specify which operation from an endpoint specific set that is contractually specified.

My conclusion is: REST was supposed to define a set of architectural constraints, but nobody is constrained by the uniform interface and many don't implement it or worse violate it. To identify those resources agreeing to abide is a contractual determination, not an architectural one.

Let's review what a uniform interface is, and then what the ROA uniform interface is. As a simpleton definition, a uniform interface means that for all things capable of responding to a request, the language to invoke the request is the same (aka uniform). This means the language is a system invariant as opposed to something that must be specified per target. This is important because it means that intermediaries can attach to the system and the semantics of any observed requests are transparent to the intermediary. This explaination is my own spin and interpretation. Consulting the authoritative source is (to me, YMMV) like reading a political speech: it says much that is often repeated by adherents but uses too many nebulously defined terms that at the end of the day don't convey any precise meaning. The uniform interface, as described by Fielding, is defined by four interface architectural constraints: identification of resources via a URI, manipulation of resources through representations, self-descriptive messages, and hypermedia as the engine of application state. Fielding later explains: "REST enables intermediate processing by constraining messages to be self-descriptive: interaction is stateless between requests, standard methods and media types are used to indicate semantics and exchange information, and responses explicitly indicate cacheability." Generally, I agree there is tremendous value in having a standard set of operations with externally known, target-independent semantics. The idea in ROA is that all resources in the system can receive GET, POST, PUT, and DELETE. Unfortunately, very few people actually implement these faithfully, and there are other HTTP operations like TRACE, HEAD, OPTIONS, and CONNECT which are even less faithfully implemented. Almost all of the internet fails to implement the uniform interface. Browsers don't actually support PUT and DELETE, by the way. Lot's of resources even violate the normal meanings. As one example, ActiveMQ's is a message broker that allows HTTP GET to destructively dequeue a message. Is this "bad"? or useful?

I hate to come back to the contract question again, but how exactly do I declare that I do faithfully use the full uniform contract of ROA. How do I shout from the rooftops that my GETs are safe? I suppose somebody could publish an atom feed of all "know uniform" sites. The sad irony is that were such a way invented, it would express nothing more than that specific targets obey a particular interface (the uniform one). This is the problem with trying to avoid contracts - it doesn't work. The sad reality is that the uniform interface is not uniformly adopted. It'd be nice if it were, but alas, this is a pipe dream. It'd be nice if people wouldn't spam me, too.

Let's pretend that everybody gets religion and tries to follow the "spirit of the web". ActiveMQ only dequeues on DELETE. Even, then, there's the problem that POST doesn't actually have a standard meaning that can be predicted in a resource independent way. It's the operation with side effects, right? If that's all we can say about it, that's lame. It's also false: there's no guarantee that POST must have side effects on any particular invocation. The REST crowd cries with horror at the "abuse of the web" that occurs when somebody creates a SOAP interface and does a POST with an operation parameter of getPurchaseOrder to return a purchase order document. The offender is often lectured about the HTTP Spec, section 9.3 and 9.5 and the Axioms of The Web, and then referred to The REST Bible. Tim Berners Lee and Roy Fielding's contributions to the sciences and useful arts deserve respect, but only because they try to explain their points without the sophistry of appeal to authority, something lost on most REST advocates. Back to the point: there is nothing in the HTTP spec section 9.3 or 9.5 that implies you cannot embed a "GET equivalent" inside a POST, as SOAP often does. One of the (many) enumerated functions POST was intended to cover is that of "providing a block of data, such as the result of submitting a form, to a data-handling process" and " The actual function performed by the POST method is determined by the server ...". To the Turing machine at the other end of your POST there is no meaningful distinction between such an allowable data block and one that invokes "operations" via a string identifier.

When you look at TBL's document above, you do find this axiom that appears to forbid the practice. His argument for the axiom is that "The introduction of any other method apart from GET which has no side effects is also incorrect, because the results of such an operation effectively form a separate address space, which violates the universality." The axiom of universality in two parts states "Any resource anywhere can be given a URI" and "Any resource of significance should be given a URI." Keep in mind that resources in the semantic web can be "...anything - real objects, abstract concepts." The idea here is that since you are "obviously" just getting a resource, you should give it a URI and GET it.

Unfortunately, to denounce getPurchaseOrder relies on your desires as client rather than what I as the implementer have agreed to. How exactly how are you concluding from the name only, that "getPurchaseOrder" has no side effect? The WSDL doesn't specify that this is true and one of the SOA principles "autonomy" says the other details are encapsulated in a single point of control. Unless I explicitly gave you a commitment (and I didn't), you really you have no such guarantee. How do you know that my getPurchaseOrder implementation doesn't increment a counter somewhere as a side affect. How do you know it doesn't assign a random sales engineer's name from a rotating pool on each request? You want getPurchaseOrder to be side effect free, but that desire is all you have. Here's the problem: functionality changes over time. I may not be willing to commit to getPurchaseOrder being side effect free for now and all times. Even if it is truly side effect free, that's just for today. Will it stay that way? Maybe it wasn't put in a GET so that I can have the flexibility to change the operation's side effects next year when we get serious about Sarbanes-Oxley. The rule has to be that the operation is guaranteed to NEVER have side effects. The "axioms of the web" really don't contradict embedding GET in POST if the side-effect free guarantee isn't established. And that is 100% up to me as implementer to determine. You as client can never verify the call is side effect free, as I don't have to tell you everything I do when I receive your call, and you can never observe "all resources" because of the open world assumption on the web.

In summary, I wish the uniform interface of REST was actually adhered to by all HTTP resources. Even if it was, there is nothing nonconformant with SOAP. I think I could defend the proposition that SOAP is RESTful. Finally,in the real world, adherence to the uniform interface is not uniform, so there is no escaping the need for contracts to declare target specific interfaces.

Hibernate Interceptor Gotchas

2008-02-18T23:15:35-06:00

I'm trying to implement a Hibernate Interceptor to track user edits to an entity. I call this an EditInterceptor and it is similar to the AuditInterceptor that many people have described. I found implementing a Hibernate Interceptor in practice to have many "gotchas". These include:

Problem 1: Doing work in onSave() you discover the database id may not have been created yet (as the SQL insert hasn't occurred yet). Similarly, onFlushDirty() and onDelete() occur before the database has actually changed.
Solution: Process items in onFlush() using Interceptor instance collections to pass them from onSave(), onFlushDirty(), and/or onDelete().

Problem 2: You cannot modify objects in the session you are working with while it is flushing.
Solution: Create a second session in postFlush using the same jdbc connection.

Problem 3: When implementing the solution to #2, you discover that session.connection() is deprecated, but that no alternative has been provided yet. Note that the Hibernate team doesn't practice proper use of deprecation here, as pointed out in the comments to HHH-2603
Solution: Use it anyway, and add @SuppressWarnings("deprecation"). Note that cloning a session from the same jdbc connection this way is safe, even if there are problems with using session.connection() to do ad hoc jdbc mischief.

Problem 4: You're using Spring's Hibernate support and realize the Interceptor instance collections from Problem 1 aren't thread safe.
Solution: Wrap them in a thread local. See this example.

Problem 5: Your updates are getting processed multiple times and you realize onFlushDirty is called multiple times, and read the javadocs and discover flush() doesn't always conclude with SQL syncronization to the database and that onFlushDirty() is still called in these cases.
Solution: Use a Set to collect changed entities and be sure to initialize this set in afterTransactionBegin().

Problem 6: You realize that in onFlushDirty, because of mulitple flushes as described above, that previousState and currentState are changing from call to call as you hit different points where hibernate automatically flushes. You need the true original state and the true final state.
Solution: Use a ThreadLocal map that associates an entity with the first previousState seen for it. Clear this map when a transaction starts and after you flush. Clear the updates in preFlush() and as usual process them in postFlush(), but when you calculate the update in onFlushDirty() do it relative to the originalState from the map, instead of previousState, which is only used to seed the map if it doesn't have a value for the entity.

The Need for Distributed Version Control in the Enterprise

2008-02-12T03:02:49-06:00

I'm looking at software configuration management tools again. My last three projects at work have all used subversion. This was a big improvement over CVS, but I'm growing tired of dealing with needing to push everything back through a central server to share it with others (or to share it between my computers). I'm also finding that the pains of branching and merging really limit the pace of change. These are the problems that some of the new SCM tools like git and mercurial were created to solve. I'm convinced that enterprises need distributed version control.

It seems many people think tools like git an mercurial are only appropriate for large open source projects, and aren't necessarily relevant to the enterprise. I submit that enterprise development teams need the new tools just as much, if for slightly different reasons. Basically, as enterprise developers, we need to write code in a feature oriented way and then group completed features together into releases. There are two approaches to dealing with how to allocate features to releases: you either decide the features in a release up front and work until you are done, or you decide the time frame for a release and pull in the completed features to it and push incomplete ones to the next release. My thesis here today is that the latter method is better, but it really stresses your ability to manage branching and merging, and that tools like git and mercurial are simply better at managing this.

The big problem in software development (as everyone knows) is that nobody can figure out how to estimate the duration of a task very well. If you think you have a solution to this problem, slap yourself, because you don't. The variability in estimating means that doing the "feature scoped" method incurs heavy waste as all features in a release go out at the pace of the slowest one. So most developers finish their tasks well before the slowest and then have degraded productivity until the next release. These problems get worse the more features are in a branch.

The degraded productivity goes away if developers can push and pull features between releases. In fact, the cost of estimating wrongly drops if we can juggle features easily, because now aren't holding a bunch of working features when we estimate wrong.

But you can start to see how all these problems rely on nimbleness in source control tools. Branching, merging, and conflict resolution have to be easy, quick, and simple. At any given time you might have a released branch, a branch under QA/test, and usually several different release branches in development: the next few minor releases and also the next major release. A feature's code can be juggled around among the unreleased versions. For example, if we find a bug, we want to apply the fix to all in process development branches. We create a branch just for this bug, maybe, and merge it's changes into all the other branches. Often we juggle a feature out of one branch and into another because it isn't ready in time. Whenever we create multiple development branches, we have to merge the earlier ones into the latter ones a lot.

The fact is that merging is extremely painful in subversion for several reasons. First, subversion doesn't really think in terms of branches. The directories in your /branches folder are really nothing more than ordinary directories. Worse, subversion doesn't help you at all in terms of keep in track of completed merge operations. Keeping track of merged revisions manually simply doesn't scale - the process becomes error prone, and anyone who has done a lot of branching and merging in subversion has stories to tell on the theme of "oops". In fact, if branching was really cheap it might be better to create a branch per feature. This would let release managers wait until the last possible moment to merge changes: they'd look in the ticket tracking system, see which ones are "ready for merge" and pull them in, or better yet have the feature developer do it, resolving conflicts first. Many of us are used to doing this with patch files, but the problem there is an ordinary diff can't understand directory structures and how things in different places really are different versions of "the same" thing. And there's also the problem of the directory structure itself.

As the number of branches goes up, there will be a natural desire for peer-to-peer merging capabilities. Your peer has a bug? Pull his branch. If two people collaborate on a feature, why make them talk to a middleman to share changes? Especially when those changes may be broken or half-baked? Ever been annoyed by doing a subversion update and getting a bunch of changes you didn't want yet? Worse, ever seen one that broke everything? What if you only pull in things you want, when you are ready and know it's coming? What if half the time you don't even have to do that because the other guy is pulling your stuff first? Now distributed SCM seems a lot more reasonable. What if you have multiple computers (home/work or PC/laptop or both). Wouldn't it be nice to share changes with yourself, even if they don't even compile at the moment? Do it in one step instead of two. Oh, and now you and your coworker can collaborate outside of the office. So go to a cafe and get some work done together.

It looks like Git and Mercurial both have a compelling improvement to offer enterprise development shops. Git is slightly faster and has a more comprehensive command set, but Mercurial's commands will be more familiar to SVN/CVS users and Mercurial has good windows support. Lots of big projects are using these tools: Linux uses Git, Java uses Mercurial. We're starting to see more medium and small shops use these too. Which is better? They're both good, and there's a tradeoff of power vs simplicity and speed vs portability. Pick the first option in either tradeoff and you should look at Git, pick the second and you should use Mercurial.

Pros and Cons of Ant, Maven, Gant

2008-01-26T02:11:04-06:00

I've been thinking about java build tools again. My projects have always ended up using ant and wondering about Maven. There's certainly some innovative ideas in Maven, but there's also some things that scare me. It's funny that Maven's reliance on "convention over configuration" easily predates the Ruby on Rails fad, but the latter gets all the credit for inventing it. There's a reason for that. Anyway, the new kid on the block is Gant, and the way that it's being used in Grails seems really interesting. Gant is a marriage of groovy with ant, which addresses some of the things that frustrate me about ant. Grails adds usability via the RoR style of simple, standard command line syntax. OK, so here's my pros and cons of Ant, Maven2, and Gant.

First I take a look at the old workhorse...

Pros
- Everybody knows it
- Everybody supports it. Ant tasks exist for everything.
- Extensible. Writing macros and tasks yourself is easy.
- Flexible, properties configuration
- Well documented, even by 3rd party tools
- Low tech and hackable: no hidden magic or voodoo.
- Path structures and wildcards are simple, yet powerful
Cons
- Everybody know it, including people who shouldn't touch a build. Ever.
- No standards - few conventions and idioms. Everybody does it different.
- Everybody does it different, even from their previous project.
- XML does not facilitate reuse.
- Limited expressiveness. I have to use ant-contrib to say "if"!?!?! or for <ac:propertycopy name="username" from="db.${db.server}.username">
- XML Sprawl. As XML goes, not that verbose, but nothing is by convention

Then it's chief rival, and the more ambitious alternative...

Pros
- Standardized build lifecycle
- Drop-in reuse of build tools
- Dependency management
- Internal repositories facilitate enterprise code sharing
- Archetype concept allows speed starting project by flavor
Cons
- Hierarchical projects clash with Eclipse (Bug 35973 is 3rd overall with 144 votes today.)
- Extending isn't simple. mojo, schmojo.
- Many mundane tasks are truly cumbersome.
- Archetypes are a great idea ruined by horrible syntax. This is why grails doesn't use Maven.
- Brittle builds. Some say broken by design. I attended a talk once by a maven project guy. His demo didn't work.
- Repository metadata quality. This is Matt Raible's #1 Complaint. Maintaining transitive dependencies is hard and brittle.
- Verbose XML syntax in POMs
- Plugin avialability lags behind Ant
- Endless downloading of jars. Go to lunch while maven "downloads the internet".

Finally, the newcomer with respect for its elders...

Pros
- Ultra-expressiveness of groovy. No more wrestling to do mundane things
- No more programming in XML.
- Smooth adoption path: first use ant's groovy task, then AntBuilder in groovy, then Gant, then wrap in a grail-like runner and scaffolding
- Grails scaffolding is simple, powerful, and immediately useful.
- Leverage the ubiquity of 3rd party ant tasks. Steal the best part of Rails but backed by ant's mountain of 3rd party tasks
- Build logic expressed in groovy classes should facilitate reuse
Cons
- Very young technology
- Groovy is just starting to penetrate the masses. Gant still largely unknown.
- Ok, I admit the only grails-like runner is Grails for now
- Documentation is poor. Really poor.
- Tooling is weak. IDE's don't support Gant or Grails style lauchers
- No standard build lifecycle if you use ant+groovy or gant without grails

Java on the Decline? Think Again

2008-01-21T01:04:49-06:00

It seems that everywhere I look people are taking pot shots at Java, declaring it passe and acting like it's on it's death bed. What a nice thing it is to see unbiased measurement of language popularity from Tiobe. That link is to the January 2008 Programming Community Index.

Here's what their unbiased measurements actually show:

Java is the most popular language, capturing 20.8% of the popularity
That lead increased over second place C (now at 13.9%) by 3.6% over a year ago
Java's absolute popularity increased from a year ago by 1.7%
Ruby's popularity is 2.3% ranking it 11th
Python was the biggest gainer over the last year, growing 2.0%, giving it 6th place
Python overtook long time rival Perl for the first time this year
Visual Basic (now 3rd) was the second biggest gainer, Java was 3rd
Ruby's popularity actually fell 0.17% over the last year
Perl, ranking 7th, is still more than twice as popular that Ruby
C lost the most ground at -1.89%
C++ lost the second most ground at -1.7%
C++ fell to 5th, as it was overtaken by both Visual Basic and PHP
PHP gained 1.25% in popularity over the year
Java is more popular than VB and C# combined (aka .net) which total 15.7%
Delphi at 3.3% surged ahead of both Javascript and Ruby.
Lua ranks 16th, up from 46th
Groovy ranks 31st

If these numbers surprise you, maybe you need to be more skeptical when you read blogs.

Fedora 8 Upgrade: Not so Smooth

2007-11-19T18:51:53-06:00

I'm a fan of Fedora, so I was happy to see Fedora 8 hit the streets. I've tried to move to it on two different boxes: one at work, and one at home. My efforts at work succeeded with one issue (the java one below). At home, though, I've met some serious difficulties. I'm installing onto a Dell E510, which is pretty common. From the volume of "me too's" on the Fedora bug tracker, I gather these issues are widespread.

I've encountered three problems total:

ISOLINUX hangs during boot after loading initrd.img with the message "Ready." See bug 239585
Anaconda hangs in some upgrades when dependency resolution reaches 26% (or 99% for some people). See bug 372011
Existing Sun JRE's have some awt problem with libX11 in Fedora 8, where GUI's fail with "xcb_xlib.c:50: xcb_xlib_unlock: Assertion 'c->xlib.lock' failed.". See Java bug 6532373 and Fedora bug 254144

I've never heard a good explanation for the first problem. I was able to solve it by hitting tab at the Fedora installer menu and adding "edd=skipmbr" to the linux command line. Others have reported using other options like "acpi=off" and a few more others, but these didn't help me.

The second issue has me stopped. This appears to be a bug in yum, and you hit it depending on what packages you are upgrading. I got lucky at work and unlucky at home. There appears to be an installation update at that some people are reporting success with available here. Unfortuantely, using these updates requires either accessing them via the network during the install or mounting a drive with the image on it. My box doesn't have a floppy and I don't use dhcp and my attempts to configure the installer with my networking config haven't gone swimmingly. So I'm stuck. I could probably get this working with some research into using Anaconda updates, but this exceeds my activation energy level at the moment.

The java issue occurs if Java finds the Xinerama extension, at which point it does something broken with locking and triggers the error. People have figured out they can work around it by faking out java into not finding this extension by using a sed script to replace the text it looks for in the binary library: sed -i 's/XINERAMA/FAKEEXTN/g' $JAVA_HOME/jre/lib/i386/xawt/libmawt.so

All in all, this is the most painful upgrade I've had yet with Fedora, which I've been with full time at home and work since Fedora 4. I've had some issues with sound cards and SE Linux, but nothing that basically prevented me from using it. My advice to people is to wait until this stuff is all worked out, unless you enjoy troubleshooting for the sport of it. A good page to look at for the quick facts is the Fedora 8 Common Bugs page.

GCal & Lightning/Thunderbird

2007-11-15T19:31:09-06:00

I'm the kind of guy who adopted linux as my primary desktop years ago. I've been lucky enough to use it at work exclusively for almost two years, and I've used it at home exclusively for five or six years. Windows actually feels really strange to me now. One issue that has been a big pain at work was calendar support. Finally I think I've found something that makes me happy: I'm using Thunderbird with Lightning and the Provider for Google Calendar. A good write up on setting this up is here.

Finally I think I have a good setup that meets all my needs. First of all, my Calendar has to live in my email program, so I can get meeting invitations and hit "accept" and it's on my calendar. There are a lot of email programs for linux that I don't like, so this was a limiting factor for a while. I need one with good support for multiple mailboxes and lots of features. I also want to be able to get to my calendar from anywhere. I use a VPN at home a lot to read my email as I somewhat dislike webmail. I want to not care which box I'm "accepting" the meeting message on. Also, I want to have a browser bookmark to my calendar or be able to navigate to it from somebody else's browser if I don't have my computer.

I actually think this combination has advantages over Outlook (besides being open source and working on any operating system). GCal and Lighning finally solve my calendar issue, so I can cross one of my biggest annoyances off my list. Now if I could just get an open source driver for my ATI graphics card, I'd be very happy.

Yes you do need a Product Road Map

2007-11-11T04:26:08-06:00

You've got to love David Heinemeier Hansson. He may be the new Mark Fleury: always ready to give you his opinion on why everything you know and do is wrong. His latest: a little essay on why product road maps are harmful.

While it's good to question the sacred cows, I'm not so sure that 37signals has really sustained growth over a long enough period of time to deserve any special credibility in this arena. There is an arrogance that often comes to upstarts who produce some innovations and get market success and I suspect DHH is surfing in the wake of his own big head on this one. You would think that if it actually worked to ignore customer input and just innovate, that it wouldn't have taken until 2007 to discover it. Certainly there has been no shortage of really smart people who tried to ignore their customers (I just can't remember any who were successful over the long laul). Yet it seems like the old rich guys all say the same thing: listen to your customers like they are God, because they are.

The problem with DHH's argument is that it is a cynical straw man. He assumes that having a road map and peddling vaporware are the same thing. They are not. The fact that road maps can be and are often abused by "bullet point" management only proves that the abuse is bad, not tool used by the abuse. Good companies almost by definition execute well on their road maps.

I suspect "most people can make do without [a road map]" is the kind of idea that only occurs to small companies who are growing really fast. While you have the big growth, you tend to take customers for granted, though most don't have the nerve to celebrate it like DHH. If you lose a few customers in that situation, who cares: you can't grow fast enough to accommodate them all anyway. Unfortunately, the problem with this is that it never lasts and sooner or later you'll wake up and find you aren't the hot thing anymore, and you'll realize that your customers have no loyalty because you've treated them like crap for years in terms of discussing their needs, desires, and goals. Once the market buzz has died, it's a whole lot better to be eBay than AOL.

DHH compares software product road maps to big upfront software design in the derided waterfall software development methodology. By the way, I recently saw Dan Pritchett of eBay say that they use waterfall. He was almost apologetic about it, during a talk about how they solve their insane scalability problems, but I digress. Even assuming that big design up front is bad for software design, I disagree with the point. Product management and software design are not at all similar. Examine java: it's hardly developed waterfall style. Most of the agile principles out there originated or found early homes in the java community. Yet java has a roadmap created by the Java Community Process and Sun's (and IBM's and BEA's and Oracle's) business strategies. The road map serves as a direction to unify cooperation. It makes the strategy tangible.

Want to know what's going to be in Java 7? It's there in detail. Want to know what's going to be in Java 8? It's there too. See something you like and want to help, click the link. For example, coming is JVM support for for dynamically typing for scripting language support, which will make groovy and jruby substantially faster. Did Sun have a road map when they hired the jruby team? Yes, you bet they did. And jruby is already at par performance-wise with ruby, and Glassfish will leverage this to be the most scalable app server for rails. Did they have a road map when they added ruby support to the Netbeans IDE? Yes, you bet they did. Did Java have a road map when they created groovy? Yes, it's called JSR-241. If DHH really believes that road maps are harmful, I suspect he's happy that there seems to be a road map within the java community of equaling or bettering all of the purported innovations of RoR within the java stack. Sooner or later, no matter who you are and how gifted, one of your competitors will innovate and jump ahead of you in some area. What do you do then? If you are smart and humble, you put closing the gap on your road map and rally your people on how to do it. If you want to lead, you have to tell people generally where you are going, unless you want to attract sheep. Maybe you do, but sheep aren't that lucrative.

Nobody can sustain their business on ground breaking innovation year after year. Do companies that innovate more do better, yes. Can innovation jump start you from where you are? Yes. Is innovation the only thing that matters, long term? No. Once you've innovated, you've got to get and keep customers. Tivo is a great example of an IT company that missed this point. The jury is still out on whether Rails has any long term staying power. I predict Rails is more like Tivo than like the Macintosh, but we'll see.

The simple fact is that people don't spend money on non-commoditized things unless they have confidence in the road map. That includes both customers and investors. I have personally been part of several large dollar purchases of software (the kind will more than one comma in the price), and you can be damn sure we asked vendors about and factored into our decisions the vendor's road map. The road map tells you whether they understand you or not. If you don't want me as a customer, don't have a road map.

Reflections on the Colorado Software Summit 2007

2007-10-28T01:24:41-05:00

I just returned from my third visit to the Colorado Software Summit. 2007 was another great year. This conference is unlike any other: it's just a bunch of developers talking shop. There are other conferences that try to do this, but this one seems to have the best speaker quality by far. I always have a strange feeling when I come back from this conference: it's sort of humbling to in an intimate setting with so many heavy hitters, yet it's sort of emboldening to have so much knowledge poured into your head in one week.

The opening keynote was a "big picture" talk by John Soyring of IBM. I really don't think I can summarize it without writing a whole blog on it other than to say: there are big social forces that require changes to our computing and programming models. The main keynote was given by Simon Phipps of Sun. He's the guy that lead the effort to GPL java, so he's on my "you rock" list. His main point: open source is going beyond main stream to the dominant paradigm. Like it or not, this means legal challenges. Patents and trademark issues matter now and we'd better learn better ways to deal with them.

A big topic this year was REST. I went to three talks on this, which reminds me that I need to finish my blog series on this. It seems that several people believe that REST is or will be the basis of "Web Services 2.0" (as Scott Davis put it). I was really hoping to understand the implications of REST's Uniform Interface better, and I certainly made progress, but I'm still struggling with translating the technical differences between it and than the SOAP approach (which I think I fully understand) into useful advice on what the architectural pros and cons are. Since I'm on record in this blog as saying that service contracts are the big problem that has to be solved for REST to take first chair from SOAP inside the enterprise, I was keen to here talk of these solutions. Most speakers acknowledged that there is more interest in this lately and talked about WSDL 2.0 or WADL. I also got a new idea that I need to consider more deeply: using ATOM for data and embed schema information within it. This is sort of a half-contract: it's uniform behavior on custom objects. I kind of like the idea of a "standard contract" on behavior.

Dan Pritchett, aka "the eBay guy" gave several talks. His talks all concerned challenges that eBay faced as it tried to scale. It's interesting to know that eBay runs on Java and Oracle, with the Java using Tomcat running servlets. eBay's search is comparable in magnitude to Google's (they may even answer more searches). Their search infrastructure is really interesting: they message their data in a "rows and columns" grid with data partitioning and replication, and the replication crosses data centers. They use different indexes for different use cases. Changes are messaged to each node and applied locally by a custom search engine indexer. Dan said they had to go to this model over the centralized index building because the latency and bandwidth involved in pushing index out with all the updates they do was a killer.

I consider mastery of a subject to mean knowing when to sacrifice the sacred cows, and Dan's discussions on ACID vs BASE transactions really impressed me. Since we were all little kids, we've been told that the quintessential example of two-phase commit (e.g. using XA) is transferring money from one account to another in a different database. eBay basically rejected this as unworkable at scale and came up with a rigorous way to meet their needs without it.

I went to a couple talks on messaging technologies. For JMS, ActiveMQ really seems like a no brainer for best-of-breed. It has beautiful Spring integration, features galore including many for HA environments, and lots of flexibility. I just wish I could get one of the STOMP clients to work from python. I still have a couple more to try. I was also impressed by AMQP, which will likely bring us platform independent fully featured messaging. I hope the JMS vendors take note and play ball with it. Gregor Hohpe gave a great look at some coding patterns for event driven programming. He has this way of making you feel like it's obviously dumb that we aren't all using messaging and event driven architectures as the norm.

Progress in the XML technologies is slow, but steady, and a couple sessions hit on this. XQuery is looking quite useful as a way to turn XPath 2.0 into a query language. XSLT 2.0 also harnesses XPath 2.0 and may be a little more grokable. I think I have had to restudy XSLT 1.0 at least five times as it just doesn't "stick" in my brain due to it's unnatural programming model. I'm very optimistic that XSLT 2.0 improves this. Noel Bergman gave a really good overview of the state of SOAP standards and the progress of the WS-I interoperability efforts. He also gave one of the talks on REST, which is probably not a coincidence.

In the "that's just cool" category: both grails and google gears look like good technologies to learn more about. One thing that grails uses that I'm really enamored with is gant. I've been dabbling with mixing in groovy to our build at work and gant takes this one step farther and turns it upside down: you mix in ant ot your groovy build scripts. I've tried twice to drink the Maven kool-aid (but couldn't swallow it), but now I think I'm going interested in with the other direction. Gears is most often viewed as an offline client tool, but I see it as having more impact as rich browser side caching.

Matt Raible gave his updated talk comparing web frameworks. He had some criteria for when to use different classes of frameworks and based on those, I think the project I'm on now at work chose well to go with GWT. Matt made two really useful observations: First, with the rise of high quality javascript libraries (e.g. Ext, YUI, Scriptaculous) it becomes easier to switch web frameworks while keeping view layer skills (and look and feel) intact. Second, your team will get a bigger productivity increase by shutting off email, canceling meetings, and using better computing resources than it will by changing to the next great web framework of the day.

Some of the other cool topics I learned a little more about: Eclipse plugin writing and grid computing. It looks like it's easier to develop an eclipse plugin than it is to think of something that needs an Eclipse plugin.

I'm also already sold that groovy is important, but I picked up some power user examples of groovy. Eric Schreiner did an awesome example of the groovy XML builder where he took a map of item/value pairs and produced an SVG chart in a rediculously small amount of code.

As always, I love this conference as a way to "scan the horizon" for frameworks, toolkits, and solutions that I hadn't heard of before or need to "bump up" my interest level on. Here's my list of "cool things to check out":

Apache Mina, a high performance and high scalability network applications framework
Apache Camel, a rule based routing and mediation engine. Sort of "EIP in a box".
Esper, an event stream processing tool in the Complex Event Processing category.
Gridgain open source grid toolkit with a cool example on distributed unit testing
Some AMQP implementations: QPID (in java) and RabbitMQ in Erlang
EclpseLink: Hibernate may not be the best JPA implementation
eXist, an open source native XML database, with XQuery support.
GANT the groovy/ant build system, as seen in Grails like RAILS but based on java/spring/hibernate
YSlow a firebug plugin that analyses web pages for performance best practices

Rails, the 15 minutes is Almost Up. Meet Erlang.

2007-09-23T11:29:42-05:00

I just bought the latest book on Erlang from the pragmatic progammers. If you haven't heard, Erlang is a functional programming language that has some unique architectural constraints that make it work really well with parallelism and concurrency. Erlang is "hot". This post isn't about Erlang, other than to note that it's what concludes the hype phase that has made Ruby on Rails the darling of fanboys everywhere and an annoyance to the rest of us. This means Ruby and Rails are going to have to stop riding on the wave of excitement by being the latest new thing and are going to have to start impressing people who have already heard about it, tried it, and unlike the fanboys, thought "nya -- needs some work, I'll check back in a few years".

You gets a lot of attention when you mouth off the way the rails zealots have. You get a lot of attention for a while, that is, until then everybody gets tired of you, exposes your flaws, and waits to see how you do fixing them in a climate of criticism. One of the reasons I still like and use java, despite its problems, verbosity, and all that, is that when valid criticisms are leveled against java, the people in the java community don't make excuses, they slowly quietly address the problems. That's a sign of maturity, and Ruby and Rails need to get it or they aren't going to be talked about in five years.

In terms of the Gartner Hype Cycle, Ruby and Rails are nearing the end of the "Peak of Inflated Expectations" and are moving into the "Trough of Disillusionment". For evidence, here's an article by a PHP guy who switched back. That is NOT supposed to happen.

My experience with Ruby is that I read the pick-axe book four or five years ago, before Rails, played with it, liked it, and added a few other books too. I liked ruby's syntax - very efficient for the original writer. I had my doubts about using it in team settings with code maintained over years. I've used Ruby to solve production problems in business settings on several occasions. I never considered adopting Ruby as my "main" language, for reasons I'll expound on below.

Then Rails appeared. Frankly, Rails ruined my enjoyment of Ruby because it attracted so many fanboys and zealots. The folks were making outrageous and ridiculous claims and were so arrogant about how Ruby would be the Java killer. These folks jumped the shark this week, with the posting of Top 10 Reasons Why Java Sucks Ass. Thank you Obie, for being so absurd that no one can deny how absurd the hype around Rails has been.

Now that the industry has played with Ruby and Rails seriously, the problems of Ruby and Rails have become well known. They are:

No high quality app server
No language specification.
Custom runtime VM that isn't supported by any well resourced organization
Open source is not a substitute for open standards at the language or runtime engine level.
You can quickly write code than nobody can understand, and several years later, people have done exactly that
Duck typing doesn't save code, as it requires more unit tests or occasionally blows up on you
Mixins aren't so great: A class's state and behavior specification shouldn't depend on its runtime history. I shouldn't need unit tests to verify my class can do what I think it's supposed to do - the unit tests waste more time than I save. Java's new static imports are a much better solution.
Total lack of high quality enterprise features: transaction support is weak, SOAP support is weak, messaging is weak, caching solutions are weak, directory integration is weak, clustering is weak, multithreading is weak, etc...
What do I do in the real world where my database already exists with 8 years of data in it and I can't design my schema to match the Rails opinions of what's best?
Dynamic typing and Mixins make it nearly impossible to have tool assisted refactoring and good code completion. They are appropriate for small tasks with a low reuse quotient for 3rd party code (aka non-enterprise development).
Opinionated software is great, if you are a follower-the-leader type who doesn't want to think about what's really best to solve your problem.
Overuse of DSL's and embedding DSL's in regular code basically assure difficulty in maintenance. Your DSL's aren't as obvoius as you think. The reason everybody hates XML in java is not because it's XML (they don't complain about XHTML or RSS do they?) it's because they are really DSL's, but at least with editor support.
Ruby is unbelievably slow. Slow. Slow. Slow. Don't say it doesn't matter because the database is slower, because it does matter. Ruby is like 6-7 times slower than Java. 2-3 times slower than Python and Perl.
Reliability and scalability are crummy

The funniest thing about the above list is that the Ruby app servers are so bad that what's going to end up happening is that the Java app servers will bolt on JRuby support and Rails will simply be absorbed into the borg that is Java. If you run Glassfish, Rails is just another java library! That's really, really funny. Nya-Nya to the "java sucks" fanboys! Once they go away, I might actually take up Ruby again -- if the Ruby community shows some interest in solving some of the problems above. Not all of them -- just show me you can listen.

It turns out that it was easier for Java to bolt on the few innovations that Ruby offered than for Ruby to absorb the many innovations that Java has offered over the years. I think the end result of Ruby is that it killed perl, and forced PHP developers to improve their sophistication (see the article from the guy who switched back to PHP). Unfortunately, it stunted the growth of enterprise language solutions in Python. Python is a lot more maintainable than Ruby, by the way. Ruby's effect on Java has been positive. The concepts of reasonable conventions are being used more. Groovy is looking good (better than Ruby except for being even slower). Most of all, Ruby has forced Java framework makers to think about simplicity, and they have. Ruby is helping make REST more mainstream. REST has a fanboy problem too, but with a lot of work could become a viable alternative to SOA. The final benefit of Ruby is that JDK 7 will finally bring closures to Java.

Speaking of closures... That's why I'm reading a book on Erlang. Ruby touts closures as their biggest strength, but if you buy into that, you should be using functional programming languages. Erlang (and Haskell and all the FP languages) are better for the purity of their closures than Ruby. Erlang gets other things right that Ruby struggles with. Ruby threading is a mess. Erlang gets this deeply and emphatically right (even Java should take note). Unlike Ruby, which falters as it tries to move up to enterprise settings, Erlang as been used for 10 years in carrier grade telecommunications applications with many 9's of reliability. Erlang is Rock solid. As in: bet your company on the fact that your apps won't be down EVER.

Do I think Erlang will take over? No. People are starting to expose it's flaws too. But this isn't as fun as doing so for Ruby, because it's origins in telecom make it immune to false euphoria. But, grid computing and multi-core processing are moving IT towards Erlang's strength. Erlang's unique insistence on non-reassignable variables allow it to deliver fault tolerance, concurrency, and parallelism that give it something that will not be easy to bolt on to Java. While functional programming can be added to Java. Though Scala is looking really interesting at bring FP to the JVM and it's performance doesn't disappoint like Ruby/Groovy, Erlang's inherent concurrency cannot be casually added the way many of the Rails innovations could be, and this means Erlang will have a suite spot that cannot be assimilated into the borg.

But this article is not really about Erlang. Erlang is here as a foil, and for it's cultural impact: the biggest feature that Erlang offers at this moment is that it will bring to a close the Rails 15 minutes of fame. Every time some Rails fanboy starts peddling their hype, the approved thing to do is to respond with Erlang. There is nothing sweeter than dropping the credible innuendo that hype is transient and fleeting and is almost over.

And because Erlang is old school telecom stuff, it will come without the fanboy culture. There something really refreshing about the the fact that it's a telecom industry's solution. After Ruby on Rails, it's hip and trendy to be stodgy and uncool, and Erlang fits the bill. Unsexy is sexy. God bless Erlang.