Once, in an ancient past, I studied philosophy for a couple of years. At the time, I could not really appreciate the subject, so I quit and switched to Artificial Intelligence. In retrospect, I think I might have been too young to appreciate it fully. Having a bit more experience in life sure helps.

That said, I don’t think it was a waste of time. I learned a great deal about logic, proper argumentation, and of course the history of philosophy. The latter might seem impractical, but it sometimes does apply, even in the field I’m currently working in.

Plato

Take Plato, for instance. Together with Socrates and Aristotle, he is one of the founders of Western culture. Famous for writing up the Socrates dialogues, he also wrote some works of his own, one of which is The Republic. In this work, he describes his famous Allegory of the Cave, which goes something like this:

Imagine prisoners, who have been chained since their childhood deep inside a cave: not only are their limbs immobilized by the chains; their heads are chained in one direction as well so that their gaze is fixed on a wall.

Behind the prisoners is an enormous fire, and between the fire and the prisoners is a raised walkway, along which puppets of various animals, plants, and other things are moved along. The puppets cast shadows on the wall, and the prisoners watch these shadows. When one of the puppet-carriers speaks, an echo against the wall causes the prisoners to believe that the words come from the shadows.

The prisoners engage in what appears to us to be a game: naming the shapes as they come by. This, however, is the only reality that they know, even though they are seeing merely shadows of objects. They are thus conditioned to judge the quality of one another by their skill in quickly naming the shapes and dislike those who play poorly.

There are many interpretations of this allegory, but for me it has always been about not knowing the true reality of things; we only see shadows cast on the cave wall.

REST

Fast-forward a couple of thousand years, and let’s look at REST, the key design principles of which are:

• Identifiable Resources
• Uniform interface
• Resource representations
• Stateless conversation
• Hypermedia

Resources are identified by a unique identifier (a URI), and you interact with the resources through a uniform interface with well-defined operations (i.e. GET, POST, etc.). Using these operations, you create representations of resources (such as a text/html representation, or application/xml). Typically, these representations contain links, making up hypermedia.

What does this have to do with Plato? Well, in REST there is no one true representation of a resource. There can be many of them, each of which is as important as the other. In other words, the representation is just a shadow of the resource, cast of the cave wall…

Throughout the years, I’ve worked on a number of apps which required user-customizable fields. For instance, consider a document-management system which has a some standard fields (document title, creation date), but also gives users the possibility of adding their own (department name, security code, etc). The question whether having such fields is a good idea is touched upon by Joel; in this post, I’d like to think about the available options for storing these fields in a relational database.

So far, I’ve come across four different patterns to store custom fields in a RDBMS:

1. the meta-database
2. the mutating table
3. the fixed table
4. the LOB

All of these have their advantages and disadvantages. I will discuss them seperately, using the document-management system as an example.

the Meta Database

This approach basically defines a database inside of the database. In its simplest form, it consists of two tables: a Field Definition table, and a Field Value table. The value table has a foreign key relationship with the definition table, and with the table containing the entities (i.e. the documents table). The following figure illustrates this:

When adding a custom field, you just add a row to the definitions table. When adding a document, you look up the defined custom fields, and add value rows for all definitions. When retrieving documents, you also retrieve all custom fields for that document.

Advantages

• Fully normalized

Disadvantages

• Requires complex queries to get to the fields values
• Slow, because of the JOINs
• The fields table become huge, because we add a row for each custom field

the Mutating Table

When using this approach, we store all custom fields in one Fields table. This table has a one-on-one relationship with the documents table, and it alters as we add or remove custom fields. Here’s a picture:

The custom fields table initially starts out with just an id column, but as custom fields are defined, columns are added to this table by using ALTER TABLE statements. When custom fields are removed, we drop the column again.

Advantages

• Simple to query: the data is just there to read
• Works nicely with reporting tools

Disadvantages

• Requires runtime permissions to alter the database
• Does not work with ORMs which require a mapping to be defined upfront

the Fixed Table

This approach is very similar to the previous one: we still have one table for all fields. The major difference is that we don’t alter this table, but initialize it with a fixed number of nullable columns: 20 or 30 will do. The columns all have unimaginative names such as CustomField1, CustomField2, etc.; in another table, we map these column names to the field names as fields are defined. It looks something like this:

Advantages

• Simple to query
• Works with ORM & reporting tools

Disadvantages

• Fixed amount of fields

the LOB

Finally, there is the option of storing custom fields in some sort of Large Object. You can serialize some Hashtable object containing the fields and store that in a BLOB, or create an XML tree and store that in a CLOB. The LOB containing this data will just end up right next to the other document fields, in the document table:

Advantages

• Fully extendable

Disadvantages

• Hard or even impossible to query

Conclusions

I’ve used all of these four approaches, and they all have their pro’s and con’s. Overall, my favorite is the Fixed Table pattern, where we simply reserve some table space for 20 or so fields. Sure, this might be less flexible, since the amount of fields is fixed, but I don’t think that’s a big problem. Quoting Joel:

Here’s a common programmer thought pattern: there are only three numbers: 0, 1, and n. If n is allowed, all n’s are equally likely. […] Thus, for example, programmers tend to think that if your program allows you to open multiple documents, it must allow you to open infinitely many documents […]. A programmer would tend to look with disdain on a program which limited you to 20 open documents. What’s 20? Why 20? It’s not even a power of 2!

In the past, when I offered the choice to clients, explaining that they could choose between a solution which was easy to implement, but had an upper limit of 20 fields; or a solution which was more complex, and doesn’t work with their existing reporting tools, they all chose the simple solution.

And besides, 20 fields ought to be enough for anybody.

Back in the day, when I was a C++ Developer on the Microsoft platform, it always struck me as weird that half of my code base consisted of String conversions¹. The reason for all this conversion fun was that there were at least half a dozen string types in Microsoft C++: there is the plain char*, the Unicode wchar*, LPSTR and friends (LPCSTR, LPWSTR, LPCWSTR, LPTSTR, LPTSTR, LPCTSTR, and LPCTSTR), the MFC CString, the STL basic_string, the Visual Basic BSTR, and its C++ wrappers _bstr_t and CComBSTR. And I’m probably forgetting quite a few. Each library used a different string, so conversion was all in a day’s work.

Imagine my joy when I started programming in Java, some 10+ years ago. “Woohoo! Just one string type! And garbage collection too! No more conversions, I am going to be so productive! Finally, I will have time for a Real Life!”

Fast-forward to the present, and I still don’t have that Real Life I wanted. I wonder what went wrong. As it turns out, conversion is still all in a day’s work. The only thing that has changed is the types we’re converting.

If you look at the classic architecture of an Enterprise Application, it will look something like this:

I am counting eight conversions in this picture. Granted, you can skip the Data Transfer Objects by using Domain Object directly, and the SQL can be generated by using an ORM tool. However, using an ORM tool does not mean that there is no conversion at all; it is just a higher-level abstraction. You still have to define the metadata (Hibernate mappings, JPA annotations, etc.) to do this conversion, of course.

In my experience, a typical Enterprise application consists of very little “Business Logic”², but mostly Conversion Logic, so I suppose that conversions are still all in a day’s work.

Guess I’ll never have that Real Life.

---

1 - The other half consisted of reference counting and releasing. Makes you wonder what’s left.
2 - An most of the Business Logic is hardly logical.

Acronym	Definition
C|N>K	Coffee Through Nose to Keyboard

For an extra laugh, note the alt tag on the image.

(Via xkcd.com.)

I just came back from JavaZone. Good fun, just like last year. I did two talks: one together with Jürgen Höller about managing large code bases, and one by myself about WS-DuckTyping.

I’ve you missed me in Oslo, not to worry! I have a few more talks coming up:

• On the 18th of September 2007 (yes, that’s tomorrow), I will be talking with Alef about Spring 2.5 and OSGi at the Cap Gemini Java Night.
• On the 9th of October 2007, I will be talking about Spring Web Services at the BeJUG SOA Event.
• From the 15th till the 18th of October 2007 I will be speaking at Expo-C. This is a small conference in the south of Sweden, which apparently brings good quality audience and speakers (which makes me wonder why they asked me ). I will be talking about WS-DuckTyping, and Effective Framework Design, which is going to be a “lessons learned when we built Spring”-sorta session.
• From the 13th till the 15th of November 2007, I will be speaking at Øredev about the architecture of large open source frameworks. Also, I will be giving an AOP course while I’m there.
• Last but not least, from December 12th till 15th 2007, I will be talking at The Spring Experience. I had a great time there last year, and if this year’s version is only half as good, it will be worth it. I will do three sessions on Spring Web Services (WS-DuckTyping, REST, and a basic introduction).

All in all quite a busy schedule. Hope to see you at one of these sessions!