The Fiftieth post

This is my 50th post. In this post I would want to write about couple of projects done over the last 9 months that I am very proud of. My contribution to these has been into initial ideation, brainstorming and design discussions. When the team was in any sort of dilemma they involved me and together we found a way out.

The first project we did was the e-Apps framework. It was designed by my colleague BR. It is one of the cleanest designs that I have come across so far. All credit to BR. The framework allows any delegation based J2EE application to be offline capable. In addition to that synchronization is provided out of the box. Most of the features are configurable and to convert an existing application, the plumbing code to be written is minimal. It has been tested on Enterprise scale insurance application and the concept works marvelously. We still haven’t got a Client in production mode, but the product’s core is robust and all set. It would be a fantastic experience to productize this baby of BR and to go through the entire experience of product management. It could be a life changer, who knows !

Versioning and Transports : Not many mid-scale product companies think about meta-data driven architectures. Among those who do, very few go to the extent of versioning their meta-data. Transports across many version repositories is even rarer. I am extremely happy to be part of an organization that thinks progressively.

We developed the complete infrastructure for versioning in a span of 3 months. We used Apache JackRabbit opensource and built on top of it. Quite a few of the functionalities we needed were not available in JackRabbit and we had to build them ourselves. Once again, I teamed up with BR who single handedly churned out all the code of the version repository. Transports are still under development but it too looks quite promising.

WYSIWG Editors (Eclipse plugin) : Such editors are quite commonplace. We use them and more or less take them for granted. But what about developing an eclipse based Form Editor that is generic, which can be bound to Objects, which can provide sophisticated layouting ? It is hell of a lot of sweat. We are yet to crack it but I am quite confident that we would. The learning for the team has been enormous and the complexity of the Component is well understood by one and all.

Performance Improvement : I was personally involved in a performance improvement exercise of a Enterprise scale Insurance application. The task was to scale it to 2500 connected users with an average response time of 2 seconds. It was a big ask but we undertook the task with gay abandon. This project reiterated the fact of getting one’s basics right. To sum it up, the lessons learnt were
1. Get your data access basics right.
2. Performance improves due to those queries that you never write.
3. Think scalability upfront. With 2500 connected users, think that every query would get fired that many times.
This will force you to look for any reasonable optimizations.
4. Respect memory even though it is Java world. Release references when not needed. Dont take memory management for granted in Java.
5. Log messages could pull you down if you aren’t careful. Use ifDebugEnabled. Be wary of String concatenations.
6. Get the right tools to monitor performance.
7. Know the tools that monitor you database activity and performance. Know the database vendor specific tools. This is an absolute must.
8. Look closely at your application logic. There are several things you could improve there.

Apart from these, there have been quite a few other projects as well. May be I’ll write about them in another post.
Overall, the last 9 months have been quite Good. Quite some learning, lucky to have BR in the team, passionate bunch of folks in the team. I do have a long list of cribs, but that isn’t for public consumption. Looking forward to test our creations in the real big bad enterprise world.

Start of Year 2

It is One year since I started this blog. It is time to look back and check what went well, what didn’t.

The purpose of this blog was to write about technical stuff as I encounter them in my day to day work. I wanted to write only about those things I have myself experienced first hand.

Including this post, there are 49 posts this year. 3 posts are in draft and may never make it. On an average, it is one post per week, and I think it is not bad.

What I am concerned about is my writing skills. I just don’t have the patience to write. My writing skills, in terms of style, fail me badly very often.

Perhaps, this year, I should reduce the frequency of posts and instead focus more on the quality and depth of the posts. The purpose would still remain the same. And there shall not exception to this.

Some stats on the readership

Total views : 2488
Total comments : 43
Subscribers via google reader : 6

Top 3 posts

Is Java the new COBOL?
Javascript Spelling corrector
Three jars problem

I should thank A and V who motivated me to start blogging. Looks like now, I need to motivate them to keep blogging

The Meta syndrome

In the recent past, the blog post that hit me hard the most was this.

I was a victim of the meta syndrome.
It was nice to know that there is a whole bunch of us who get caught in the meta syndrome at some point.

Firstly, it is healthy to think about what we are doing. As Jeff mentions, the “sharpening the saw” part is indeed very healthy. It is when we dont know where to stop, we end up contracting meta syndrome.

Discussions and brainstorming when taken to extreme levels lands in the meta syndrome.
We keep talking about how to build the flawless new system that no matter what solution we arrive at,
we subconsciously tend to find some shortcomings and thereby prolong the meta talks/discussions.
This is when we get addicted to “meta”.

Needless to say, Meta syndrome makes you unproductive. It could wreck your career if you are not careful enough.

So … What is the way out of this meta syndrome ?

1. After reasonable time spent on meta thinking/talking, get down to building stuff.
2. Let the process of building be Iterative.
3. The subsequent meta sessions should be fuelled by the learnings in the previous development cycle.
This sets certain scope and boundaries and doest let the meta talks go unbounded.
4. “Ideal state” is fluid. Cant arrest it. Can you ever freeze the requirements for any product ?
5. Be pragmatic. Ideal solutions are almost impossible and not many need them.
6. Dont fall into the “perfection” trap. Build something that is good enough under the given circumstances.
7. Think -> Do -> Think -> Do … and so on.
It should NEVER be Think -> Think -> Think some more …

Remember that
Thinkers are rare, Doers are rarer, Thinkers and Doers are the rarest.

Be a Thinker and Doer. Stay away from the meta syndrome !

Accessing Samba shares in Terminal

I use Ubuntu for my daily work. Rest of my team members use Windows. So I end up using samba for all filesharing.

Many times, I need to do some stuff on the terminal window … like grep or find-replace using sed or some log file analysis. I have a tough time mounting these shares in fstab … its a pain.

I no longer undergo any such pain. I recently discovered that when I access the samba shares via Nautilus, it does all
the mounting itself. All I need to know is the mount point in the local filesystem.

It is ~/.gvfs

I am overjoyed !

Right tools at the Wrong Hands

In this post, I wish to share my cribs on how ORM (Object Relational Mapping) tools get abused by naive developers.
The impact is such that one tends to think something is perhaps not right with ORMs.

Object Relational Mapping tools (ORM) map the Object world on to the RDBMS world.
This is a huge productivity benefit. OO developers are comfortable with the notion of Objects and dealing with objects throughout makes it all the more easier. Popular ORM implementations (in the Java world) are Hibernate, iBatis and TopLink.

The single most drawback I see with ORM in the hands of unseasoned developers is that “ORM cultivates in them a disregard for databases and DB concepts“. This isn’t an issue with ORM per se. It is the unseasoned developer who is to be blamed and all. Fine … But then, it results in badly written applications whose performance suck.

ORM’s treat a row in the database as an object (more or less). It is indeed a huge productivity gain for developers to work with objects without bothering about how it came all the way from database. Whereas there is indeed a productivity gain, the down side is the developer’s insensitivity towards the database concepts.

For example, The specification document reads, “…Get all the orders for the given customer. For each order, set the is_delivered flag to TRUE”.

The unseasoned developer translates the specification faithfully in this way

Collection myorders = OrderDAO.findbyID(“customerID”);
for (order : myorders){
order.setIsDelivered(“TRUE”);
order.save();
}

Let us look at the above trivial code from a database perspective.

Collection myorders = OrderDAO.findbyID(“someID”);

This results in the Query like “SELECT * FROM ORDERS WHERE CUSTID = ?”

Inside the loop, the followign statments
order.setIsDelivered(“TRUE”);
order.save();
translate to
UPDATE ORDER SET IS_DELV=1 WHERE ORDERID = ?

Please note that the UPDATE statement is executed for EACH order.

Instead, what we actually need is

“UPDATE ORDER SET IS_DELV=1 WHERE CUSTID = ?”

If there are N orders for a Customer, then N+1 Queries are executed in place of just One Query.

The fundamental difference between a language like Java and SQL is that while Java is a programming language
SQL is a DECLARATIVE language. SQL tells what needs to be done and omits the how. Java programming is mostly about the how.

Another classic example is … the specification reads “If there are records meeting this criteria, then update them with this value…”

The unseasoned ORM-er would do it this way

Collection myorders = order.findbyParams(x,y,z,p,q,r);
if (myorders.size > 0){
order.setSomeParam(somevalue);
order.save();
}

I cannot blame ORM for such abuse of SQL. Even developers using plain jdbc (and no ORM)
would translate the above spec as

select count(*) from ORDER WHERE
if (rowcount > 0) {
//perform the update
}

The SQL which is actually needed is

UPDATE ORDER SET SOME_PARAM = :somevalue WHERE

Obviously, if the criteria matches no records, the update would not happen anyway.

ORMs make it easier for naive developers to overlook the basics of databases.
ORM works best at the hands of those who have solid DB fundamentals and use ORM the right way.

The issues get magnified when say, 5000 users are connected, and thus there is a potential for each query to be fired 5000 times ! This means, a single query that is found to be unnecessary, amounts to firing 5000 queries. What a colossal waste of resources. Add to this any missing index etc and the problem gets compounded.

Also ORMs prevent the naive developer from appreciating joins – both complex and simple.
ORMs make it easier for developers to follow the specifications to the dot without any database considerations.

For example, A specifcation reads

“Get the f_id from Table1. Using this query on Table2 to obtain px_y. Use px_y from Table3 to get the m_imp_data.”

The ORM savvy developer faithfully follows the specifications.

String str_f_id = Table1DAO.find(“someid”).getfid();
String str_px_y = Table2DAO.findByBlah(str_f_id).getpxy();
String str_imp_data = Table3.DAO.findByBlah(str_px_y).getimpdata();

Well, as I said before, a naive developer using plain jdbc could as well do the same

String Qry1 = “select f_id from Table1 where blahID = ?”
String Qry2 = “select px_y from Table2 where x_f_id = ?”
String Qry3 = “select m_imp_data from Table3 where x_px_y = ?”

But the point is that ORMs make it a lot easier as the developer is out of touch with SQL (almost).
What is needed here is a simple inner join. You can read more about Joins here.

In a future post, I shall provide examples on the right usage of ORM and how ORM and plain jdbc should co-exist for better performance of the application.