Sunday, 30 August 2009

Care about the Data

Developing software is a hard and time-consuming task. We can certainly do much more in much less time than when assembly was the only option, but expectations and needs about software have grown proportionally, and perhaps even faster, than software development processes and tools themselves.
For this reason, improvements in productivity and cost-effectiveness of software development never seem to be enough and it's even more the case because smaller and smaller companies require, and expect to be able to afford with smaller and smaller budgets, the benefits that software and correct information management processes can bring to them in terms of effectiveness and competitive edge.
As a consequence, software development companies try hard to find ways to develop software in less time and with less (and less skilled) people. There are several ways to try to reach this goal, here are a few:
  • Stimulating staff growth, experience, speed and overall professional roundness
  • Studying, evaluating and adopting increasingly sophisticated tools (more and more expressive languages, frameworks, libraries; more mature development management tools and processes; ...), i.e. technology and expertise advance;
  • Relying on an asset of already built libraries or application templates for common cases;
  • Automation and software generation;
  • Of course, last but not least (and most frequent sometimes), trying to sell low quality, fast-hacked solutions as if they were much better than they are, relying on the substantial ignorance of the customer.
Let alone the fact that the latter is clearly a blind and short-termed strategy, there is some risk of quality loss in the other ones as well. This risk is clearly stimulated by software markets that are shaped out of the needs of companies short on budget and with scarce IT awareness (typically frequent, in my experience, with small oens just starting to let in IT) that will easily end up just choosing the cheapest solution.
Quality loss is very dangerous in short, medium and long terms and in every area but I'm perhaps most sensible about the data storage area. Why? Simply because:
  • Understandable and accessible data is easily the most valuable asset for many companies;
  • Application data survives the application itself and even application providers; in fact, any information can be considered legacy as soon as it is stored for use in a production environment;
  • Changing and sometimes even evolving persisted data structures is hard (more so if they're not yours);
  • Most often than not, application data must be easily accessed directly through storage tools usage by the customer itself for maintenance, fixes, queries, OLAP and BI. As there's always little time for docs and for instruction, the more the data structure is clean, clear and open to 3rd-party understanding, the better for anyone;
  • Even if all of the above is satisfied, you'll find yourself fiddling with customer data directly in the storage more often than you might think (and this means daily in most cases) because of bugs and planned / unplanned support or upgrades. You'd soon regret a poorly designed storage for this reason alone.
Another lesson to be learnt is: develop schema and data migration strategies as soon as possible in the development process. You'll soon be happy of having thought about it when you'll find you need a different storage solution for whatever reason (new performance requirements, new data access patterns, ...).

So, for the health of your business (and of yourself) here's my warmest and solicit suggestion: Take Big Care of the Data.


  1. Interesting post.

    "develop schema and data migration strategies"
    That's the point I like the most.
    Plan for change.

  2. The limp phase that the IT sector and the software industry is going through in the present times can be gauged from the fact that the pay of the employees in these two fields all over the world no longer remain lucrative. In many countries, software companies are also chucking out employees, especially those employees who have been “sitting on the bench”. This was an unheard of concept a few years back. There are chances that the software companies take more such radical steps to fight the damp phase.