Architecture and Data Blog

Automated testing of custom sql

Tue, 23 Sep 2025 00:00:00 +0000

Lots of time hand coded SQL is written in the applications for performance sake, getting data using complex SQL out of the tables. As the database structure changes it becomes really hard to find out all the SQL that needs to be changed. In other cases SQL is generated based on certain conditions in the code and its hard to find if the generated SQL is valid after database changes are done.

Generate boiler plate code

Sat, 17 May 2025 00:00:00 +0000

When accessing the database using stored procedures for basic Create, Read, Update and Delete (CRUD) functions, or when you want to write triggers that capture the before and after values in tables, or when you want to create Plain Old Java Objects (POJO’s) that match the database objects, writing these by hand takes a lot of effort and also since the requirements are changing in an agile projects at a frequent rate, there will be design changes to meet the requirement changes, so the triggers, CRUD stored procedures or Data Access Objects (DAO) are going to be out of data, instead of hand coding, its better to generate the code, using the metadata of the database

Data specialists should pair with developers

Thu, 01 Aug 2024 00:00:00 +0000

Traditionally the data-team is used to sitting in their own area and working for many project teams by handling requests either via a ticketing system or vi email. The hand-over of work or throwing of work over the wall creates knowledge silos and inefficiencies.

Publish data models in CI Pipeline

Sat, 16 Sep 2023 00:00:00 +0000

Many a times ER models are created by the data team and are not shared outside of the data team generally for the lack of tools licenses, since its not feasible for the entire team to purchase licenses for the ER modelling tools such as Erwin Data Modeller or Er Studio

Trusting search results from Google & Others

Fri, 04 Dec 2020 00:00:00 +0000

Search engines have been great to find information on the internet with their ease of use and the ability to find cross linked content with specific keywords. This ease of use is being exploited by scammers and imposters.

SSL Connection to AWS Aurora

Mon, 15 Jul 2019 00:00:00 +0000

On a recent project we had to connect to AWS Aurora postgres 10.6 version of the database in SSL mode using JDBC and Java 11 JRE. When the Aurora cluster is setup, we can force all connections to use SSL by using the options group settings (forceSSL=true), establishing secure connection from the application to the database is not as easy as it looks.

Static Analysis of PL/SQL code

Tue, 01 Aug 2017 00:00:00 +0000

In all new development and sometimes during legacy codebase modernization, developers tend to add code quality checks and static analysis of codebase such as style checks, bug finders, cyclomatic complexity checking etc. into the CI/CD pipeline. When we inherit a codebase that has much PL/SQL and there is a desire to put the PL/SQL code base through the same types of code analysis, what options does a developer/dba have?

ID generation with ORM's Table, Sequence or Identity strategy

Mon, 12 Jun 2017 00:00:00 +0000

In this blog post, I will discuss ID generation techniques using the Object Relation Mapping frameworks such as Hibernate, Toplink, ActiveRecord, Entity Framework. When using hibernate or any other ORM mapping framework. There is a need to generate primary key values for the “id” columns. These values can be generated by using IDENTITY, SEQUENCE or TABLE strategies. Generating custom values for primary keys or other values is a topic for another blog post.

Using liquibase to load data and ignore some columns

Mon, 15 May 2017 00:00:00 +0000

Loading data into tables is needed many times on projects to load test, Liquibase provides a method to load data into tables with lots of customization. In the example shown below, I’m loading zip code data with the following column layout

"Zipcode","ZipCodeType","City","State","LocationType","Lat","Long","Location","Decommisioned","TaxReturnsFiled","EstimatedPopulation","TotalWages"

Two factor authentication to authorize credit"

Wed, 13 Jul 2016 00:00:00 +0000

Automatically adding columns to Rails migrations

Mon, 25 Apr 2016 00:00:00 +0000

Many projects need addition of identical columns to all the tables created by the project. Audit columns are an example of such a requirement. The requirement is to add columns such as created_by, created_date, modified_by and modified_date to all the tables, these columns store, who created the row, when the row was created, who modified the row last and when was it modified. created_by and created_date are required to be present when the row is inserted and thus are required to be not nullable. Adding these columns to each and every table is a lot of work for developers.

Synonyms as abstraction layer

Fri, 15 Apr 2016 00:00:00 +0000

In many development shops, developers are not allowed to access the database schema directly, and are not allowed to create tables, indexes, views etc, instead are given access via a different schema that allows SELECT, UPDATE and DELETE access on data. The general reason is to avoid developers creating database objects without

Behavior Driven Database Development

Sun, 16 Aug 2015 00:00:00 +0000

When Behavior Driven Development BDD was introduced, some of the key principles were

Requirements are behavior,
Provides “ubiquitous language” for analysis,
Acceptance criteria should be executable.
Design constraints should be made into executable tests.

Using rake and activerecord to generate boilerplate DB Code

Mon, 20 Jul 2015 00:00:00 +0000

IN many projects, there are tables which need default audit columns such as Created_By, Created_Date, Modified_By, Modified_date and other columns that need to be updated every time some actions are done against the tables and/or columns. This type of functionality can be implemented using triggers.

Database naming conventions in different environments

Wed, 10 Jun 2015 00:00:00 +0000

In every enterprise and every project we end up having multiple environments, especially the database side of the enterprise tends to stick around for a longer period of time and has much more dependencies or application integration as opposed to application urls etc. Given this, how to name the servers, databases and schemas becomes a very important decision, do these names provide for an easy way to use the application and not make it harder or the developers to access the database.

8 Techniques for testing migration of data from legacy systems

Fri, 23 Jan 2015 00:00:00 +0000

Many of the projects we end up working on are replacing existing systems with existing data either wholly or in part. In all of the above projects we end up writing data migration or data conversion code to move the data from legacy systems to the new systems. Many stake holders of the project such as business users, project managers, business analysts really care about the data conversion scripts and the quality of the conversion. Since this conversion is business entity related and matters a lot as future business/functionality depends on the data being logically equivalent to the legacy system.

Migrations in NoSQL databases

Tue, 14 Oct 2014 00:00:00 +0000

In relational database usage the pattern of migrations is well understood and has gained widespread acceptance. Frameworks such as DBDeploy, DBMaintain, MyBatis migrations, Flyway, Liquibase, Active Record Migrations and many others. These tools allow to migrate the database and maintain the version history of the database in the database.

With the rise of NoSQL Databases and their adoption in development teams we are faced with the problem of migrations in NoSQL databases. What are the patterns of data migrations that work in NoSQL databases? as NoSQL databases are schema free and the database does not enforce any schema validation, the schema of the data is in the application and thus allows for different techniques of data migration.

10 node mongodb ReplicaSet on a Single Machine"

Mon, 09 Jun 2014 00:00:00 +0000

While doing evalauation of NoSQL databases, we had a 10 node riak cluster and wanted check how a similar setup would work with mongodb. So started to setup a 10 node mongodb cluster. Since this was for initial spikes, we decided to set this up on a single machine as with the other test setup using Riak.

Before I explain how we setup 10 node mongodb ReplicaSet, let me talk about replica sets. MongoDB implements replication, providing high availability using replica sets. In a replica set, there are two or more nodes participating in an asynchronous master-slave replication. The replica-set nodes elect the master node, or primary node, among themselves and when the primary node goes down, the rest of the node elect the new primary node.

Moved my blog to octopress

Fri, 31 Jan 2014 00:00:00 +0000

Its been about a month since my blog moved to octopress, wanted to write about my experience. I had been running my blog for a some time now using Movable Type upgrading as and when new versions where released. Over time I realized that upgrading was fraught with errors as lot of steps had to be done manually. Customizing the layout was risky as there was no way to preview your changes and commit only when I was comfortable. With the release of Movable Type 6 there is no longer a free version to download.

Usage of mixed case database object names is dangerous

Tue, 06 Aug 2013 00:00:00 +0000

Some versions back, Oracle would not allow to create database object names with mixed cases, even if we tried to create them, we could not. In newer versions of Oracle we can create tables, columns, indexes etc using mixed case or lower case, when the names are put inside double quotes. For example

10 node Riak cluster on a single machine

Thu, 25 Apr 2013 00:00:00 +0000

When trying to evaluate NoSQL databases, its usually better to try them out. While trying them out, its better to use them with multiple node configurations instead of running single node. Such as clusters in Riak or Replica-set in mongodb maybe even a sharded setup. On our project we evaluated a 10 node Riak cluster so that we could experiment with N, R and W values and decide which values where optimal for us. In Riak here is what N, R and W mean.

Transactions using Groovy.SQL with Spring annotations and connection pools

Mon, 14 Jan 2013 00:00:00 +0000

When using Groovy with Spring framework, interacting with the database can be done using the Groovy.SQL class which provides a easy to use interface. When using Groovy.SQL, if we have a need to do transactions, we have the .withTransaction method that accepts a closure, to which we can pass in code to execute within the transaction. In our project since we were using spring already, using annotations to define transactions would be a great. Standard @Transactional annotations with Groovy.SQL will not work, since every place where the Groovy.SQL is used a new connection is acquired from the connection pool causing the database work to span multiple connections, which can result in dead-locks on the database. What we really want is that the database connection be the same across all invocations of Groovy.SQL with-in the same transaction started by the annotated method.

Backup in mongodb replica-set configurations

Thu, 27 Dec 2012 00:00:00 +0000

There are multiple ways to take backups of mongodb is different configuraitions, one of the configuration that I have been involved recently is replica-sets. When mongodb is running in replica-set configuration, there is a single primary node and multiple secondary nodes. To take backup of the replica-set we can either do a mongodump of one of the nodes or shutdown one of the secondary nodes and take file copies, since in a replica-set all nodes have the same data (except arbiter). Lets see how we could deal with mongodump method of taking backup.

Back to blogging

Sun, 22 Apr 2012 00:00:00 +0000

There has been a long pause in my blogging activity. I was trying to finish of my latest writing engagement in regards to NoSQL. Working with Martin Fowler on NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence was really fun. This book will provide a concise text and easy way to understand for everyone the rise of the NoSQL movement and help with what kinds of trade-offs need to be made while working with NoSQL.

MSSQL JDBC Driver behavior"

Sun, 22 Apr 2012 00:00:00 +0000

My latest project involves talking to MS-SQL Server using the JDBC driver and Java. While doing this we setup the database connection and had a simple SQL to get the first_name and last_name for a unique user_id from the application_user table in the database.

SELECT first_name,last_name
FROM application_user
WHERE user_id = ?

Given the above SQL, we did not think too much about performance as the user_id was indexed. The java code as below was used to run the SQL.

With so much pain, why are stored procedures used so much

Wed, 19 Jan 2011 00:00:00 +0000

I keep encountering situations where all the business logic for the applications is in stored procedures and the application layer is just calling the stored procedures to get the work done and return the data. There are many problems with this approach some of them are.

Writing stored procedure code is fraught with danger as there are no modern IDE’s that support refactoring, provide code smells like “variable not used”, “variable out of scope”.

Replica sets in MongoDB

Sun, 31 Oct 2010 00:00:00 +0000

Replica sets is a feature of MongoDB for Automatic Failover, in this setup there is a primary server and the rest are secondary servers. If the primary server goes down, the rest of the secondary servers choose a new primary via an election process, each server can also be assigned number of votes, so that you can decide the next primary based on data-center location, machine properties etc, you can also start mongo database processes that act only as election tie-breakers these are known as arbiters, these arbiters will never have data, but just act as agents that break the tie.

Schema less databases and its ramifications.

Tue, 12 Oct 2010 00:00:00 +0000

In the No-SQL land schema-less is a power full feature that is advertised a lot, schema-less basically means you don’t have to worry about column names and table names in a traditional sense, if you want to change the column name you just start saving the data using the new column name Lets say you have a document database like mongoDB and you have JSON document as shown below.

{
"_id":"4bc9157e201f254d204226bf",
"FIRST_NAME":"JOHN",
"MIDDLE_NAME":"D",
"LAST_NAME":"DOE",
"CREATED":"2010-10-12"
}

You have some corresponding code to read the documents from the database and lets say you lots of data in the database in the order of millions of documents. If you want to change the name of some attributes or columns at this point and the new JSON would look like

Effective use of data for better customer experience.

Tue, 31 Aug 2010 00:00:00 +0000

For more than seven years I have been getting offers for credit cards from Airlines and Banks. One particular bank has been sending me these solicitations for more than seven years. That is 12 mailings per year, more than 72 mailings so far, remember these are physical paper mailings not the electronic kind. I don’t like the junk, it hurts the environment and worst of all I think its not good use of the data they have. How hard is it to design a system around the data they have.

Schema design in a document database

Wed, 28 Apr 2010 00:00:00 +0000

We are using MongoDB on our project, since mongo is document store, schema design is somewhat different, when you are using traditional RDBMS data stores, one thinks about tables and rows, while using a document database you have to think about the schema in a some what different way. Lets say, we want to save a customer object, when using a RDBMS we would come up with Customer, Address, Phone, Email. They are related to each other as shown below.

My experience with MongoDB

Sun, 18 Apr 2010 00:00:00 +0000

The current project I’m on is using MongoDB. MongoDB is a document based database, it stores JSON objects as BSON (Binary JSON objects). MongoDB provides a middle ground between the traditional RDBMS and the NOSql databases out there, it provides for indexes, dynamic queries, replication, map reduce and auto sharding, its open source and can be downloaded Mongodb, starting up mongodb is pretty easy.

./mongod --dbpath=/user/data/db

is all you need, where /user/data/db is the path where you want mongo to create its data files. There are many other options that you can use to customize the mongo instance.

Workshop at Enterprise Data World 2010

Fri, 29 Jan 2010 00:00:00 +0000

Doing a workshop on Agile Database Development at Enterprise Data World 2010 at SF. See you there.

Testing in data conversion projects

Wed, 18 Nov 2009 00:00:00 +0000

When working on projects involving Conversion of data or Migration/Moving of data from a legacy database. The testing effort is enormous and testing takes a lot of time, some test automation can help this effort.

Since data is moved/changed from a source database to destination database, we can write sql which should provide results for the types of tests you want to perform, for example: write a sql to give us number of customers, write a sql to give us account balance for a specific account.

Ruby OCI 2.0 Array binding

Thu, 08 Oct 2009 00:00:00 +0000

We have been doing some data moving lately using Ruby and Ruby-OCI. We started with Ruby OCI 1.0 and did use prepared statements with bind variables (since we are using oracle database and pulling data from an oracle database and pushing data to an oracle database). Later we found this really cool feature in Ruby-OCI8 2.0 where you can bind a whole array and just make one database trip for many database operations.

Create an Index for all FK Columns in the database

Thu, 03 Sep 2009 00:00:00 +0000

Most of the time I have seen database foreign key constraints on tables without indexes on those columns. Lets say the application is trying to delete a row from the CUSTOMER table

DELETE FROM CUSTOMER WHERE CUSTOMERID = 1000;

When the database goes about deleting the customerId of 1000, if there are foreign key constraints defined on customerId, then the database is going to try to find if the customerId of 1000 is used in any of those tables. Lets say ORDER table has the customerId column, the database is going to issue

Materialized views and database links in oracle.

Mon, 10 Aug 2009 00:00:00 +0000

Recently one of my colleague Jeff Norris had a weird error. He was trying to build a materialized view over some tables in his local database and some tables in his remote database using database links the sql to create the view ran fine and provided the results as expected, but when put inside a materialized view statement complained with ORA-00942 errors.

Lets say the two databases in question are local and remote, so the sql to create the materialized view to load immediately and refresh everyday is

Perfectly good data.. wasted

Wed, 05 Aug 2009 00:00:00 +0000

Okay this is kind of a rant, maybe I’m too picky or just that I hate to see perfectly good data not being used. This is how it goes..

I go regularly to this store to get Horizon organic milk for my family, about 60% of the time I see milk I need NOT in stock, okay I can live with that, may be lots of folks are buying organic milk, but not when it happens frequently, especially when the store knows how much milk was ordered (or supplied from the warehouse) and how much milk was sold, the store should be able to figure out that organic milk gets sold out pretty fast, putting my Business Intelligence (BI) hat on, I think the store should be able to predict when they are going to run out of organic milk ( for that matter any product), its especially frustrating when they have all the data they need to get it done.

Explicitly rollback when you encounter a deadlock.

Tue, 26 May 2009 00:00:00 +0000

Dead lock is caused in the database when you have resources (connections) waiting for other connections to release locks on the rows that are needed by the session, resulting in all session being blocked. Oracle automatically detects deadlocks are resolves the deadlock by rolling back the statement in the transaction that detected the deadlock. Thing to remember is that last statement is rolled back and not the whole transaction, which means that if you had other modifications, those rows are still locked and the application should make sure that it does a explicit rollback on the connection.

Oracle for the Mac

Thu, 14 May 2009 00:00:00 +0000

Ever since I moved to the Mac, I had to run some other OS inside a VM so that I could run Oracle and use it, since Oracle was not available for the the Mac. Now that is no longer the case. Oracle 10gR2 (10.2.0.4) is now available for Mac

This is especially nice since the Oracle for Mac was the most voted requirement on mix.oracle.com

In Oracle 11g password is case sensitive

Wed, 06 May 2009 00:00:00 +0000

In Oracle 10g and before we all know that passwords are not case sensitive, so PASSWORD, Password, password would let you in and everything would be okay.

If you upgrade to Oracle 11g (I know lot of you are waiting for 11gR2), you will find that passwords are case sensitive. Here is an example of case sensitive passwords.

c:\Software>sqlplus bddd/bddd@dosa
SQL*Plus: Release 11.1.0.6.0 - Production on Wed May 6 15:17:43 2009
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the Partitioning and OLAP options
BDDD@dosa >

Lets try to connect with a upper case password

Oracle Metadata can be mis-leading

Tue, 31 Mar 2009 00:00:00 +0000

Oracle has metadata about all its objects in various tables/views. One such view is the USER_OBJECTS or ALL_OBJECTS, this view has a column named as STATUS which shows you if the given object is VALID or INVALID. The status applies to DB Code (Stored Procedures, Functions, Triggers etc).

To find all the INVALID objects in the schema, issue

SELECT * FROM USER_OBJECTS WHERE STATUS='INVALID';

One problem with the way oracle maintains this metadata is, changing the underlying table on which the DB Code depends, oracle marks the objects are INVALID even though the underlying table may have changed in such a way, that it does not affect the DB Code at all (like adding a new column, or making a colum nullable). Here is some code which shows you what I mean. Run it through SQLPlus.

Presentation on Database Refactoring

Thu, 26 Mar 2009 00:00:00 +0000

My Presentation on Database Refactoring at QCon was recorded and is live now on infoQ here

Storing just the time in Oracle

Thu, 12 Feb 2009 00:00:00 +0000

We came across a need to save just the Time in the database, the requirement is to store time of the day, like say the user likes to have Breakfast at 8.15AM and Lunch at 12.32PM etc. Off course oracle does not have a time only data type. So we ended up using DATE as the data type and just setting the time. for example:

CREATE TABLE FOO (PREFERRED_TIME DATE NULL);
INSERT INTO FOO (TO_DATE('11:34','HH24:MI'));

oracle automatically sets the date to the first day of the current month. so when you do a select from the FOO table the data would be

Considerations when using stored procedures for doing CRUD

Tue, 03 Feb 2009 00:00:00 +0000

Some environments like to have access to the database tables routed via stored procedures. Instead of using Create/Read/Update/Delete aka CRUD with DML, stored procedures are invoked with the parameters to perform the required operation. I’m not arguing about the benefits/pitfalls of this approach, if you have to do stored procedures, here are some things to look at.

Make the stored procedure handle one object/table only and not multiple objects or tables.
Do not commit open transactions inside the stored procedures.
Do not do business logic in stored procedures.
If they are straight CRUD stored procedures, see if you can generate the stored procedure code using some metadata?
Make sure creation and execution of the stored procedures is part of your Continuous Integration build and developer build.
Make sure stored procedures (or the metadata used to generate them) is under Version Control, have seen many problems when the stored procedure version does not match application code version
Develop against the production stack database.
Make sure exceptions thrown by the database are passed back to the application.

Article about Behavior Driven Database Design is out

Mon, 22 Dec 2008 00:00:00 +0000

Recently I have been thinking a lot about making specification and behavior expected about the database and the code that interfaces with the databases executable. The Behavior Driven Design has a lot of parallels in the database world.

Just finished writing a article about Behavior Driven Design applied to Databases or Behavior Driven Database Design for Methods and Tools.

Hibernate weirdness with property names

Fri, 24 Oct 2008 00:00:00 +0000

Consider this Hibernate mapping

@Column(name = "qReferenceId")
public Long getQReferenceId() {
return qReferenceId;
}

Where qReferenceId is data provided to our application via a external reference, we do not have a QReference Object or Table for FK references. When trying to query this object using DetachedQuery, this Simple expression was used.

public List<Movie> findByQReferenceId(Long id) {
     final SimpleExpression matchesId = Property.forName("qReferenceId").eq(id);
    DetachedCriteria criteria = DetachedCriteria.forClass(Movie.class);
    criteria = criteria.add(matchesId);
    List<Movie> movies = (List<Movie>) getHibernateTemplate().findByCriteria(criteria);
    return movies;
}

When running this method, I kept getting errors shown below.

Using Explicit Order By in your SQL Statements

Tue, 30 Sep 2008 00:00:00 +0000

Recently when our test databases where upgraded new version of Oracle, we started noticing that the order in which some drop down lists were being displayed was not correct. It turns out that the SELECT statement we had, did not have a ORDER BY clause and the data was being returned in the ORDER of the creation of rows (in the order of ROWID) when the database was upgraded these ROWID’s got changed and hence the ORDER of the data being shown in the drop down lists.

Moved to a Mac

Mon, 01 Sep 2008 00:00:00 +0000

Couple of weeks back I was given a choice to upgrade my work Laptop to a Mac Book Pro or a Windows Laptop. I choose Mac ( I know everyone is into macs nowadays). The transition was pretty good, with the exception of moving my oracle database from windows to mac, since there is no native installation of oracle on mac I had to use VMWare fusion to install oracle.

Create a Index for every Foreign Key constraint created

Tue, 15 Jul 2008 00:00:00 +0000

When creating a Foreign Key constraint on the database as shown below

ALTER TABLE BOOK ADD
    (CONSTRAINT FK_BOOK_AUTHOR FOREIGN KEY (AUTHORID)
     REFERENCES AUTHOR)
/

In the above example we are telling the database to check if the BOOK.AUTHORID is a valid value in the Author.AuthorID. When the Author table is being changed, the database does data verification on the BOOK table using SELECT against the BOOK table for the AUTHORID some thing like this

Version Control your work..

Fri, 06 Jun 2008 00:00:00 +0000

So we version control/source control everything on our project.. code/data/artifacts/diagrams etc. yesterday I said why not extend it to my writings to everything I have. So I started this long journey of refactoring my folder layout and making a nice folder structure to hold all the things I have written about have other artifacts in the process of writing and moved them all to subversion, now all my example code and writings are all under version control that gets backed up everyday…. feels liberating

Japanese Version released"

Wed, 04 Jun 2008 00:00:00 +0000

Japanese translation of Refactoring Databases: Evolutionary Database Designhas been released, thanks to Yasuo Honda for the information. The Japanese version can be found here

Tool support for Database Refactoring

Tue, 20 May 2008 00:00:00 +0000

I’m always on the lookout for better tool support to do database refactoring. Just noticed that liquibase has come out with a IntelliJ plugin to support database refactoring.

This is really cool and hopefully one of long list of tools that will support database refactoring in the future. so enjoy

Writing a SQL to generate a SQL

Tue, 06 May 2008 00:00:00 +0000

We had a weird requirement on our project recently..

Find all the Rows in All the tables that do not comply with the Constraints that we have in development but not in QA environments

Setup and Teardown of database during testing

Thu, 17 Apr 2008 00:00:00 +0000

When doing Performance Testing or running Unit/Functional tests on a database, there is a need to periodically get the database to a known state, so that the tests behave in a predictable way and to get rid of all the data created by the tests. Some of the ways to get a clean database are.

Using Scripts: Recreate the database using scripts, the same scripts that are used in development environment.

Podcast on Keeping Gray code Fit

Fri, 18 Jan 2008 00:00:00 +0000

Me, Andy, Jeff and Marjorie discuss how to keep a long running project fit in this Podcast, also on iTunes. We discuss the management of technology, people, processes and tools on longer and more mature applications. Specific topics such as refactoring, knowledge management, innovation, staffing, production support and others are covered.

Experience using DBDeploy on my project

Mon, 14 Jan 2008 00:00:00 +0000

We have been using DBDeploy on my project for more than 6 months now and wanted to show how things are going. First lets talk about set up, we are using dbdeploy in our Java development environment with ANT as our build scripting tool, against a Oracle 10g database.

Define the ANT task first

<taskdef name="dbdeploy" classname="net.sf.dbdeploy.AntTarget" classpath="lib/dbdeploy.jar"/>

Now we create the main dbinitialize task a ANT task to create you database schema, using the upgrade generated by the dbdeploy file shown below. The thing to note is that dbdeploy generates the upgrade file but does not run it against your database, so we have to make sure we call the generated upgrade file via a sql ANT task.

Lessons from Re-Learning

Thu, 27 Dec 2007 00:00:00 +0000

I have been working on a project that I had worked in 2005, trying to get a handle on what I had done about 3 years back. Exploring code and the database has been fun, also discovering the data layout and building new set of data for production has been extremely entertaining. What I learnt from this whole experience was this if your code(application or other wise) is not expecting data, this data should not be provided or even considered valid by the database, the database should be designed such that it does not even allow invalid combinations of the data.

Cannot make it out to XP Day London

Tue, 13 Nov 2007 00:00:00 +0000

After a lot of frustration about my schedule, I have had to come to this conclusion, that I cannot physically make it to London, XP day.

I’m going to miss it. Nick Ashley is going to take up my spot and I know he will do a great job.

Why do Evolutionary Design

Mon, 05 Nov 2007 00:00:00 +0000

Why do Evolutionary Design or Iterative Design or Incremental Design? Everyone who has not worked in an evolutionary manner asks this? My answer, if you think the system you designed is NOT GOING TO CHANGE EVER then sure you can do design once and deploy once and you are done, move on to next project. But tell me one project you have been on, that does not have any changes in requirements, changes in technology, changes in look and feel etc after it was deployed.

ThoughtWorks at Oracle Open World

Tue, 30 Oct 2007 00:00:00 +0000

Thoughtworks is going to be at Oracle Open World. I’m excited about this especially since it will give ThoughtWorks and Me to talk about software practices and how to apply these software practices to the database development world, off course I will talk about my books Refactoring Databases and Continuous Database Integration. ThoughtWorks will have a booth at 343 Moscone South and I will be there on Nov 14.

Frameworks should not constrain your table design

Wed, 03 Oct 2007 00:00:00 +0000

Recently talking to someone about a persistence framework that they are using, this framework creates a rows in the database table with just Primary Key value and then later on updates the table row with values for other columns. Because of this framework, the tables cannot have any NOT-NULL constraints defined.

For example, the framework is doing

INSERT INTO customer (customerId, name, birthdate) VALUES (1,null,null);
UPDATE customer set name = 'FOO Name', birthdate = '12/12/1978' WHERE customerId = 1;

You cannot have NON-NULL constraint defined on name or birthdate column, since the INSERT statement would blow up, forcing you to change table design

Lesson learnt changing firewall settings

Thu, 13 Sep 2007 00:00:00 +0000

On any machine running network related services, like in my case Oracle Listener and Dispatch Services. Don’t rebuild/stop and restart the firewall stuff like iptables.

Had to spend a lot of time, figuring out what was going on. So hard lesson learnt don’t mess with iptables when you are running oracle listener/dispatcher

When does evolutionary design happen?

Tue, 28 Aug 2007 00:00:00 +0000

A question I get, mostly related to the evolutionary database design and development. When the pair (team) gets a new feature (story) to work on, the team looks at the existing table/database design and sees if the current design is enough to implement the feature they are working on. If the currency database design does support the feature they are trying to implement, then they do not have to change the database at all, they will move on to implement the feature and change the application code as necessary.

Parsing mapping files for usage information

Thu, 19 Jul 2007 00:00:00 +0000

Currently working on a legacy application, thats been in production for a long time now. I wanted to find out what are the Tables and Columns being used by the application. Since we could see that some table columns where not being used. We are using a Object Relational mapping framework on the project, so we decided to write some code that would parse all the mapping files and gives us a list of table names and columns. We used this list to create rows in a table with two columns tablename and columnname. Once the table had this data, we just ran one more SELECT against the metadata of the database and our table which pretty much gave us a list of Table and Columns that we are not using

Long Running Data Migrations during Database Refactorings

Fri, 29 Jun 2007 00:00:00 +0000

When you are refactoring large databases, you will have certain tables that have millions of rows, so lets say we are doing the Move Column refactoring, moving the TaxAmount column from Charge table which has millions of rows to TaxCharge table. Create the TaxAmount column in the TaxCharge table. Then have to move the data from the TaxAmount column in the Charge table to the TaxAmount column you created in the TaxCharge table.

Enforcing your assumptions about database functionality

Fri, 08 Jun 2007 00:00:00 +0000

When you are writing xUnit tests you are in certain ways trying to make sure that the test breaks when the code that is being tested changes the assumptions you made when writing the Test and Production code.

Similarly if you are relying on the database to throw a error when you put invalid data, then you should write a test around this assumption, so that when someone changes the database to not behave the way you assumed it to behave, the test you wrote will break and it will force the team to think about the change to the database that is being undertaken. If the change is really required, then the team would fix the test else rollback the change being made.

XP2007 Tutorial

Fri, 08 Jun 2007 00:00:00 +0000

I will presenting about Database Refactoring: Evolutionary Database Design at XP 2007 here is the Tutorial schedule

ThoughtWorks Master class series in India

Tue, 08 May 2007 00:00:00 +0000

I will be presenting about Evolutionary Database Design and Database Refactoring at ThoughtWorks Master Class Series 2007 at Bangalore on May 19th and Pune on May 26. This will be the first time I will be presenting in India.

The Master Class Series is an annual seminar organized by ThoughtWorks India. It focuses on topics which are cutting edge, but the content of the seminars is drawn from real-life experiences on live projects. The presenters are all people with extensive hands-on experience and have delivered successful projects using the concepts they are talking about.

My Latest eBook is Out"

Mon, 23 Apr 2007 00:00:00 +0000

After finishing the first Refactoring Databases book. I started on a short ebook project, this book was going to tackle on a very specific technical topic mentioned in the first book. I wanted to write about all the specific scenarios and all the techniques I follow on the various projects.

The result of this effort was Recipes for Continuous Database Integration: Evolutionary Database Development, thanks to Martin Fowler for the title. This ebook laid out in a recipes kind of format provides various ways you can integrate you database into you development cycle and make evolutionary database development a fun thing to work with

Moscow

Wed, 11 Apr 2007 00:00:00 +0000

Last week I was at SD Best Practices in Moscow, doing a presentation on “Refactoring Databases: Evolutionary Database Design”. Moscow seems like a interesting place, loads of huge buildings, squares, fountains and roads. Things some how feel rundown, feels like a player trying to regain his former ability or glory.

Opening Keynote by Jim McCarthy about how teams should operate was interesting, he proposed 11 principals or protocols as he calls them, to be followed by members in a team so that the team becomes more productive, many of these protocols are about avoiding waste and promoting clear communication channels.

Promoting Thoughtworks

Tue, 03 Apr 2007 00:00:00 +0000

I have been working at Thoughtworks for 8 years now, its a fun place to work. Reason I’m blogging about this, ThoughtWorks is hiring in the US, UK, Australia, India, China and Canada. So go ahead send your resume to work@thoughtworks.com

Winning a award

Tue, 03 Apr 2007 00:00:00 +0000

Last week I received the good news. The book I co-authored with Scott Ambler won the 2007 Jolt Productivity Award in the Technical Books category. I was dumb enough not to attend the awards ceremony and receive the award, anyway when I started on the book project couple of years back I was afraid if the book would do justice to the Martin Fowler signature series, under which this book appears. The Jolt award award and all the comments I have received from many people in the last year, put me at easy, give me the feeling that finally I can relax and not worry about letting down Martin’s signature series.

Data Quality and Code Quality

Thu, 15 Feb 2007 00:00:00 +0000

Recently we had peculiar problem. Some of the data in the database was not being created in a proper fashion. Once we found that out we fixed the problem in the application. The customer still had the perception that the code is still broken, because the fixed code was now interacting with the data that was broken (since it was created much earlier by code that was broken). Data has a life of its own (more on this later)

Nulls need special love

Thu, 15 Feb 2007 00:00:00 +0000

The following SELECT statement in code

stmt = DB.prepare("select id,name,state,zip " +
        "from customer " +
        "where " +
        "phone = ? " +
        "and active = ?");
stmt.setString(1, customerPhone);
stmt.setBoolean(2, isActive);
stmt.execute();

where customerPhone and isActive are values you would pass in to the SELECT before its executed. Everything is fine when one day the value passed for customerPhone is NULL. For a database (Oracle is what I know most) a NULL will never be equal to NULL , the SELECT will not return rows where the customer.phone is NULL, leading to wrong results. The SELECT will have to be changed to

Implementing Make Column Non Nullable

Fri, 15 Dec 2006 00:00:00 +0000

While working on a Legacy Application with Legacy Database design as part of fixing a bug, I thought this bug would not have ever happened if a particular column was defined as Non Nullable since this particular column was the identifier to the parent table.

We had a Customer table and the all the names a customer could have like LegalName, LongName, ShortName etc are stored in the CustomerName table. CustomerName cannot exist without Customer hence its logical that the CustomerName.CustomerID column cannot be nullable and should also have a Foreign Key constraint enforcing the relationship. Implementing just the Foreign Key constraint is not enough since the application could potentially be inserting null values in the CustomerName.CustomerID creating orphan records.

Database Testing revisited

Thu, 09 Nov 2006 00:00:00 +0000

Some time ago I wrote about what it means to do database testing.. more I think about this and having had some strange situations recently I want to add more to the list of things we should be testing.

Persistence Layer We should persist the objects to the database using the applications persistence layer and retrieve the objects using the same mechanism and test that we get the same object back. If we have a lot of business logic in out persistence layer we may also want to retrieve the object using Direct SQL and test that the correct values got persisted.

Kids teaching International Relations"

Tue, 26 Sep 2006 00:00:00 +0000

You learn a lot from kids, and this lesson I will never forget.

So I’m at Lyon (in France) airport and there are a couple of kids (4-5 year olds) playing in the airport play area which had slides, call these two kids TA and TB my daughter call her AA joins these kids and they all start playing with the slide.

All is going well when a family with two kids, mom, dad and grandma join in, mom and dad are staying away and grandma takes the kids lets call them FA and FB to play on the slides. Grandma makes the previous kids (AA, AB, AA) stop playing and clears the slide for her grand kids (EA and EB) to play the slide. Grandkids start playing for a while and TA and TB are just waiting around not knowing what to do. AA by now has moved on to a different toy in the airport.

Database Migration Utility

Mon, 28 Aug 2006 00:00:00 +0000

I have taken up the hobby of searching the opensource landscape for tools that help me do Agile database development. I’m going to write about all the Tools that I come across that help me, my preference is opensource software but not limited to it. I will try to provide some sound examples and share my experiences with all that tools that I come across and share the example code I used.

Inactivity"

Wed, 26 Jul 2006 00:00:00 +0000

I had been on vacation for sometime. Started back work and getting connected with work and life again.

Database Testing

Tue, 23 May 2006 00:00:00 +0000

What does it mean to test your Database? usually when someone mentions database testing, what is that they want to test. The application code that interacts with the database, or the sql code the resides in the database like stored procedures and triggers etc. I see all these aspects to database testing as important.

Testing the applications persistence mechanism We should test that the application persists what its supposed to save and retrieve the data using SQL and see if the database contains the same information that is being saved, this kind of testing makes sense when the application has complex persistence layer. This type of testing can be achieved using unit tests, functional tests etc.

Move your DBAs to the Project team locations"

Tue, 23 May 2006 00:00:00 +0000

Many IT organizations I have seen have groups of specialists, typical are UNIX Group, DBA group etc.

When a project starts the developers on the project have to meet with all the Groups (I have more experience with the DBA group, so I write with the DBA group in mind) that they need to interact with and explain to the groups their design, the projects operational needs and other requirements, later when development starts they have to email these groups about all the setup that needs to be done and also the day to day changes that are needed. This way of working slows down the productivity of the team and the organization.

Automated Tablespace deployment

Mon, 01 May 2006 00:00:00 +0000

In development mode you don’t want to worry about which table goes into what Tablespace in production as it complicates development environments. The production DBA’s want to have their input and control over deciding what table goes into what Tablespace. To allow for this I used a mapping scheme as shown below.

Lets assume we have 3 tables in our system Customer, CustomerOrder, OrderStatus. Where we are expecting Customer table to have large numbers of rows and CustomerOrder to have significanly large number of rows while OrderStatus would have few rows and not change as much. In development environments all these tables and their indexes will be put under the same tablespace. In production like environments we want to put them into seperate tablespaces.

The day I received my Boo

Sun, 26 Mar 2006 00:00:00 +0000

This last week I received my copies of the book, for the first time I could touch what I had worked on for almost a year and half now, I could see it flip the pages. When I opened the UPS box and saw the book for the first time it felt great but in a minute or two I was not feeling anything, after some tea and some walking around I took the book in my hands and started flipping the pages this is when the real feeling started to sink in, but still I was not sure of the situation, was not really sure if it was real, after about 2-3 hours when I saw the book and held the book and started reading, I was started feeling the joy the happiness and all the goodness. I think when I see it being read by others then I will feel a lot different, the other day I saw Nick take out the book out of his backpack I was thrilled to see it being read.

To allow NULLs or NOT

Wed, 08 Feb 2006 00:00:00 +0000

Database designs I have seen tend to not constrain the data in the database. For example make the Item.ManufacturerID non-nullable and make it a foreign key to the Manufacturer table. Similarly Manufacturer.Name and Item.Rate as non-nullable columns. In any greenfield application (existing production application is a topic for another post). When you design table(s) lets say you have Item and Manufacturer table as shown below

CREATE TABLE Item
(ItemID NUMBER NOT NULL,
ManufacturerID NUMBER,
Name VARCHAR2(128),
Rate NUMBER,
CONSTRAINT PK_ITEM
              PRIMARY KEY (ItemID)
);

CREATE TABLE Manufacturer
(ManufacturerID NUMBER NOT NULL,
Name VARCAHR2(128),
CONSTRAINT PK_MANUFACTURER
              PRIMARY KEY (ManufacturerID)
);

For now let’s talk about NOT NULL constraint. Many say they don’t make columns non-nullable, because they don’t know the requirements at design time, or they don’t want their tests to create elaborate sets of data, or that the application enforces the constraint then why enforce the constraint on the database?

Communication

Fri, 03 Feb 2006 00:00:00 +0000

Why is that many a time I say something and the person hears something else.

I have been wondering is it me or is it just the way people listen or interpret my words. How can I communicate better, making it easier for myself and cause less of an hassle for myself and all the ones around me.

When I get to the bottom of this, it will be one heck of an achievement.

Refactoring Data

Thu, 19 Jan 2006 00:00:00 +0000

Many a times Refactoring is talked about in the context of code, recently I finished working with Scott Ambler on Database Refactoring

Lately I have been working on changing data in an production database, and have been wondering how do I define it, Data Refactoring? what are the patterns of Data Refactoring. First let me talk about what I mean by Data Refactoring.

When a given application goes into production, and starts life as a live application we find bugs with the application, these bugs create a weird data in the database, also with the way people change data through the app and some times through the database (yikes) and these data changes do lead to bad data. How do you go about fixing these data problems, are there patterns to these fixes.

Mon, 01 Jan 0001 00:00:00 +0000

Pramod Sadalage is Director at ThoughtWorks leading the Modern Data Architecture service for North America, he enjoys the rare role of bridging the divide between database professionals and application developers. He is usually sent in to clients with particularly challenging data needs, which require new technologies and techniques. In the early 00’s he developed techniques to allow relational databases to be designed in an evolutionary manner based on version-controlled schema migrations.

Mon, 01 Jan 0001 00:00:00 +0000

Software Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures

by Neal Ford, Mark Richards, Pramod Sadalage & Zamak Dehghani

All software architecture involves trade offs. But traditional analysis tools don’t work well for today’s distributed systems.

This book provides techniques to help you discover and weigh the trade-offs as you confront the issues you face as an architect. It investigates why architecture is so difficult and provides proven mechanisms to address these complex problems and make them understandable. Co-authors Neal Ford, Mark Richards, Pramod Sadalage, and Zhamak Dehghani examine everything from how to determine service granularity, manage workflows and orchestration, manage and decouple contracts, and manage distributed transactions to how to optimize operational characteristics, such as scalability, elasticity, and performance.

Mon, 01 Jan 0001 00:00:00 +0000

Books I have read recently and enjoyed.

Mon, 01 Jan 0001 00:00:00 +0000

Over the many years I have given many talks at conferences, here are some selected talks available freely.