Posts on Architecture and Data Blog

Collaborate to improve data quality

Thu, 24 Sep 2026 00:00:00 +0000

When application has been running in production or uat for some time there are some bugs that get reported because of bad data quality. This data is either generated by the application or is created by data migration scripts, data conversion scripts. The application suffers with non-deterministic errors especially is some specific environments while similar errors are not noticed in other environments.

Automated testing of custom sql

Tue, 23 Sep 2025 00:00:00 +0000

Lots of time hand coded SQL is written in the applications for performance sake, getting data using complex SQL out of the tables. As the database structure changes it becomes really hard to find out all the SQL that needs to be changed. In other cases SQL is generated based on certain conditions in the code and its hard to find if the generated SQL is valid after database changes are done.

Generate boiler plate code

Sat, 17 May 2025 00:00:00 +0000

When accessing the database using stored procedures for basic Create, Read, Update and Delete (CRUD) functions, or when you want to write triggers that capture the before and after values in tables, or when you want to create Plain Old Java Objects (POJO’s) that match the database objects, writing these by hand takes a lot of effort and also since the requirements are changing in an agile projects at a frequent rate, there will be design changes to meet the requirement changes, so the triggers, CRUD stored procedures or Data Access Objects (DAO) are going to be out of data, instead of hand coding, its better to generate the code, using the metadata of the database

Data specialists should pair with developers

Thu, 01 Aug 2024 00:00:00 +0000

Traditionally the data-team is used to sitting in their own area and working for many project teams by handling requests either via a ticketing system or vi email. The hand-over of work or throwing of work over the wall creates knowledge silos and inefficiencies.

Publish data models in CI Pipeline

Sat, 16 Sep 2023 00:00:00 +0000

Many a times ER models are created by the data team and are not shared outside of the data team generally for the lack of tools licenses, since its not feasible for the entire team to purchase licenses for the ER modelling tools such as Erwin Data Modeller or Er Studio

Trusting search results from Google & Others

Fri, 04 Dec 2020 00:00:00 +0000

Search engines have been great to find information on the internet with their ease of use and the ability to find cross linked content with specific keywords. This ease of use is being exploited by scammers and imposters.

SSL Connection to AWS Aurora

Mon, 15 Jul 2019 00:00:00 +0000

On a recent project we had to connect to AWS Aurora postgres 10.6 version of the database in SSL mode using JDBC and Java 11 JRE. When the Aurora cluster is setup, we can force all connections to use SSL by using the options group settings (forceSSL=true), establishing secure connection from the application to the database is not as easy as it looks.

Static Analysis of PL/SQL code

Tue, 01 Aug 2017 00:00:00 +0000

In all new development and sometimes during legacy codebase modernization, developers tend to add code quality checks and static analysis of codebase such as style checks, bug finders, cyclomatic complexity checking etc. into the CI/CD pipeline. When we inherit a codebase that has much PL/SQL and there is a desire to put the PL/SQL code base through the same types of code analysis, what options does a developer/dba have?

ID generation with ORM's Table, Sequence or Identity strategy

Mon, 12 Jun 2017 00:00:00 +0000

In this blog post, I will discuss ID generation techniques using the Object Relation Mapping frameworks such as Hibernate, Toplink, ActiveRecord, Entity Framework. When using hibernate or any other ORM mapping framework. There is a need to generate primary key values for the “id” columns. These values can be generated by using IDENTITY, SEQUENCE or TABLE strategies. Generating custom values for primary keys or other values is a topic for another blog post.

Using liquibase to load data and ignore some columns

Mon, 15 May 2017 00:00:00 +0000

Loading data into tables is needed many times on projects to load test, Liquibase provides a method to load data into tables with lots of customization. In the example shown below, I’m loading zip code data with the following column layout

"Zipcode","ZipCodeType","City","State","LocationType","Lat","Long","Location","Decommisioned","TaxReturnsFiled","EstimatedPopulation","TotalWages"

Two factor authentication to authorize credit"

Wed, 13 Jul 2016 00:00:00 +0000

Automatically adding columns to Rails migrations

Mon, 25 Apr 2016 00:00:00 +0000

Many projects need addition of identical columns to all the tables created by the project. Audit columns are an example of such a requirement. The requirement is to add columns such as created_by, created_date, modified_by and modified_date to all the tables, these columns store, who created the row, when the row was created, who modified the row last and when was it modified. created_by and created_date are required to be present when the row is inserted and thus are required to be not nullable. Adding these columns to each and every table is a lot of work for developers.

Synonyms as abstraction layer

Fri, 15 Apr 2016 00:00:00 +0000

In many development shops, developers are not allowed to access the database schema directly, and are not allowed to create tables, indexes, views etc, instead are given access via a different schema that allows SELECT, UPDATE and DELETE access on data. The general reason is to avoid developers creating database objects without

Behavior Driven Database Development

Sun, 16 Aug 2015 00:00:00 +0000

When Behavior Driven Development BDD was introduced, some of the key principles were

Requirements are behavior,
Provides “ubiquitous language” for analysis,
Acceptance criteria should be executable.
Design constraints should be made into executable tests.

Using rake and activerecord to generate boilerplate DB Code

Mon, 20 Jul 2015 00:00:00 +0000

IN many projects, there are tables which need default audit columns such as Created_By, Created_Date, Modified_By, Modified_date and other columns that need to be updated every time some actions are done against the tables and/or columns. This type of functionality can be implemented using triggers.

Database naming conventions in different environments

Wed, 10 Jun 2015 00:00:00 +0000

In every enterprise and every project we end up having multiple environments, especially the database side of the enterprise tends to stick around for a longer period of time and has much more dependencies or application integration as opposed to application urls etc. Given this, how to name the servers, databases and schemas becomes a very important decision, do these names provide for an easy way to use the application and not make it harder or the developers to access the database.

8 Techniques for testing migration of data from legacy systems

Fri, 23 Jan 2015 00:00:00 +0000

Many of the projects we end up working on are replacing existing systems with existing data either wholly or in part. In all of the above projects we end up writing data migration or data conversion code to move the data from legacy systems to the new systems. Many stake holders of the project such as business users, project managers, business analysts really care about the data conversion scripts and the quality of the conversion. Since this conversion is business entity related and matters a lot as future business/functionality depends on the data being logically equivalent to the legacy system.

Migrations in NoSQL databases

Tue, 14 Oct 2014 00:00:00 +0000

In relational database usage the pattern of migrations is well understood and has gained widespread acceptance. Frameworks such as DBDeploy, DBMaintain, MyBatis migrations, Flyway, Liquibase, Active Record Migrations and many others. These tools allow to migrate the database and maintain the version history of the database in the database.

10 node mongodb ReplicaSet on a Single Machine"

Mon, 09 Jun 2014 00:00:00 +0000

While doing evalauation of NoSQL databases, we had a 10 node riak cluster and wanted check how a similar setup would work with mongodb. So started to setup a 10 node mongodb cluster. Since this was for initial spikes, we decided to set this up on a single machine as with the other test setup using Riak.

Moved my blog to octopress

Fri, 31 Jan 2014 00:00:00 +0000

Its been about a month since my blog moved to octopress, wanted to write about my experience. I had been running my blog for a some time now using Movable Type upgrading as and when new versions where released. Over time I realized that upgrading was fraught with errors as lot of steps had to be done manually. Customizing the layout was risky as there was no way to preview your changes and commit only when I was comfortable. With the release of Movable Type 6 there is no longer a free version to download.

Usage of mixed case database object names is dangerous

Tue, 06 Aug 2013 00:00:00 +0000

Some versions back, Oracle would not allow to create database object names with mixed cases, even if we tried to create them, we could not. In newer versions of Oracle we can create tables, columns, indexes etc using mixed case or lower case, when the names are put inside double quotes. For example

10 node Riak cluster on a single machine

Thu, 25 Apr 2013 00:00:00 +0000

When trying to evaluate NoSQL databases, its usually better to try them out. While trying them out, its better to use them with multiple node configurations instead of running single node. Such as clusters in Riak or Replica-set in mongodb maybe even a sharded setup. On our project we evaluated a 10 node Riak cluster so that we could experiment with N, R and W values and decide which values where optimal for us. In Riak here is what N, R and W mean.

Transactions using Groovy.SQL with Spring annotations and connection pools

Mon, 14 Jan 2013 00:00:00 +0000

When using Groovy with Spring framework, interacting with the database can be done using the Groovy.SQL class which provides a easy to use interface. When using Groovy.SQL, if we have a need to do transactions, we have the .withTransaction method that accepts a closure, to which we can pass in code to execute within the transaction. In our project since we were using spring already, using annotations to define transactions would be a great. Standard @Transactional annotations with Groovy.SQL will not work, since every place where the Groovy.SQL is used a new connection is acquired from the connection pool causing the database work to span multiple connections, which can result in dead-locks on the database. What we really want is that the database connection be the same across all invocations of Groovy.SQL with-in the same transaction started by the annotated method.

Backup in mongodb replica-set configurations

Thu, 27 Dec 2012 00:00:00 +0000

There are multiple ways to take backups of mongodb is different configuraitions, one of the configuration that I have been involved recently is replica-sets. When mongodb is running in replica-set configuration, there is a single primary node and multiple secondary nodes. To take backup of the replica-set we can either do a mongodump of one of the nodes or shutdown one of the secondary nodes and take file copies, since in a replica-set all nodes have the same data (except arbiter). Lets see how we could deal with mongodump method of taking backup.

Back to blogging

Sun, 22 Apr 2012 00:00:00 +0000

There has been a long pause in my blogging activity. I was trying to finish of my latest writing engagement in regards to NoSQL. Working with Martin Fowler on NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence was really fun. This book will provide a concise text and easy way to understand for everyone the rise of the NoSQL movement and help with what kinds of trade-offs need to be made while working with NoSQL.

MSSQL JDBC Driver behavior"

Sun, 22 Apr 2012 00:00:00 +0000

My latest project involves talking to MS-SQL Server using the JDBC driver and Java. While doing this we setup the database connection and had a simple SQL to get the first_name and last_name for a unique user_id from the application_user table in the database.

With so much pain, why are stored procedures used so much

Wed, 19 Jan 2011 00:00:00 +0000

I keep encountering situations where all the business logic for the applications is in stored procedures and the application layer is just calling the stored procedures to get the work done and return the data. There are many problems with this approach some of them are.

Replica sets in MongoDB

Sun, 31 Oct 2010 00:00:00 +0000

Replica sets is a feature of MongoDB for Automatic Failover, in this setup there is a primary server and the rest are secondary servers. If the primary server goes down, the rest of the secondary servers choose a new primary via an election process, each server can also be assigned number of votes, so that you can decide the next primary based on data-center location, machine properties etc, you can also start mongo database processes that act only as election tie-breakers these are known as arbiters, these arbiters will never have data, but just act as agents that break the tie.

Schema less databases and its ramifications.

Tue, 12 Oct 2010 00:00:00 +0000

In the No-SQL land schema-less is a power full feature that is advertised a lot, schema-less basically means you don’t have to worry about column names and table names in a traditional sense, if you want to change the column name you just start saving the data using the new column name Lets say you have a document database like mongoDB and you have JSON document as shown below.

Effective use of data for better customer experience.

Tue, 31 Aug 2010 00:00:00 +0000

For more than seven years I have been getting offers for credit cards from Airlines and Banks. One particular bank has been sending me these solicitations for more than seven years. That is 12 mailings per year, more than 72 mailings so far, remember these are physical paper mailings not the electronic kind. I don’t like the junk, it hurts the environment and worst of all I think its not good use of the data they have. How hard is it to design a system around the data they have.

Schema design in a document database

Wed, 28 Apr 2010 00:00:00 +0000

We are using MongoDB on our project, since mongo is document store, schema design is somewhat different, when you are using traditional RDBMS data stores, one thinks about tables and rows, while using a document database you have to think about the schema in a some what different way. Lets say, we want to save a customer object, when using a RDBMS we would come up with Customer, Address, Phone, Email. They are related to each other as shown below.

My experience with MongoDB

Sun, 18 Apr 2010 00:00:00 +0000

The current project I’m on is using MongoDB. MongoDB is a document based database, it stores JSON objects as BSON (Binary JSON objects). MongoDB provides a middle ground between the traditional RDBMS and the NOSql databases out there, it provides for indexes, dynamic queries, replication, map reduce and auto sharding, its open source and can be downloaded Mongodb, starting up mongodb is pretty easy.

Workshop at Enterprise Data World 2010

Fri, 29 Jan 2010 00:00:00 +0000

Doing a workshop on Agile Database Development at Enterprise Data World 2010 at SF. See you there.

Testing in data conversion projects

Wed, 18 Nov 2009 00:00:00 +0000

When working on projects involving Conversion of data or Migration/Moving of data from a legacy database. The testing effort is enormous and testing takes a lot of time, some test automation can help this effort.

Ruby OCI 2.0 Array binding

Thu, 08 Oct 2009 00:00:00 +0000

We have been doing some data moving lately using Ruby and Ruby-OCI. We started with Ruby OCI 1.0 and did use prepared statements with bind variables (since we are using oracle database and pulling data from an oracle database and pushing data to an oracle database). Later we found this really cool feature in Ruby-OCI8 2.0 where you can bind a whole array and just make one database trip for many database operations.

Create an Index for all FK Columns in the database

Thu, 03 Sep 2009 00:00:00 +0000

Most of the time I have seen database foreign key constraints on tables without indexes on those columns. Lets say the application is trying to delete a row from the CUSTOMER table

Materialized views and database links in oracle.

Mon, 10 Aug 2009 00:00:00 +0000

Recently one of my colleague Jeff Norris had a weird error. He was trying to build a materialized view over some tables in his local database and some tables in his remote database using database links the sql to create the view ran fine and provided the results as expected, but when put inside a materialized view statement complained with ORA-00942 errors.

Perfectly good data.. wasted

Wed, 05 Aug 2009 00:00:00 +0000

Okay this is kind of a rant, maybe I’m too picky or just that I hate to see perfectly good data not being used. This is how it goes..

I go regularly to this store to get Horizon organic milk for my family, about 60% of the time I see milk I need NOT in stock, okay I can live with that, may be lots of folks are buying organic milk, but not when it happens frequently, especially when the store knows how much milk was ordered (or supplied from the warehouse) and how much milk was sold, the store should be able to figure out that organic milk gets sold out pretty fast, putting my Business Intelligence (BI) hat on, I think the store should be able to predict when they are going to run out of organic milk ( for that matter any product), its especially frustrating when they have all the data they need to get it done.

Explicitly rollback when you encounter a deadlock.

Tue, 26 May 2009 00:00:00 +0000

Dead lock is caused in the database when you have resources (connections) waiting for other connections to release locks on the rows that are needed by the session, resulting in all session being blocked. Oracle automatically detects deadlocks are resolves the deadlock by rolling back the statement in the transaction that detected the deadlock. Thing to remember is that last statement is rolled back and not the whole transaction, which means that if you had other modifications, those rows are still locked and the application should make sure that it does a explicit rollback on the connection.

Oracle for the Mac

Thu, 14 May 2009 00:00:00 +0000

Ever since I moved to the Mac, I had to run some other OS inside a VM so that I could run Oracle and use it, since Oracle was not available for the the Mac. Now that is no longer the case. Oracle 10gR2 (10.2.0.4) is now available for Mac

In Oracle 11g password is case sensitive

Wed, 06 May 2009 00:00:00 +0000

In Oracle 10g and before we all know that passwords are not case sensitive, so PASSWORD, Password, password would let you in and everything would be okay.

If you upgrade to Oracle 11g (I know lot of you are waiting for 11gR2), you will find that passwords are case sensitive. Here is an example of case sensitive passwords.

Oracle Metadata can be mis-leading

Tue, 31 Mar 2009 00:00:00 +0000

Oracle has metadata about all its objects in various tables/views. One such view is the USER_OBJECTS or ALL_OBJECTS, this view has a column named as STATUS which shows you if the given object is VALID or INVALID. The status applies to DB Code (Stored Procedures, Functions, Triggers etc).

Presentation on Database Refactoring

Thu, 26 Mar 2009 00:00:00 +0000

My Presentation on Database Refactoring at QCon was recorded and is live now on infoQ here

Storing just the time in Oracle

Thu, 12 Feb 2009 00:00:00 +0000

We came across a need to save just the Time in the database, the requirement is to store time of the day, like say the user likes to have Breakfast at 8.15AM and Lunch at 12.32PM etc. Off course oracle does not have a time only data type. So we ended up using DATE as the data type and just setting the time. for example:

Considerations when using stored procedures for doing CRUD

Tue, 03 Feb 2009 00:00:00 +0000

Some environments like to have access to the database tables routed via stored procedures. Instead of using Create/Read/Update/Delete aka CRUD with DML, stored procedures are invoked with the parameters to perform the required operation. I’m not arguing about the benefits/pitfalls of this approach, if you have to do stored procedures, here are some things to look at.

Article about Behavior Driven Database Design is out

Mon, 22 Dec 2008 00:00:00 +0000

Recently I have been thinking a lot about making specification and behavior expected about the database and the code that interfaces with the databases executable. The Behavior Driven Design has a lot of parallels in the database world.

Hibernate weirdness with property names

Fri, 24 Oct 2008 00:00:00 +0000

Consider this Hibernate mapping

@Column(name = "qReferenceId")
public Long getQReferenceId() {
return qReferenceId;
}

Where qReferenceId is data provided to our application via a external reference, we do not have a QReference Object or Table for FK references. When trying to query this object using DetachedQuery, this Simple expression was used.

Using Explicit Order By in your SQL Statements

Tue, 30 Sep 2008 00:00:00 +0000

Recently when our test databases where upgraded new version of Oracle, we started noticing that the order in which some drop down lists were being displayed was not correct. It turns out that the SELECT statement we had, did not have a ORDER BY clause and the data was being returned in the ORDER of the creation of rows (in the order of ROWID) when the database was upgraded these ROWID’s got changed and hence the ORDER of the data being shown in the drop down lists.

Moved to a Mac

Mon, 01 Sep 2008 00:00:00 +0000

Couple of weeks back I was given a choice to upgrade my work Laptop to a Mac Book Pro or a Windows Laptop. I choose Mac ( I know everyone is into macs nowadays). The transition was pretty good, with the exception of moving my oracle database from windows to mac, since there is no native installation of oracle on mac I had to use VMWare fusion to install oracle.

Create a Index for every Foreign Key constraint created

Tue, 15 Jul 2008 00:00:00 +0000

When creating a Foreign Key constraint on the database as shown below

ALTER TABLE BOOK ADD
    (CONSTRAINT FK_BOOK_AUTHOR FOREIGN KEY (AUTHORID)
     REFERENCES AUTHOR)
/

In the above example we are telling the database to check if the BOOK.AUTHORID is a valid value in the Author.AuthorID. When the Author table is being changed, the database does data verification on the BOOK table using SELECT against the BOOK table for the AUTHORID some thing like this

Version Control your work..

Fri, 06 Jun 2008 00:00:00 +0000

So we version control/source control everything on our project.. code/data/artifacts/diagrams etc. yesterday I said why not extend it to my writings to everything I have. So I started this long journey of refactoring my folder layout and making a nice folder structure to hold all the things I have written about have other artifacts in the process of writing and moved them all to subversion, now all my example code and writings are all under version control that gets backed up everyday…. feels liberating

Japanese Version released"

Wed, 04 Jun 2008 00:00:00 +0000

Japanese translation of Refactoring Databases: Evolutionary Database Designhas been released, thanks to Yasuo Honda for the information. The Japanese version can be found here

Tool support for Database Refactoring

Tue, 20 May 2008 00:00:00 +0000

I’m always on the lookout for better tool support to do database refactoring. Just noticed that liquibase has come out with a IntelliJ plugin to support database refactoring.

This is really cool and hopefully one of long list of tools that will support database refactoring in the future. so enjoy

Writing a SQL to generate a SQL

Tue, 06 May 2008 00:00:00 +0000

We had a weird requirement on our project recently..

Find all the Rows in All the tables that do not comply with the Constraints that we have in development but not in QA environments

Setup and Teardown of database during testing

Thu, 17 Apr 2008 00:00:00 +0000

When doing Performance Testing or running Unit/Functional tests on a database, there is a need to periodically get the database to a known state, so that the tests behave in a predictable way and to get rid of all the data created by the tests. Some of the ways to get a clean database are.

Podcast on Keeping Gray code Fit

Fri, 18 Jan 2008 00:00:00 +0000

Me, Andy, Jeff and Marjorie discuss how to keep a long running project fit in this Podcast, also on iTunes. We discuss the management of technology, people, processes and tools on longer and more mature applications. Specific topics such as refactoring, knowledge management, innovation, staffing, production support and others are covered.

Experience using DBDeploy on my project

Mon, 14 Jan 2008 00:00:00 +0000

We have been using DBDeploy on my project for more than 6 months now and wanted to show how things are going. First lets talk about set up, we are using dbdeploy in our Java development environment with ANT as our build scripting tool, against a Oracle 10g database.

Lessons from Re-Learning

Thu, 27 Dec 2007 00:00:00 +0000

I have been working on a project that I had worked in 2005, trying to get a handle on what I had done about 3 years back. Exploring code and the database has been fun, also discovering the data layout and building new set of data for production has been extremely entertaining. What I learnt from this whole experience was this if your code(application or other wise) is not expecting data, this data should not be provided or even considered valid by the database, the database should be designed such that it does not even allow invalid combinations of the data.

Cannot make it out to XP Day London

Tue, 13 Nov 2007 00:00:00 +0000

After a lot of frustration about my schedule, I have had to come to this conclusion, that I cannot physically make it to London, XP day.

I’m going to miss it. Nick Ashley is going to take up my spot and I know he will do a great job.

Why do Evolutionary Design

Mon, 05 Nov 2007 00:00:00 +0000

Why do Evolutionary Design or Iterative Design or Incremental Design? Everyone who has not worked in an evolutionary manner asks this? My answer, if you think the system you designed is NOT GOING TO CHANGE EVER then sure you can do design once and deploy once and you are done, move on to next project. But tell me one project you have been on, that does not have any changes in requirements, changes in technology, changes in look and feel etc after it was deployed.

ThoughtWorks at Oracle Open World

Tue, 30 Oct 2007 00:00:00 +0000

Thoughtworks is going to be at Oracle Open World. I’m excited about this especially since it will give ThoughtWorks and Me to talk about software practices and how to apply these software practices to the database development world, off course I will talk about my books Refactoring Databases and Continuous Database Integration. ThoughtWorks will have a booth at 343 Moscone South and I will be there on Nov 14.

Frameworks should not constrain your table design

Wed, 03 Oct 2007 00:00:00 +0000

Recently talking to someone about a persistence framework that they are using, this framework creates a rows in the database table with just Primary Key value and then later on updates the table row with values for other columns. Because of this framework, the tables cannot have any NOT-NULL constraints defined.

Lesson learnt changing firewall settings

Thu, 13 Sep 2007 00:00:00 +0000

On any machine running network related services, like in my case Oracle Listener and Dispatch Services. Don’t rebuild/stop and restart the firewall stuff like iptables.

Had to spend a lot of time, figuring out what was going on. So hard lesson learnt don’t mess with iptables when you are running oracle listener/dispatcher

When does evolutionary design happen?

Tue, 28 Aug 2007 00:00:00 +0000

A question I get, mostly related to the evolutionary database design and development. When the pair (team) gets a new feature (story) to work on, the team looks at the existing table/database design and sees if the current design is enough to implement the feature they are working on. If the currency database design does support the feature they are trying to implement, then they do not have to change the database at all, they will move on to implement the feature and change the application code as necessary.

Parsing mapping files for usage information

Thu, 19 Jul 2007 00:00:00 +0000

Currently working on a legacy application, thats been in production for a long time now. I wanted to find out what are the Tables and Columns being used by the application. Since we could see that some table columns where not being used. We are using a Object Relational mapping framework on the project, so we decided to write some code that would parse all the mapping files and gives us a list of table names and columns. We used this list to create rows in a table with two columns tablename and columnname. Once the table had this data, we just ran one more SELECT against the metadata of the database and our table which pretty much gave us a list of Table and Columns that we are not using

Long Running Data Migrations during Database Refactorings

Fri, 29 Jun 2007 00:00:00 +0000

When you are refactoring large databases, you will have certain tables that have millions of rows, so lets say we are doing the Move Column refactoring, moving the TaxAmount column from Charge table which has millions of rows to TaxCharge table. Create the TaxAmount column in the TaxCharge table. Then have to move the data from the TaxAmount column in the Charge table to the TaxAmount column you created in the TaxCharge table.

Enforcing your assumptions about database functionality

Fri, 08 Jun 2007 00:00:00 +0000

When you are writing xUnit tests you are in certain ways trying to make sure that the test breaks when the code that is being tested changes the assumptions you made when writing the Test and Production code.

XP2007 Tutorial

Fri, 08 Jun 2007 00:00:00 +0000

I will presenting about Database Refactoring: Evolutionary Database Design at XP 2007 here is the Tutorial schedule

ThoughtWorks Master class series in India

Tue, 08 May 2007 00:00:00 +0000

I will be presenting about Evolutionary Database Design and Database Refactoring at ThoughtWorks Master Class Series 2007 at Bangalore on May 19th and Pune on May 26. This will be the first time I will be presenting in India.

My Latest eBook is Out"

Mon, 23 Apr 2007 00:00:00 +0000

After finishing the first Refactoring Databases book. I started on a short ebook project, this book was going to tackle on a very specific technical topic mentioned in the first book. I wanted to write about all the specific scenarios and all the techniques I follow on the various projects.

Moscow

Wed, 11 Apr 2007 00:00:00 +0000

Last week I was at SD Best Practices in Moscow, doing a presentation on “Refactoring Databases: Evolutionary Database Design”. Moscow seems like a interesting place, loads of huge buildings, squares, fountains and roads. Things some how feel rundown, feels like a player trying to regain his former ability or glory.

Promoting Thoughtworks

Tue, 03 Apr 2007 00:00:00 +0000

I have been working at Thoughtworks for 8 years now, its a fun place to work. Reason I’m blogging about this, ThoughtWorks is hiring in the US, UK, Australia, India, China and Canada. So go ahead send your resume to work@thoughtworks.com

Winning a award

Tue, 03 Apr 2007 00:00:00 +0000

Last week I received the good news. The book I co-authored with Scott Ambler won the 2007 Jolt Productivity Award in the Technical Books category. I was dumb enough not to attend the awards ceremony and receive the award, anyway when I started on the book project couple of years back I was afraid if the book would do justice to the Martin Fowler signature series, under which this book appears. The Jolt award award and all the comments I have received from many people in the last year, put me at easy, give me the feeling that finally I can relax and not worry about letting down Martin’s signature series.

Data Quality and Code Quality

Thu, 15 Feb 2007 00:00:00 +0000

Recently we had peculiar problem. Some of the data in the database was not being created in a proper fashion. Once we found that out we fixed the problem in the application. The customer still had the perception that the code is still broken, because the fixed code was now interacting with the data that was broken (since it was created much earlier by code that was broken). Data has a life of its own (more on this later)

Nulls need special love

Thu, 15 Feb 2007 00:00:00 +0000

The following SELECT statement in code

stmt = DB.prepare("select id,name,state,zip " +
        "from customer " +
        "where " +
        "phone = ? " +
        "and active = ?");
stmt.setString(1, customerPhone);
stmt.setBoolean(2, isActive);
stmt.execute();

where customerPhone and isActive are values you would pass in to the SELECT before its executed. Everything is fine when one day the value passed for customerPhone is NULL. For a database (Oracle is what I know most) a NULL will never be equal to NULL , the SELECT will not return rows where the customer.phone is NULL, leading to wrong results. The SELECT will have to be changed to

Implementing Make Column Non Nullable

Fri, 15 Dec 2006 00:00:00 +0000

While working on a Legacy Application with Legacy Database design as part of fixing a bug, I thought this bug would not have ever happened if a particular column was defined as Non Nullable since this particular column was the identifier to the parent table.

Database Testing revisited

Thu, 09 Nov 2006 00:00:00 +0000

Some time ago I wrote about what it means to do database testing.. more I think about this and having had some strange situations recently I want to add more to the list of things we should be testing.

Kids teaching International Relations"

Tue, 26 Sep 2006 00:00:00 +0000

You learn a lot from kids, and this lesson I will never forget.

So I’m at Lyon (in France) airport and there are a couple of kids (4-5 year olds) playing in the airport play area which had slides, call these two kids TA and TB my daughter call her AA joins these kids and they all start playing with the slide.

Database Migration Utility

Mon, 28 Aug 2006 00:00:00 +0000

I have taken up the hobby of searching the opensource landscape for tools that help me do Agile database development. I’m going to write about all the Tools that I come across that help me, my preference is opensource software but not limited to it. I will try to provide some sound examples and share my experiences with all that tools that I come across and share the example code I used.

Inactivity"

Wed, 26 Jul 2006 00:00:00 +0000

I had been on vacation for sometime. Started back work and getting connected with work and life again.

Database Testing

Tue, 23 May 2006 00:00:00 +0000

What does it mean to test your Database? usually when someone mentions database testing, what is that they want to test. The application code that interacts with the database, or the sql code the resides in the database like stored procedures and triggers etc. I see all these aspects to database testing as important.

Move your DBAs to the Project team locations"

Tue, 23 May 2006 00:00:00 +0000

Many IT organizations I have seen have groups of specialists, typical are UNIX Group, DBA group etc.

When a project starts the developers on the project have to meet with all the Groups (I have more experience with the DBA group, so I write with the DBA group in mind) that they need to interact with and explain to the groups their design, the projects operational needs and other requirements, later when development starts they have to email these groups about all the setup that needs to be done and also the day to day changes that are needed. This way of working slows down the productivity of the team and the organization.

Automated Tablespace deployment

Mon, 01 May 2006 00:00:00 +0000

In development mode you don’t want to worry about which table goes into what Tablespace in production as it complicates development environments. The production DBA’s want to have their input and control over deciding what table goes into what Tablespace. To allow for this I used a mapping scheme as shown below.

The day I received my Boo

Sun, 26 Mar 2006 00:00:00 +0000

This last week I received my copies of the book, for the first time I could touch what I had worked on for almost a year and half now, I could see it flip the pages. When I opened the UPS box and saw the book for the first time it felt great but in a minute or two I was not feeling anything, after some tea and some walking around I took the book in my hands and started flipping the pages this is when the real feeling started to sink in, but still I was not sure of the situation, was not really sure if it was real, after about 2-3 hours when I saw the book and held the book and started reading, I was started feeling the joy the happiness and all the goodness. I think when I see it being read by others then I will feel a lot different, the other day I saw Nick take out the book out of his backpack I was thrilled to see it being read.

To allow NULLs or NOT

Wed, 08 Feb 2006 00:00:00 +0000

Database designs I have seen tend to not constrain the data in the database. For example make the Item.ManufacturerID non-nullable and make it a foreign key to the Manufacturer table. Similarly Manufacturer.Name and Item.Rate as non-nullable columns. In any greenfield application (existing production application is a topic for another post). When you design table(s) lets say you have Item and Manufacturer table as shown below

Communication

Fri, 03 Feb 2006 00:00:00 +0000

Why is that many a time I say something and the person hears something else.

I have been wondering is it me or is it just the way people listen or interpret my words. How can I communicate better, making it easier for myself and cause less of an hassle for myself and all the ones around me.

Refactoring Data

Thu, 19 Jan 2006 00:00:00 +0000

Many a times Refactoring is talked about in the context of code, recently I finished working with Scott Ambler on Database Refactoring

Lately I have been working on changing data in an production database, and have been wondering how do I define it, Data Refactoring? what are the patterns of Data Refactoring. First let me talk about what I mean by Data Refactoring.