Architecture and Data Blog

Thoughts about intersection of data, devops, design and software architecture

Frameworks should not constrain your table design

Recently talking to someone about a persistence framework that they are using, this framework creates a rows in the database table with just Primary Key value and then later on updates the table row with values for other columns. Because of this framework, the tables cannot have any NOT-NULL constraints defined.

For example, the framework is doing

INSERT INTO customer (customerId, name, birthdate) VALUES (1,null,null);
UPDATE customer set name = 'FOO Name', birthdate = '12/12/1978' WHERE customerId = 1;

You cannot have NON-NULL constraint defined on name or birthdate column, since the INSERT statement would blow up, forcing you to change table design


Lesson learnt changing firewall settings

Don't change network configuration, when critical network configuration is in use

On any machine running network related services, like in my case Oracle Listener and Dispatch Services. Don’t rebuild/stop and restart the firewall stuff like iptables.

Had to spend a lot of time, figuring out what was going on. So hard lesson learnt don’t mess with iptables when you are running oracle listener/dispatcher


When does evolutionary design happen?

What part of the product, project, iteration, story or day does evolutionary design happen?

A question I get, mostly related to the evolutionary database design and development. When the pair (team) gets a new feature (story) to work on, the team looks at the existing table/database design and sees if the current design is enough to implement the feature they are working on. If the currency database design does support the feature they are trying to implement, then they do not have to change the database at all, they will move on to implement the feature and change the application code as necessary.


Parsing mapping files for usage information

Currently working on a legacy application, thats been in production for a long time now. I wanted to find out what are the Tables and Columns being used by the application. Since we could see that some table columns where not being used. We are using a Object Relational mapping framework on the project, so we decided to write some code that would parse all the mapping files and gives us a list of table names and columns. We used this list to create rows in a table with two columns tablename and columnname. Once the table had this data, we just ran one more SELECT against the metadata of the database and our table which pretty much gave us a list of Table and Columns that we are not using


Long Running Data Migrations during Database Refactorings

how to manage long running data migrations

When you are refactoring large databases, you will have certain tables that have millions of rows, so lets say we are doing the Move Column refactoring, moving the TaxAmount column from Charge table which has millions of rows to TaxCharge table. Create the TaxAmount column in the TaxCharge table. Then have to move the data from the TaxAmount column in the Charge table to the TaxAmount column you created in the TaxCharge table.


Enforcing your assumptions about database functionality

Testing the behavior of the database

When you are writing xUnit tests you are in certain ways trying to make sure that the test breaks when the code that is being tested changes the assumptions you made when writing the Test and Production code.

Similarly if you are relying on the database to throw a error when you put invalid data, then you should write a test around this assumption, so that when someone changes the database to not behave the way you assumed it to behave, the test you wrote will break and it will force the team to think about the change to the database that is being undertaken. If the change is really required, then the team would fix the test else rollback the change being made.


XP2007 Tutorial

Speaking at XP conference

I will presenting about Database Refactoring: Evolutionary Database Design at XP 2007 here is the Tutorial schedule


ThoughtWorks Master class series in India

I will be presenting about Evolutionary Database Design and Database Refactoring at ThoughtWorks Master Class Series 2007 at Bangalore on May 19th and Pune on May 26. This will be the first time I will be presenting in India.

The Master Class Series is an annual seminar organized by ThoughtWorks India. It focuses on topics which are cutting edge, but the content of the seminars is drawn from real-life experiences on live projects. The presenters are all people with extensive hands-on experience and have delivered successful projects using the concepts they are talking about.


My Latest eBook is Out"

After finishing the first Refactoring Databases book. I started on a short ebook project, this book was going to tackle on a very specific technical topic mentioned in the first book. I wanted to write about all the specific scenarios and all the techniques I follow on the various projects.

The result of this effort was Recipes for Continuous Database Integration: Evolutionary Database Development, thanks to Martin Fowler for the title. This ebook laid out in a recipes kind of format provides various ways you can integrate you database into you development cycle and make evolutionary database development a fun thing to work with


Moscow

How to use stored procedures as interface to the data

Last week I was at SD Best Practices in Moscow, doing a presentation on “Refactoring Databases: Evolutionary Database Design”. Moscow seems like a interesting place, loads of huge buildings, squares, fountains and roads. Things some how feel rundown, feels like a player trying to regain his former ability or glory.

Opening Keynote by Jim McCarthy about how teams should operate was interesting, he proposed 11 principals or protocols as he calls them, to be followed by members in a team so that the team becomes more productive, many of these protocols are about avoiding waste and promoting clear communication channels.