In a real-life project developed by one or more teams, the number of change sets can quickly grow. What are the best practices for organizing Liquibase changelog files? Should all change sets be put to one large change log file or does it makes sense to have a separate file for each release? Well. That depends on the particular project, but I can tell you how I usually do that.
A few days ago, one of the students of my Liquibase course asked me about best practices on project structure with Liquibase change log files. It let me realize, that I have not published anything about that. I was focused on various aspects of using Liquibase, but this subject deserves more attention. It seemingly is simple. You create a change log file and sequentially add change sets there. Actually, real-life projects may have a significant number of change sets. Then, a proper project structure is crucial for further development, maintenance, and overall readability of the project.
Factors driving project structure
Designing a structure of directories and deciding on how to spread change sets across change log files are driven by a few goals. The following factors have been important in my practice with Liquibase:
- Goal 1 - when you have to create a new change set, finding a proper change log file should be straightforward. Nobody wants to waste 10 minutes trying to find the change log file to add a single column to a table. Or debate whether the new change set fits this or that file. All change log files should be in a single place and/or have a predictable name, which is easy to search by. To eliminate doubts about where the new change set should go, the rules should be easy. Choosing a file is not the most important part of our job, so make it simple.
- Goal 2 - editing a file should be supported by at least a code completion. Modern IDEs are powerful. They follow the code structure and suggest what keyword should go next. They are aware of already existing objects and properties, so they prevent us from making typos. In Liquibase XML documents, having the XSD files they support us by providing a list of possible and most probable attributes and child nodes. There is no reason to resign from that enormous help. Unfortunately, these great editors may give up themselves, if the edited file is too big.
- Goal 3 - applying change sets to a database should be easy. Software development is not only about writing code. It is also deploying it. If we have taken simplified adding new change sets, let's make applying them on a database easy as well. If a project has multiple change log files, and each must be applied separately, that is not a good design.
These goals are mine. I mean, they have been important in projects I have worked on. There is a small chance, that you may have additional ones. But most probably, those three above are also goals in your projects.
Liquibase project structure
Over the years of working with Liquibase, I have got to this project structure that works well for me. Of course, not all elements are the same in every project, but I consider it as a template from which I can get the parts that I need.
Everything related to a database, that is not a Java code is inside a db directory. The upgrade.bat and rollback.bat files are there, but they are only examples. You may have them or not, depending on how you deploy Liquibase changes.
I have a special directory for all change log files - changelogs. There is changelog-master.xml (I will get back to it in a moment) directly in the folder and two subdirectories: regular and rerun.
For sure you need the regular one. Probably most change sets, that you create go there. All these createTable, addColumn, but also adding, updating, and deleting data are regular changes that should be executed once. If a particular change set does not have a special place, it should go to the regular directory. Ok, but the screenshot above shows three files in it. So to which one should I add the new change set? That is simple - to the latest one - the one with the highest number. In this example, to changelog-03.xml. The latest file will grow along with the project development. So at some point, it will have a few thousand lines. In my opinion, that is a good moment to create a new file - changelog-04.xml and start adding new change sets there. Just to prevent growing any of these files too much. Of course, I strongly suggest not moving existing change sets from the old files, just leave them there and focus on the new ones. Otherwise, Liquibase may protest during an upgrade, as the DATABASECHANGELOG table contains a file name of each change set.
The second directory - rerun is a special one. It contains change sets that are potentially run more than once, because of runOnChange or runAlways attributes. A common case is stored database code like procedures, functions, and triggers. But you may have other types of changes like reloading some data - translations, dictionaries, etc. Depending on your case, you may have all of that in a single file or spread them across multiple files.
I promised to go back to the changelog-master.xml file. It shouldn't contain any change sets by definition. It just gathers all other change log files together. Here is a sample content, that matches the case I described above.
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xmlns:pro="http://www.liquibase.org/xml/ns/pro"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd
http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd
http://www.liquibase.org/xml/ns/pro http://www.liquibase.org/xml/ns/pro/liquibase-pro-latest.xsd">
<include file="regular/changelog-01.xml" />
<include file="regular/changelog-02.xml" />
<include file="regular/changelog-03.xml" />
<include file="regular/stored-functions-01.xml" />
<include file="regular/stored-procedures-01.xml" />
</databaseChangeLog>
In the beginning, all files from the regular directory are included one by one. It is important to keep the order. Of course, when you create a new regular change log file, you should add a relevant include tag. These regular files are first because they usually create a table structure, which should go before creating stored procedures. After the regular files, change log files related to stored code can go.
It may seem weird, that we add change sets to the changelog-03.xml file, which is in the middle of the master file. But it is all good. Even if you have all stored procedures and functions, you still may add a new table to the database.
I hope you now know why I have the master file - it is because of goal 3. I can run Liquibase upgrade using that master file and it will apply all other change log files in the right order.
This or similar project structures have worked well in my projects so far. I understand that some of you may have slightly different needs, but I still believe that this structure is at least a good starting point.