atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[Minicom.nl]]></title>
  <link href="http://blog.minicom.nl/atom.xml" rel="self"/>
  <link href="http://blog.minicom.nl/"/>
  <updated>2015-04-03T19:51:35+02:00</updated>
  <id>http://blog.minicom.nl/</id>
  <author>
    <name><![CDATA[Michael de Jong]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[Revisiting profiling DDL statements - MySQL's return]]></title>
    <link href="http://blog.minicom.nl/blog/2015/04/03/revisiting-profiling-ddl-statements-mysqls-return/"/>
    <updated>2015-04-03T16:02:59+02:00</updated>
    <id>http://blog.minicom.nl/blog/2015/04/03/revisiting-profiling-ddl-statements-mysqls-return</id>
    <content type="html"><![CDATA[<p>In the previous blogpost I covered the results of a series of experiments I ran to profile DDL statements for PostgreSQL 9.3.6 and MySQL 5.5.41. But people on Twitter rightfully pointed out that I profiled databases which were around 2 years old, and newer versions have been released since. People claimed that in both PostgreSQL 9.4 and MySQL 5.6 there have been some improvements with regard to the blocking nature of DDL statements. So I figured this warrants a re-run of the experiments I ran earlier.</p>

<h1>Setup</h1>

<p>In this post I will be comparing the performance and locking properties of DDL statements of PostgreSQL 9.3.6, PostgreSQL 9.4.1, MySQL 5.5.41, and MySQL 5.6.19. The reason I&rsquo;m re-running the experiments for the older version of PostgreSQL and MySQL is because I no longer have the same server, and am now running these experiments on a Digital Ocean droplet with 8 cores, 16 GB memory, and a 160 GB SSD drive. The biggest difference compared to the previous setup is the SSD drive, so I&rsquo;m expecting better performance from both databases.</p>

<p>In addition Nemesis has been modified to execute slightly different DDL statements for MySQL, which allows you to specify a specific algorithm and/or type of lock.</p>

<h1>Results</h1>

<p>Like in the previous blogpost I&rsquo;m running through several scenarios where the schema of a database is being modified while it is in use. For each scenario you&rsquo;ll find a image, which graphs the properties of the queries Nemesis performed on the database. The x-axis resembles time - each mark on this axis equals a second. The y-axis resembles the duration of a query - 1 pixel equals 1 millisecond (this is only to give a general feel for query performance). Each color represents a different type of query being performed.</p>

<ul>
<li><strong>Green</strong>: INSERT statements</li>
<li><strong>Red</strong>: DELETE statements</li>
<li><strong>Blue</strong>: UPDATE statements</li>
<li><strong>Yellow</strong>: SELECT statements</li>
<li><strong>Black</strong>: DDL statements</li>
</ul>


<p>The grey area resembles the period of time during which the DDL statement is being executed.</p>

<h2>Adding a nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/add-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/add-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/add-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/add-nullable-column.png" alt="" /></p>

<p>We can see that PostgreSQL 9.4 and 9.3 both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and does not seem to show any negative impact on performance afterwards. For MySQL 5.6 we can see that there is a big improvement over MySQL 5.5. Although the operation takes a long time to complete, all DML queries continue to work while the DDL statement is running.</p>

<h2>Adding a non-nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/add-non-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/add-non-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/add-non-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/add-non-nullable-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.4 and 9.3 both perform this operation in a blocking fashion. The DDL statement takes a long time to complete and doesn&rsquo;t allow for other DML queries to complete while this operation is in progress. For MySQL 5.6 we can see that there is another big improvement over MySQL 5.5 (and PostgreSQL). Although the DDL operation still takes a long time to complete, all DML queries continue to work while the DDL statement is running.</p>

<h2>Renaming a nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/rename-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/rename-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/rename-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/rename-nullable-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.4 and 9.3 both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and does not seem to show any negative impact on performance afterwards. For MySQL 5.6 we can see that the table is put in a read-only mode while this operation is in progress. MySQL 5.6 performs a lot better and similar to PostgreSQL. The operation is quick to complete with no obvious performance issues afterwards.</p>

<h2>Renaming a non-nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/rename-non-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/rename-non-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/rename-non-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/rename-non-nullable-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.4 and 9.3 both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and does not seem to show any negative impact on performance afterwards. For MySQL 5.6 we can see that the table is again put in a read-only mode while this operation is in progress. MySQL 5.6 performs a lot better and more similar to PostgreSQL. The operation is quick to complete with no obvious performance issues afterwards.</p>

<h2>Dropping a nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/drop-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/drop-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/drop-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/drop-nullable-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.4 and 9.3 again both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and does not seem to show any negative impact on performance afterwards. For MySQL 5.6 we can see that the table is put in a read-only mode while this operation is in progress. MySQL 5.6 performs a lot better. The operation takes a long time to complete, but doesn&rsquo;t stop other DML queries from being executed.</p>

<h2>Dropping a non-nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/drop-non-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/drop-non-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/drop-non-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/drop-non-nullable-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.4 and 9.3 again both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and does not seem to show any negative impact on performance afterwards. For MySQL 5.6 we can see that the table is put in read-only mode while this operation is in progress. MySQL 5.6 performs a lot better compared to MySQL 5.5. The operation takes a long time to complete, but doesn&rsquo;t stop other DML queries from being executed.</p>

<h2>Creating an index</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/create-index-on-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/create-index-on-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/create-index-on-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/create-index-on-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.4 and 9.3 again both perform this operation in a non-blocking fashion. The DDL statement takes a long time to complete, but doesn&rsquo;t block other DML queries from being executed. For MySQL 5.6 we can see that the table is put in read-only mode while this operation is in progress. MySQL 5.6 performs a lot better. Similar to PostgreSQL, the operation takes a long time to complete, but during this period DML queries can still be executed.</p>

<h2>Renaming an existing index</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/rename-index.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/rename-index.png" alt="" /></p>

<p>This is strange. In PostgreSQL 9.3 this operation quickly completed without any obvious performance issues. In PostgreSQL 9.4, this operation took around 1 second to complete, and blocked other DML queries from executing. Depending on your requirements, I would label this operation as &ldquo;Use at own risk&rdquo; for production environments. MySQL 5.5 doesn&rsquo;t have support for renaming indices. MySQL 5.6 seems to have support, but I couldn&rsquo;t get this to work for the scenario as defined in Nemesis.</p>

<h2>Dropping an index</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/drop-index-on-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/drop-index-on-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/drop-index-on-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/drop-index-on-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.4 and 9.3 again both perform this operation in a non-blocking fashion. The DDL statement takes less than a second to complete, and doesn&rsquo;t block other DML queries from being executed. For MySQL 5.5 we can clearly see that other DML queries are being blocked from executing for just over 2 seconds. MySQL 5.6 performs a lot better. The operation completes quickly, and doesn&rsquo;t show any signs of performance issues.</p>

<h2>Making a column nullable</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/make-column-nullable.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/make-column-nullable.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/make-column-nullable.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/make-column-nullable.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.4 and 9.3 again both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and doesn&rsquo;t show any performance issues. For MySQL 5.5 and MySQL 5.6 we can clearly see that the table is put in a read-only mode for a prolonged period of time.</p>

<h2>Making a column non-nullable</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/make-column-non-nullable.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/make-column-non-nullable.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/make-column-non-nullable.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/make-column-non-nullable.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.3 and 9.4 both perform this operation in a blocking fashion. The DDL statement takes 13 and 25 seconds respectively, and clearly blocks other DML queries from being executed. For MySQL 5.5 and MySQL 5.6 we can clearly see that the table is put in a read-only mode for a prolonged period of time.</p>

<h2>Setting a default value on a nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/set-default-expression-on-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/set-default-expression-on-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/set-default-expression-on-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/set-default-expression-on-nullable-column.png" alt="" /></p>

<p>Hurray! Both versions of both MySQL and PostgreSQL seem to be doing this in a non-blocking fashion. The operation is quick to complete in all cases, with no obvious performance issues.</p>

<h2>Setting a default value on a non-nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/set-default-expression-on-non-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/set-default-expression-on-non-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/set-default-expression-on-non-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/set-default-expression-on-non-nullable-column.png" alt="" /></p>

<p>Another hurray! Both versions of both MySQL and PostgreSQL seem to be doing this in a non-blocking fashion as well. The operation is quick to complete in all cases, with again no obvious performance issues.</p>

<h2>Modifying the type of a nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/modify-data-type-on-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/modify-data-type-on-nullable-column.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/modify-data-type-on-nullable-column.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/modify-data-type-on-nullable-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.3 and 9.4 both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and doesn&rsquo;t show any obvious performance issues. For MySQL 5.5 we can clearly see that the table is put in a read-only mode, blocking DML queries other than SELECT. For MySQL 5.6 we can clearly see that the operation is non-blocking. It&rsquo;s quick to complete without any performance issues.</p>

<h2>Modifying the type of a non-nullable column</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/modify-data-type-on-non-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/modify-data-type-on-non-nullable-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.3 and 9.4 both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and doesn&rsquo;t show any obvious performance issues. For both versions of MySQL this scenario could not be performed, <a href="http://stackoverflow.com/questions/3466872/why-cant-a-text-column-have-a-default-value-in-mysql">as it seems</a> that these versions of MySQL don&rsquo;t support having default values for &ldquo;TEXT&rdquo; type columns, which was needed to perform this scenario.</p>

<h2>Modifying a column&rsquo;s type from integer to text</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/modify-data-type-on-nullable-column.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/modify-data-type-on-nullable-column.png" alt="" /></p>

<p>Here we see that PostgreSQL 9.3 and 9.4 both perform this operation in a non-blocking fashion. The DDL statement is quick to complete, and doesn&rsquo;t show any obvious performance issues. For both versions of MySQL this scenario could not be performed, <a href="http://stackoverflow.com/questions/3466872/why-cant-a-text-column-have-a-default-value-in-mysql">as it seems</a> that these versions of MySQL don&rsquo;t support having default values for &ldquo;TEXT&rdquo; type columns, which was needed to perform this scenario.</p>

<h2>Adding a nullable foreign key</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/add-nullable-foreign-key.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/add-nullable-foreign-key.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/add-nullable-foreign-key.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/add-nullable-foreign-key.png" alt="" /></p>

<p>Ah this is terrible. For both versions of PostgreSQL and MySQL this operation is performed in a blocking fashion. This operation takes a while and blocks other DML queries while it is being executed. Both versions of MySQL do allow for SELECT statements to be executed while this operation is in progress.</p>

<h2>Adding a non-nullable foreign key</h2>

<h3>PostgreSQL 9.3</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.3/add-non-nullable-foreign-key.png" alt="" /></p>

<h3>PostgreSQL 9.4</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/postgresql-9.4/add-non-nullable-foreign-key.png" alt="" /></p>

<h3>MySQL 5.5</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.5/add-non-nullable-foreign-key.png" alt="" /></p>

<h3>MySQL 5.6</h3>

<p><img src="http://blog.minicom.nl/images/ddl-profiling/mysql-5.6/add-non-nullable-foreign-key.png" alt="" /></p>

<p>This scenario is terrible as well. For both versions of PostgreSQL and MySQL this operation is performed in a blocking fashion. This operation takes a while and blocks other DML queries while it is being executed. Both versions of MySQL do allow for SELECT statements to be executed while this operation is in progress.</p>

<h2>Conclusion.</h2>

<p>We&rsquo;ve looked at various scenarios involving several &ldquo;refactorings&rdquo; one can apply to tables in a SQL databases. We&rsquo;ve seen that some operations appear to be safe enough to be used on live production databases whereas other should be avoided. We have seen that there have been significant improvements for MySQL 5.6 compared to MySQL 5.5. I have not observed any improvements for PostgreSQL 9.4 over 9.3. Same as in the previous blogpost, there are some caveats to these experiments:</p>

<ul>
<li>Tuning the configuration of a database may yield better performance which may lower the duration of some of these operations. For these experiments I only assigned more memory to be used by PostgreSQL and MySQL (around ~4GB), and to make better use of the SSD by lowering disk lookup penalties for PostgreSQL. I didn&rsquo;t tune any other aspect of the databases which may have yielded better performance.</li>
<li>The duration of these operations also seems to take significantly longer when the database is starved of resources. Having a slow disk, or a almost fully utilizing the CPU seems to negatively affect the performance of these DDL statements. In this series I used an SSD drive, which significantly sped up all queries done by Nemesis on both MySQL and PostgreSQL.</li>
<li>These operations may show different blocking or non-blocking behavior in different circumstances. Applying these operations to table with a different structure, constraints, or references may yield different results.</li>
</ul>


<p>In other words: our measurements apply only to this particular setup in which we have tested these scenarios. If you wish to find out if a certain operation can be safely performed in your production environment, you would do well to test it on a staging environment which replicates the production environment (and it&rsquo;s workload) as best it can.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Profiling DDL statements]]></title>
    <link href="http://blog.minicom.nl/blog/2015/03/08/profiling-ddl-statements/"/>
    <updated>2015-03-08T14:37:05+01:00</updated>
    <id>http://blog.minicom.nl/blog/2015/03/08/profiling-ddl-statements</id>
    <content type="html"><![CDATA[<p>It has been a while since my last post. This is due to writing and publishing a technical paper for the Release Engineering conference (RELENG 2015), and traveling during a one month honeymoon.</p>

<p>Last September/November I visited the CHISEL group at UVIC in Victoria, Canada. With borrowed hardware, I profiled DDL statements in both PostgreSQL 9.3.6 and MySQL 5.5.41. DDL statements are statements which alter the structure of tables and relations in SQL databases. I did this with the intention to find out which DDL statements block other DML queries (SELECT, UPDATE, INSERT, and DELETE) from running, and in what kind of way.</p>

<h1>Method</h1>

<p>To do this I created a small profiler tool named <a href="http://github.com/quantumdb/nemesis">Nemesis</a>. Nemesis works as followed:</p>

<ol>
<li>Nemesis connects to either a MySQL or PostgreSQL database, and prepares an initial database containing a &ldquo;users&rdquo; table with 50 million records.</li>
<li>Nemesis then starts a user-defined amount of worker threads which continuously perform SELECT, UPDATE, INSERT, and DELETE statements on the &ldquo;users&rdquo; table to simulate a software application reading from and writing to the database. The start time, end time, and duration for each query is recorded.</li>
<li>After some time a migration scenario is executed which transforms the structure of the &ldquo;users&rdquo; table in some way. We also record the start time, end time and duration for these operations (typically one DDL statement).</li>
<li>We drop the database.</li>
<li>Repeat steps 1-4 for every defined migration scenario. But note that some scenarios cannot be performed in both MySQL and PostgreSQL, due to missing features in that database.</li>
</ol>


<h1>Setup</h1>

<p>For this series of experiments I gratefully used a server with a quad-core Intel Core i7-3770, a 1TB hard disk, and 16GB memory on loan from the CHISEL lab at UVIC. On this server I installed both PostgreSQL 9.3.6 and MySQL 5.5.41 on a Ubuntu 14.04 LTS installation. Both databases were slightly tweaked to use more of the available memory than the default settings. For those who are interested, MySQL used the InnoDB engine by default.</p>

<p>The results below have been taken from a series of experiments run both on MySQL as well as on PostgreSQL. In both cases, all scenarios were run with 6 worker threads in total:</p>

<ul>
<li>2 workers for SELECT statements.</li>
<li>2 workers for UPDATE statements.</li>
<li>1 worker for INSERT statements.</li>
<li>1 worker for DELETE statements.</li>
</ul>


<h1>Results</h1>

<p>The results are interesting as they differ significantly between MySQL and PostgreSQL. Below I&rsquo;ve posted several migration scenarios one might perform on an existing table. For each scenario you&rsquo;ll find a image, which graphs the properties of the queries Nemesis performed on the database. The x-axis resembles time - each mark on this axis equals a second. The y-axis resembles the duration of a query - 1 pixel equals 1 millisecond (this is only to give a general feel for query performance). Each color represents a different type of query being performed.</p>

<ul>
<li><strong>Green</strong>: INSERT statements</li>
<li><strong>Red</strong>: DELETE statements</li>
<li><strong>Blue</strong>: UPDATE statements</li>
<li><strong>Yellow</strong>: SELECT statements</li>
<li><strong>Black</strong>: DDL statements</li>
</ul>


<p>The grey area resembles the period of time during which the DDL statement is being executed.</p>

<h2>Adding a nullable column</h2>

<p>In this scenario we add a new column to the existing &ldquo;users&rdquo; table which may contain NULL values.</p>

<h3>PostgreSQL</h3>

<p>PostgreSQL seems to be doing this very well. We can see only a very small grey area, which means that this operation completed quickly. There seems to been no obvious impact on the performance of the other queries. This is good news if you need to perform this operation on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-add-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>MySQL shows very different behavior for this scenario. Unlike PostgreSQL, MySQL takes considerably longer to perform this operation (in fact I terminated this operation after 60 seconds), as can be seen by the very wide grey area. Interestingly enough though, MySQL seems to allow SELECT statements to continue to work (since we can see the yellow lines in the grey area) but blocks all other types of DML queries. In essence, during this operation the table seems to be put in a sort of read-only state.</p>

<p><img src="http://blog.minicom.nl/images/mysql-add-nullable-column.png" alt="" /></p>

<h2>Adding a non-nullable column</h2>

<p>In this scenario we perform almost the exact same operation as in the previous scenario with the minor exception that the new column may not store any NULL values.</p>

<h3>PostgreSQL</h3>

<p>PostgreSQL stumbles quite dramatically here. As opposed to the previous scenario where the DDL operation completed quickly, in this scenario it takes a lot longer as indicated by the grey area. Even worse is that during this period, none of the other queries are able to continue. It seems that this operation is not suitable to be executed on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-add-non-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Here we see that MySQL performs similar to PostgreSQL. This operation was also halted after 60 seconds, but throughout this period SELECT statements were still being executed while the other DML queries were blocked as indicated by the yellow lines inside of the grey area. It seems that during this period the table is again in some sort of read-only mode.</p>

<p><img src="http://blog.minicom.nl/images/mysql-add-non-nullable-column.png" alt="" /></p>

<h2>Renaming a nullable column</h2>

<p>This scenario is all about renaming an existing column in the &ldquo;users&rdquo; table. In this particular scenario the column is nullable - allowed to store NULL values.</p>

<h3>PostgreSQL</h3>

<p>This is another operation which PostgreSQL seems to be quite comfortable with. There is only a very narrow grey area visible, which means that the DDL statement completes quickly. From the other plotted queries we can derive that there is no obvious performance impact once this operation has completed. It seems that this operation is therefore also suitable to be performed on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-rename-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Compared to PostgreSQL, MySQL seems to take considerably longer to perform this operation as indicated by the very wide grey area. In fact I terminated this operation after 60 seconds. Similar to previous scenarios, MySQL seems to operate in some sort of read-only state while this operation is performed.</p>

<p><img src="http://blog.minicom.nl/images/mysql-rename-nullable-column.png" alt="" /></p>

<h2>Renaming a non-nullable column</h2>

<p>Much like the previous scenario, this scenario renames an existing column. Only in this scenario the column is non-nullable - does not allow NULL values to be stored.</p>

<h3>PostgreSQL</h3>

<p>Unlike the scenario where adding a new column had different impact depending on whether the column was nullable or non-nullable, in this scenario PostgreSQL seems to perform similar to the rename nullable column scenario. There are some very small disruptions in the performance of INSERT statements after completing the DDL statement, but this seems acceptable. It seems that this operation could also be performed on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-rename-non-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Again, MySQL seems to perform worse than PostgreSQL. Where PostgreSQL completes this operation quickly, MySQL took a lot longer. This operation was also terminated after 60 seconds. In addition we seen the same read-only mode we&rsquo;ve seen before with other scenarios for MySQL.</p>

<p><img src="http://blog.minicom.nl/images/mysql-rename-non-nullable-column.png" alt="" /></p>

<h2>Dropping a nullable column</h2>

<p>In this scenario we&rsquo;re dropping an existing nullable column.</p>

<h3>PostgreSQL</h3>

<p>PostgreSQL doesn&rsquo;t seem to have any problems dropping an existing nullable column from a table. There&rsquo;s a very thin grey area, meaning that the DDL operation completes quickly. There also doesn&rsquo;t seem to be any significant disturbances in the performance of the other queries.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-drop-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Again, MySQL seems to perform worse than PostgreSQL. I terminated the operation after 60 seconds, but again throughout this period, the table seems to be in some sort of read-only mode.</p>

<p><img src="http://blog.minicom.nl/images/mysql-drop-nullable-column.png" alt="" /></p>

<h2>Dropping a non-nullable column</h2>

<p>Similar to the previous scenario, this scenario involves dropping a column. In this case a non-nullable column is dropped.</p>

<h3>PostgreSQL</h3>

<p>Although PostgreSQL again completes the DDL statement very quickly as can be seen by the short black line (the operation is so quick the grey area doesn&rsquo;t even show in the graph). However as can be seen, several seconds later the performance of the other DML queries is severely impacted. Some queries block for several seconds, which would result in severely degraded performance for a real application. Whether this is actually due to the DDL statement is hard to tell, but it might be due to the limiting factor of his server: IOPS on the slow hard disk. Perhaps PostgreSQL&rsquo;s VACUUM process is running there, or we&rsquo;ve run into some kind of background task which decimates the server&rsquo;s performance.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-drop-non-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Again we see the same performance we&rsquo;ve seen from MySQL in previous scenarios. The operation was again terminated after 60 seconds, while the table remained in a read-only mode during this period.</p>

<p><img src="http://blog.minicom.nl/images/mysql-drop-non-nullable-column.png" alt="" /></p>

<h2>Creating an index</h2>

<p>In this scenario we create a new index on a column in the existing &ldquo;users&rdquo; table.</p>

<h3>PostgreSQL</h3>

<p>PostgreSQL claims that it can create indices in a non-blocking fashion by using the &ldquo;CONCURRENTLY&rdquo; keyword in the DDL statement. As we can see in the corresponding graph the gray area is very large and extends beyond the end of the graph (in fact I terminated this operation after 60 seconds). However, we can clearly see that the other queries continue to perform without disruption or obvious performance issues. It seems that this can be safely used on live production databases.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-create-index-on-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Unlike PostgreSQL, MySQL does not have any additional syntax to specify that an index should be created concurrently, or in a non-blocking fashion. As can be seen in the graph, MySQL performs very similar to previous scenarios. The DDL statement takes a while (and I terminated it again after 60 seconds), while during this period the table is put in a read-only mode.</p>

<p><img src="http://blog.minicom.nl/images/mysql-create-index-on-column.png" alt="" /></p>

<h2>Renaming an existing index</h2>

<p>In this scenario we rename an existing index on the &ldquo;users&rdquo; table.</p>

<h3>PostgreSQL</h3>

<p>PostgreSQL also seems to be able to rename an index without problems during active use. No grey area is visible, and the black line for the DDL statement is very short indicating that this operation completed quickly. Since the graph doesn&rsquo;t reveal any performance issues or disruptions we can assume that this operation is also safe to use on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-rename-index.png" alt="" /></p>

<h3>MySQL</h3>

<p>MySQL doesn&rsquo;t seem to have any support for renaming an existing index. However, it seems that support for this <a href="http://dev.mysql.com/doc/refman/5.7/en/mysql-nutshell.html">may be added in MySQL 5.7</a>.</p>

<h2>Dropping an index</h2>

<p>In this scenario we drop a previously created index on the &ldquo;users&rdquo; table.</p>

<h3>PostgreSQL</h3>

<p>Like the &ldquo;Create index&rdquo; scenario PostgreSQL also offers a &ldquo;CONCURRENTLY&rdquo; keyword in the DDL statement for dropping an index. However as we can see from the graph, the performance of DML statements is severely reduced and disrupted while the index is being dropped. It would seem that this operation is not safe to use on live production databases.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-drop-index-on-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Surprise! MySQL seems to be able to drop an existing index relatively quickly (roughly 1 second in our case). It&rsquo;s too hard to determine if during this period the table is put in a read-only mode, but since this operation completed so quickly, it may be perfectly safe to use on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/mysql-drop-index-on-column.png" alt="" /></p>

<h2>Making a column nullable</h2>

<p>This scenario concerns making an already existing non-nullable column in the &ldquo;users&rdquo; table nullable. This effectively allows NULL values to be stored in this column when this operation has been performed.</p>

<h3>PostgreSQL</h3>

<p>Again PostgreSQL seems to have no trouble performing this operation. The grey area is very narrow, meaning that the DDL operation completed quickly. In addition no performance issues or disruptions can be derived from the graph. It seems that this operation is also safe to use on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-make-column-nullable.png" alt="" /></p>

<h3>MySQL</h3>

<p>Unfortunately we see the same behavior from MySQL here as we&rsquo;ve seen in most scenarios. The operation was terminated after 60 seconds, but throughout this period the table did not block SELECT queries. Other DML queries were blocked however.</p>

<p><img src="http://blog.minicom.nl/images/mysql-make-column-nullable.png" alt="" /></p>

<h2>Making a column non-nullable</h2>

<p>This scenario does the exact opposite of the previous scenario. It makes an already existing nullable column non-nullable. This effectively prohibits NULL values from being stored in this column when the operation has completed.</p>

<h3>PostgreSQL</h3>

<p>Here PostgreSQL clearly stumbles again. The DDL operation which makes the column non-nullable takes just over 3 seconds, but as we can see in the graph during this time none of the other queries are able to complete. Depending on your requirements, this operation may not be safe enough to be used on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-make-column-non-nullable.png" alt="" /></p>

<h3>MySQL</h3>

<p>Again, MySQL performs worse compared to PostgreSQL. The operation was terminated after 60 seconds again, but during this period the table was again put in a read-only mode.</p>

<p><img src="http://blog.minicom.nl/images/mysql-make-column-non-nullable.png" alt="" /></p>

<h2>Setting a default value on a nullable column</h2>

<p>This scenario is about setting or changing the default value of an already existing nullable column.</p>

<h3>PostgreSQL</h3>

<p>As we can see from the graph, PostgreSQL has no problems performing this operation. The very narrow grey area means that the DDL operation completes very quickly. The graph doesn&rsquo;t reveal any performance issues or disruptions for the other DML queries. It appears that this operation is also safe to use on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-set-default-expression-on-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Here it appears that MySQL performs similar to PostgreSQL for this scenario. We can see a very narrow grey area which means that the DDL operation is completed quickly. It would appear that this operation is safe to use on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/mysql-set-default-expression-on-nullable-column.png" alt="" /></p>

<h2>Setting a default value on a non-nullable column</h2>

<p>This scenario is about setting or changing the default value of an already existing non-nullable column.</p>

<h3>PostgreSQL</h3>

<p>This graph is very similar to the graph of the previous scenario for PostgreSQL. The operations seems to take slightly longer since the grey area is slightly wider, but the graph doesn&rsquo;t reveal any performance issues or disruptions for the other DML queries. So it appears that this operation (like the previous) is also safe to use on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-set-default-expression-on-non-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Compared to the previous scenario the performance is slightly worse. MySQL seems to take slightly longer to complete the DDL statement (still less than a second). It&rsquo;s too hard to determine if the other DML queries are blocked during this period. It would seem that this operation is also safe to use on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/mysql-set-default-expression-on-non-nullable-column.png" alt="" /></p>

<h2>Modifying the type of a nullable column</h2>

<p>This scenario changes the type of an already existing nullable column in the &ldquo;users&rdquo; table from a &ldquo;varchar(255)&rdquo; type to a &ldquo;text&rdquo; type.</p>

<h3>PostgreSQL</h3>

<p>PostgreSQL seems to handle this well. Like many other operations, the grey area is very narrow which means that the DDL statement completes quickly. In addition, the graph doesn&rsquo;t seem to show signs of any obvious performance issues or disruptions. It seems that this operation can be safely performed on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-modify-data-type-on-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>Unfortunately MySQL yet again performs a lot worse than PostgreSQL. We can clearly see that this operation takes a long time to complete (and I again had to terminate it after 60 seconds). throughout this period the table was again put in a read-only mode.</p>

<p><img src="http://blog.minicom.nl/images/mysql-modify-data-type-on-nullable-column.png" alt="" /></p>

<h2>Modifying the type of a non-nullable column</h2>

<p>Like the previous scenario, this scenario attempts to modify the type of an already existing column from a &ldquo;varchar(255)&rdquo; type to a &ldquo;text&rdquo; type. In this instance the column is non-nullable, which means it doesn&rsquo;t allow for NULL values to be stored.</p>

<h3>PostgreSQL</h3>

<p>Since the graph is essentially the same compared to the previous scenario, we can draw the same conclusion: This operation can safely be performed on a live production database.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-modify-data-type-on-non-nullable-column.png" alt="" /></p>

<h3>MySQL</h3>

<p>This scenario could not be performed, <a href="http://stackoverflow.com/questions/3466872/why-cant-a-text-column-have-a-default-value-in-mysql">as it seems</a> that the used version of MySQL doesn&rsquo;t support having default values for &ldquo;TEXT&rdquo; type columns, which was needed to perform this scenario.</p>

<h2>Modifying a column&rsquo;s type from integer to text</h2>

<p>In this scenario we modify the type of an already existing non-nullable column from a &ldquo;bigint&rdquo; type to a &ldquo;text&rdquo; type. Assuming there&rsquo;s some more elaborate kind of conversion required than in the previous two scenarios, we should expect some sort of disruption or performance degradation for the DML queries.</p>

<h3>PostgreSQL</h3>

<p>The graph reveals that PostgreSQL is indeed having trouble with this scenario. The DDL operation does not complete very quickly (in fact I terminated it after 60 seconds). throughout this period we can see that all other DML queries have been blocked from executing, resulting in an unusable database while this operation is in progress. This operation is obviously not safe to be executed on a live production database. In addition, it seems that the behavior of the DDL operation may be blocking or non blocking depending on from, and to which type the column is being transformed.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-modify-data-type-from-int-to-text.png" alt="" /></p>

<h3>MySQL</h3>

<p>This scenario could not be performed, <a href="http://stackoverflow.com/questions/3466872/why-cant-a-text-column-have-a-default-value-in-mysql">as it seems</a> that the used version of MySQL doesn&rsquo;t support having default values for &ldquo;TEXT&rdquo; type columns, which was needed to perform this scenario.</p>

<h2>Adding a nullable foreign key</h2>

<p>This scenario adds a new foreign key to an already existing nullable column in the &ldquo;users&rdquo; table. This foreign key references records in a new &ldquo;addresses&rdquo; table which stores fictional addresses for our users.</p>

<h3>PostgreSQL</h3>

<p>This operation seems to take PostgreSQL just under 4 seconds to complete. During this period the other DML queries are blocked from executing. However after this brief period the disruption disappears, and a similar performance returns. Depending on the requirements of the application, this 4 second period may be acceptable, or it may not.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-add-nullable-foreign-key.png" alt="" /></p>

<h3>MySQL</h3>

<p>Again MySQL takes a while to complete this operation. It was again terminated after 60 seconds, but throughout this period allowed SELECT queries to be processed, while blocking other DML queries.</p>

<p><img src="http://blog.minicom.nl/images/mysql-add-nullable-foreign-key.png" alt="" /></p>

<h2>Adding a non-nullable foreign key</h2>

<p>Finally we take a look at a scenario which closely resembles the previous scenario. The only difference in this scenario is that the foreign key constraint is applied to a non-nullable column. Since the column is pre-filled with references to the &ldquo;addresses&rdquo; table we suspect that the validation phase of creating this constraint will take longer, as it needs to verify that all the current values stored in this column really do reference a value as is defined in the referenced table.</p>

<h3>PostgreSQL</h3>

<p>If we look more closely at the graph for PostgreSQL we can see that it indeed takes longer to perform this operation (just over 10 seconds). Similar to the previous scenario, during this period the other DML queries are clearly blocked, but performance of these queries is restored when the operation completes. This operation obviously blocks, and applying it on a live production database may be too costly.</p>

<p><img src="http://blog.minicom.nl/images/postgresql-add-non-nullable-foreign-key.png" alt="" /></p>

<h3>MySQL</h3>

<p>We see similar performance here compared to the previous scenario. The operation was again terminated after 60 seconds, and appears to put the table in a read-only mode for the duration of the operation.</p>

<p><img src="http://blog.minicom.nl/images/mysql-add-non-nullable-foreign-key.png" alt="" /></p>

<h1>Conclusion</h1>

<p>We&rsquo;ve looked at various scenarios involving several &ldquo;refactorings&rdquo; one can apply to tables in a SQL databases. We&rsquo;ve seen that some operations appear to be safe enough to be used on live production databases. There are however some caveats to these experiments:</p>

<ul>
<li>Tuning the configuration of a database may yield better performance which may lower the duration of some of these operations. For these experiments I only assigned more memory to be used by PostgreSQL and MySQL. I didn&rsquo;t tune any other aspect of the databases which may have yielded better performance.</li>
<li>The duration of these operations also seems to take significantly longer when the database is starved of resources. Having a slow disk, or a almost fully utilizing the CPU seems to negatively affect the performance of these DDL statements.</li>
<li>These operations may show different blocking or non-blocking behavior in different circumstances. Applying these operations to table with a different structure, constraints, or references may yield different results.</li>
</ul>


<p>In other words: our measurements apply only to this particular setup in which we have tested these scenarios. If you wish to find out if a certain operation can be safely performed in your production environment, you would do well to test it on a staging environment which replicates the production environment (and it&rsquo;s workload) as best it can.</p>

<p>Most interesting I found to be the big differences between MySQL and PostgreSQL. MySQL seems to put tables in a read-only mode when performing DDL statements on tables. Unfortunately MySQL takes considerably longer to perform these operations when compared to PostgreSQL.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Dealing with mixed-state in SQL databases]]></title>
    <link href="http://blog.minicom.nl/blog/2014/11/09/dealing-with-mixed-state-in-sql-databases/"/>
    <updated>2014-11-09T01:45:23+01:00</updated>
    <id>http://blog.minicom.nl/blog/2014/11/09/dealing-with-mixed-state-in-sql-databases</id>
    <content type="html"><![CDATA[<p>In the previous blog post I gave a very brief introduction to the concept of Continuous Deployment and how you can achieve it by using a load balancer. In this post I&rsquo;ll be digging deeper into Continuous Deployment but mainly focussing on one of the biggest problems implementors of Continuous Deployment face: storing and retrieving data stored in databases. In particular SQL databases.</p>

<p>It&rsquo;s a given that software evolves over time for various reasons: bug fixes, changing requirements, reducing technical debt, or various other reasons. As a consequence, that means that the way software stores and retrieves persistent data, also changes over time. Many (web)-applications use relational databases to store persistent data. In particular SQL databases like MySQL, PostgreSQL, MSSQL or Oracle are used often. The SQL language and concepts themselves were introduced in the 1980s, and have remained largely unchanged. Despite the growing popularity of NoSQL alternatives, they still remain to be extensively used in most web applications in some capacity.</p>

<h2>A more accurate problem description</h2>

<p>Today&rsquo;s relational SQL databases actually have two problems when used in a Continuous Deployment setting. The first and most obvious problem is dealing with the blocking nature of some operations in SQL databases, making schema evolution in SQL databases prohibitive in live production environments. The second and less obvious problem is dealing with mixed-state. This is a problem NoSQL databases also face, but is largely ignored in both areas.</p>

<h3>Dealing with the blocking nature of DDL statements</h3>

<p>DDL statements are statements which describe an action on the schema of a database. Examples would be creating a table, or dropping a column, or adding a foreign key. In SQL databases these operations are often regarded as statements used during maintenance, and often assume that the database is not in active use while they are performed. As a result, database developers have implemented them in such a way that they often require a write lock on the entire table and thus don&rsquo;t play very well with other DML queries. In addition they are typically slow and disruptive under stressed circumstances. If you want your software to be continuously deployable, you will need to find a way around these blocking operations. This greatly depends on the SQL database you are using and your use case (nature of your application, method of application deployment, etc).</p>

<h3>Dealing with mixed-state</h3>

<p>As explained in the previous blog post, when you are in the process of evolving a schema from one version to the next there is a period of time where both versions of the schema are active. This period is more commonly referred to as the mixed-state. This mixed-state is due to the fact that even though the switchover from the old to the new version of the web application might be atomical, there is a period where both versions of the web application are active at the same time, although one is not yet handling incoming user requests. Both require a different schema to operate on, and can/should not operate on each other&rsquo;s schema.</p>

<h2>Current state of schema evolution in practice</h2>

<p>If you wish to use relational databases in a production environment with Continuous Deployment there are currently two practical ways of doing this. The first is maintain everything manually, letting your application deal with the mixed-state and use a technique called <code>expand-contract</code> to evolve the database schema. The alternative is using a tool like the Percona toolkit, which will perform this expand-contract technique in an automated fashion, and performs an atomic table rename to switch from one schema to the other. <strong>Let&rsquo;s examine why neither option is favorable</strong>.</p>

<h3>Manual evolution</h3>

<p>The first option is to handle the evolution and migration manually. Let&rsquo;s imagine we have a web application running in production on a certain database schema and our application is deployed using rolling upgrades. The SQL database currently contains a <code>users</code> table which holds information on every user that has ever registered with the web application. To be able to suspend the accounts of their users in some cases, the software engineers have decided that they want to add a new column called <code>suspended</code> to the <code>users</code> table, signifying if an account has been suspended or is still active. However simply executing a query which adds this column to the existing <code>users</code> table would require a write lock on the entire table, and block other queries from retrieving data or updating data in this table. Since this web application hosts many millions of users, and the <code>users</code> table itself if constantly used, this is deemed too costly since it would require some downtime.</p>

<p><img src="http://blog.minicom.nl/images/manual-migration-path-step0.png" alt="" /></p>

<p>Instead the software engineers create a new table called <code>users_v2</code> with the same columns that the original <code>users</code> table contains plus the new <code>suspended</code> column. Next they deploy a new version of the application which will read and write user information from and to the <code>users</code> table, but also write these records to the new <code>users_v2</code> table whenever a change is made to one of the rows in the <code>users</code> table. In addition to this, they start a background task which copies over all entries in the <code>users</code> table to the new <code>users_v2</code> table, to ensure that all users are copied within a finite amount of time.</p>

<p><img src="http://blog.minicom.nl/images/manual-migration-path-step1.png" alt="" /></p>

<p>Finally when the two tables contain an equal amount of records, the migration has achieved the mixed-state successfully, and a new version of the application can be deployed which will read and write from the new <code>users_v2</code> table, but also update the records in the <code>users</code> table if a records is changed in the <code>users_v2</code> table. This has to be done since we&rsquo;re using the rolling upgrades method to deploy our application. During the deployment of the new application two different versions are active and might write modifications to both the old and the new <code>users</code> table.</p>

<p><img src="http://blog.minicom.nl/images/manual-migration-path-step2.png" alt="" /></p>

<p>Once this deployment has completed another version of the application can be deployed which will only read and write from the new <code>users_v2</code> table. Once this deployment has succesfully completed, the original <code>users</code> table may be deleted.</p>

<p><img src="http://blog.minicom.nl/images/manual-migration-path-step3.png" alt="" /></p>

<p>Please note that it is also possible to use triggers in many SQL databases to keep these two tables synchronized. Whenever a record is changed in a table that has a trigger added to it, it may run additional queries or snippets of code. If you add a trigger which upon changes in data in the <code>users</code> table, writes these changes to the <code>users_v2</code>, it is possible to keep these these two tables synchronized.</p>

<h4>Evaluation</h4>

<p>This might seem like a logical way to evolve the schema of the database under these strict conditions, but there are several issues with this method:</p>

<ul>
<li>Deploying a change to the schema of the database requires several (re)deployments of your web application, tightly tying schema changes to your application development. These in-between versions of the application are specifically aimed at dealing with mixed-state, and need to synchronize data between two tables. It would be a lot easier if you (as a developer) would not have to deal with this state.</li>
<li>This method makes the application responsible for dealing with mixed-state situations. This means that the database layer in your application has to deal with this added complexity. Although not impossible it increases the complexity of the database layer.</li>
<li>Probably the most useful feature in relational databases is the <strong>foreign key constraint</strong>. How do you deal with foreign key constraints in this method? A table like the <code>users</code> table is often referred to by other tables in web applications, so duplicating and replacing this table using this method while maintaining referential integrety through foreign key constraints is quite a challenge if not impossible.</li>
<li>How do you roll back your changes to the application and the schema of the database in case of problems? Perhaps it is no longer possible to deploy older versions of your application.</li>
</ul>


<p>So there are quite some issues to think about before employing this method for evolving the schema of your database. Luckily there are some automated variations of this approach.</p>

<h3>Automated evolution</h3>

<p>So manually performing evolution of schema evolution is quite complex, and has a number of caveats. In this section I&rsquo;ll cover some of the tools that automate the same process and which address some of these issues.</p>

<h4><a href="https://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/html/introduction.html">OpenArk Kit</a> &amp; <a href="http://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html">Percona Toolkit</a></h4>

<p>The first two tools are the OpenArk Kit and the Percona Toolkit. These toolkits consists of several utility command-line tools which handle everyday maintenance tasks for MySQL databases. Amongst these tools, there are two tools called &ldquo;oak-online-alter-table&rdquo; (OpenArk Kit) and &ldquo;pt-online-schema-change&rdquo; (Percona Toolkit), which aim to make ALTER TABLE operations on tables in MySQL databases non-blocking. They both do this by creating <strong>ghost-tables</strong> (also known as mirror-tables or shadow-tables), which is more or less the same process as we&rsquo;ve seen in the <em>Manual evolution</em> section.</p>

<p>These tools create an empty ghost-table with the new structure based on the original table. They then install triggers on the original table which ensures that whenever a row is changed in the original table, that is reflected in the ghost-table. A background task then fills the ghost-table up with data originating from the original table. Once this process has completed, the names of the original and the ghost-table are swapped in an atomic table rename operation (either manually or automated), and the trigger is removed. This effectively completes the evolution.</p>

<p>If you study the documentation of these tools you&rsquo;ll find that they do not support foreign keys very well. For instance OpenArk Kit, does not work when the table that is being modified has foreign keys defined on it. Percona handles things slightly better, but is also not sufficient. For instance the Percona documentation states the following about foreign keys:</p>

<blockquote><p>Foreign keys complicate the tool&rsquo;s operation and introduce additional risk. The technique of atomically renaming the original and new tables does not work when foreign keys refer to the table.</p></blockquote>

<p>In order to deal with this situation, it offers the following options:</p>

<ul>
<li><strong>rebuild_constraints</strong>, which basically means that the foreign keys are dropped and re-recreated (a process which itself can take very long and as you&rsquo;ll see in the next blog post is also blocking).</li>
<li><strong>drop_swap</strong>, this method first disables foreign key checks for the entire database. This means that referential integrity is no longer enforced, and will allow the database to become inconsistent/invalid. Then the original table is dropped (a process which cannot be rolled back), and then the new table is renamed to the original table&rsquo;s name effectively taking its place. At this point the foreign keys check can be re-enabled.</li>
<li><strong>none</strong>, which does the same as <strong>drop_swap</strong> but does not rename the new table to the original table&rsquo;s name.</li>
</ul>


<h4><a href="https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932">Facebook&rsquo;s Online Schema Change</a></h4>

<p>Facebook being Facebook, has created its own solution to this problem. <strong>Online Schema Change</strong> (OSC) is a tool for MySQL databases which does things slightly different. OSC creates an exact copy of the original table (both structure and data). In the mean time OSC captures all changes that happen to rows stored in the original table using triggers. These triggers store the modifications in a <strong>deltas</strong> table. Once the copy has been created OSC applies the operation on the ghost-table that was originally intented to be execute on the original table (for example adding a column). Since these operations often block this step may take a while. Once this operation has completed, OSC then starts replaying all the changes that happened to the original table from the deltas table onto the modified ghost-table. The final step is the &ldquo;cut-over&rdquo; step which atomically swaps the table names for both the original table and the ghost-table, when the <strong>deltas</strong> table has (almost) been exhausted.</p>

<p>However, as it turns out, OSC also does not support foreign keys. One of the limitations of OSC (see link above) is that there should no foreign keys present on the table to migrate.</p>

<h4><a href="https://github.com/freels/table_migrator">Table Migrator</a></h4>

<p>Table Migrator is another attempt at solving this problem for MySQL databases. Like the previous approaches it creates a ghost-table and applies the changes to this table that should originally be applied to the original table. However it does not copy data over using triggers, but a background process which copies batches of rows from the original table to the ghost table. To do this correctly, it requires a column called updated_at (if the row is mutable) or created_at (if the row is immutable), which stores a timestamp when the row was modified or created. By comparing the timestamps of the original table and the ghost-table it can determine which records are outdated in the ghost table and must be updated using data from the original table. It does this in multiple passes over all records in the original table, until only a small number of rows is still outdated. At this point it acquires a write lock on the table (blocking other queries from using that table), copies over the last remaining outdated rows, and swaps the table names for both tables before releasing the write lock again.</p>

<p>This approach does not use triggers like the other solutions use, but also does not support foreign keys. In addition it seems to require some additional instructions when rows can be deleted from the original table during migration in order to stay consistent. Interestingly enough this tool does attempt to integrate with ActiveRecord, which perhaps allows for some better version management of database schema changes. The other tools covered so far are command-line tools, which have no support for versioning database schemas.</p>

<h4><a href="https://developers.soundcloud.com/blog/soundcloud-mobile-%E2%80%93-proxies.html">SoundCloud&rsquo;s Large Hadron Migrator</a></h4>

<p>The last solution I&rsquo;ll cover in this blog post is SoundCloud&rsquo;s Large Hadron Migrator (LHM). LHM is also aimed at evolving schema of MySQL databases. LHM creates a journal table which will record modifications to the original table (using triggers) much like OSC created and populated the deltas table. In addition LHM creates the ghost-table like all the other solutions have done, and applies its structural changes to this table instead of the original table. LHM will then proceed to copy data from the original table to the new ghost-table (up to data that was present at the start of the migration). Once this process has completed the table names are atomically switched, and the journal entries are replayed.</p>

<p>It&rsquo;s worth noting that similar to previously covered solutions, LHM also does not support foreign keys.</p>

<h2>Conclusion</h2>

<ul>
<li>I&rsquo;ve shown how manual schema evolution is no easy feat and introduced substantial complexity to the development of the web application.</li>
<li>Foreign keys are pretty much not supported, significantly reducing the value of these tools and approaches.</li>
<li>Versioning the evolution of the schema is difficult, and typically an afterthought.</li>
<li>Evolution of schema is limited because only small steps can be undertaken. This hinders bigger refactorings of code when they require significant changes in the existing database schema.</li>
<li>The tools covered in this blog post offer no solution for the mixed-state problem. They only aim at solving the problem of blocking DDL statements in SQL databases.</li>
</ul>


<h2>Next up</h2>

<p>In the next blog post I&rsquo;ll be covering some of my result of profiling DDL statements in both MySQL and PostgreSQL. Do these operations really show blocking behaviour? How so? How long? And how could this help me with evolving database schemas without causing downtime for its users.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Introduction to Continuous Deployment]]></title>
    <link href="http://blog.minicom.nl/blog/2014/09/19/an-introduction-to-continuous-deployment/"/>
    <updated>2014-09-19T20:15:00+02:00</updated>
    <id>http://blog.minicom.nl/blog/2014/09/19/an-introduction-to-continuous-deployment</id>
    <content type="html"><![CDATA[<p>We rely increasingly more on web services in our lives. We cannot think of our world without the web services of Google, Facebook, Twitter and many others. But as we rely more and more on these services, we are less accepting when such a service is temporarily offline - regardless of whether this was due to a failure or maintenance. This puts additional strain on teams which are responsible for improving and operating these services. Especially when a web service has to be updated to a newer version while still remaining online and available to its users. In this text we&rsquo;ll explore some of the ways to achieve Continuous Deployment and the challenges that come with it.</p>

<h2>What is Continuous Deployment?</h2>

<p>In recent years development teams have been transitioning to alternative methods of development. Moving towards more Agile methodologies and concepts they are now able to produce production-ready deliverables on a continuous basis. These deliverables are automatically - and continuously - tested using an automated unit- and integration test suite. By being able to produce these production-ready deliverables in quick succession, these development teams are able to quickly adapt to customer requirements and are quick to deal with bugs. But being able to produce these deliverables in a continuous pace is not enough to fulfill customer requirements. These deliverables must also be deployed into production for customers to benefit from these improvements. This is where Continuous Deployment comes into view.</p>

<p>The goal of Continuous Deployment is to ensure that (almost) every production-ready deliverable is automatically deployed to all production servers, thereby replacing the older version. There are various tools, techniques and strategies to achieve this, each with their own advantages and disadvantages. In this post I will focus only on this upgrade process.</p>

<h2>The Art of Load balancing</h2>

<p>Deploying a new version of a web service without causing downtime is typically done using some form of load balancing. A load balancer is a piece of software which sits between the various servers running the web service - called a server pool - and the connection to the internet. A load balancer is able to redirect incoming user requests to one particular server in the server pool based on a set of rules. By changing these rules, one is able to remove a server from the server pool - effectively not letting that server process anymore user requests, or adding a server to the server pool - which will cause some of the traffic to be redirected to that server. There are two strategies available using a load balancer which can help teams deploy new versions of their web service without the need to go offline. These are known as &ldquo;rolling upgrades&rdquo; and &ldquo;big-flips&rdquo; (also known as atomic switches).</p>

<h3>Rolling upgrades</h3>

<p>The typical setup for rolling upgrades is one (or more) load balancers redirecting incoming user traffic to a set of servers. This group of servers is called a server pool. When you wish to upgrade a server, you instruct the loadbalancer to remove that server from the server pool. In essence this means that the server will no longer receive any traffic coming through the load balancer from users. You can then gracefully terminate, upgrade, and restart the service(s) installed on the server safely. When these steps have been completed, the server can be re-added to the server pool. As a result the server will then start receiving traffic again through the load balancer. This procedure completes when every server has been taken from the server pool, is upgraded, and then re-added to the server pool.
<img src="http://blog.minicom.nl/images/rolling-upgrade.png" alt="" /></p>

<p>To avoid a significant loss of capacity or performance you can choose to upgrade one server at a time. This is usefull for small environments with few servers where there is little over-provisioning of computing resources. Alternatively you can also choose to upgrade several servers concurrently, which will take less time to complete the entire upgrade, but will adversely affect your capacity and maybe even performance of your web service.</p>

<h3>Big-flips</h3>

<p>Another option is the use of big-flips (also known as atomic switches). With big-flips you need a different setup of your servers. You will typically have a load balancer which redirects incoming user traffic to a server pool much like with rolling upgrades, but when you wish to perform an upgrade to your service you will need a duplicate server pool with equivalent resources to take over the workload from the initial server pool. So in order to perform the actual upgrade you must first install the upgraded version of the service on the alternate server pool. When that has been done, you will need to instruct the load balancer to stop redirecting traffic to the initial server pool, and instead redirect it to the alternate server pool. You can then safely terminate the services running on the initial server pool and release the computational resources used in this server pool.
<img src="http://blog.minicom.nl/images/big-flip.png" alt="" /></p>

<p>Big-flips are very costly in terms of computational resources. For every upgrade you require a duplicate of your production environment, and only for a very short period of time when you are performing the upgrade. This can partially be offset by using cloud providers which will rent you these computational resources per minute or hour and allow you to dispose of the initial server pool once the upgrade has succesfully completed.</p>

<h3>Trade offs</h3>

<p>I&rsquo;ve briefly introduced you to two different strategies you can use to deploy a new version of a web service. You might have noticed that these two have two very different types of &ldquo;cost&rdquo; and &ldquo;advantage&rdquo;. While rolling upgrades are slow when doing an entire upgrade sequentially for every server, they are very effecient in terms of computational resources - they require very little over-provisioning of resources, whereas big flips are quicker to perform the upgrade they require double the computational resources, which can be (too) costly.</p>

<p>So in a sense these two strategies sit at opposite ends of the same trade off: Effecient use of computational resources or a faster upgrade process.</p>

<h2>The challenge of mixed-state</h2>

<p>An often ignored part of achieving Continuous Deployment is the term &ldquo;Mixed-state&rdquo;. No upgrade or deployment is immune to it, and has to come up with some way of dealing with it. When you envision the entire web service - running on multiple servers - as a distributed software system, you need to ensure that every separate part can interface correctly with its dependencies. Mixed-state refers to the situation where mid-upgrade or mid-deployment you have two versions of the same software running concurrently, using the same dependencies. In a sense, the distributed software system is in two states now. This means that any dependency on which both versions operate, must be compatible with both versions. This is typically solved using versioned and static APIs which allow dependencies to communicate amongst each other without problems. There is however one notable exception: databases.</p>

<p>Imagine running two versions of the same web service on the same database, which has stored all its persistent data in some structure or schema. If the never version requires a slightly different schema, the older version might cease to function which could result in breaking the web service, until the upgrade of the system has completed and moved from a mixed-state to a single state again. This is aggrevated when the upgrade process takes a long time to complete.</p>

<p>This is an ongoing challenge, there is no off the shelf solution, and in this upcoming blog series I will cover it more in-depth. I will be showing you some of my research and results in this area. Stay tuned for more&hellip;</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Contributing to RxJava]]></title>
    <link href="http://blog.minicom.nl/blog/2013/07/14/contributing-to-rxjava/"/>
    <updated>2013-07-14T22:54:00+02:00</updated>
    <id>http://blog.minicom.nl/blog/2013/07/14/contributing-to-rxjava</id>
    <content type="html"><![CDATA[<p>I&rsquo;m a MSc Computer Science student at the Delft University of Technology. During my master track I&rsquo;ve had the pleasure to have followed two courses given by Erik Meijer (former Microsoft employee). Erik was involved in the creation of the Rx framework in Microsoft&rsquo;s .NET C#. If you&rsquo;re not familiar with Rx, I would recommend some of the videos found on <a href="http://channel9.msdn.com/tags/Rx/">Channel9</a>.</p>

<p>One of the courses Erik has been teaching is Reactive Programming where he covers the Rx framework and the thoughts behind it. As a part of the course, we were asked to contribute to the Rx ecosystem in whatever programming language we liked. I choose to contribute to Netflix&rsquo;s RxJava. Last week I presented my contributions and experiences to my fellow students. Below you can find the presentation slides I used.</p>

<script async class="speakerdeck-embed" data-id="c50bca20cee7013097764230e68ed5e3" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Automated web application testing for humans]]></title>
    <link href="http://blog.minicom.nl/blog/2013/04/27/automated-web-application-testing-for-humans/"/>
    <updated>2013-04-27T22:10:00+02:00</updated>
    <id>http://blog.minicom.nl/blog/2013/04/27/automated-web-application-testing-for-humans</id>
    <content type="html"><![CDATA[<p>Web applications have become a very popular breed of software. The fact that the software needed to use a web application is already installed on every internet-capable computer adds to its appeal. For a software developer there are many different architectures and even more frameworks to choose from to build web applications with. But one important part of software development - testing - can be very hard in web applications.</p>

<p>There are a few tools out there to help you with testing web applications. The most popular one seems to be Selenium. Selenium offers you two options for creating automated tests for web applications: <a href="http://docs.seleniumhq.org/projects/ide/">Selenium IDE</a>, and <a href="http://docs.seleniumhq.org/projects/webdriver/">Selenium WebDriver</a>. Selenium IDE is a simple plugin for FireFox which enables you to record interactions with a browser and replay those interactions. It&rsquo;s most suitable for testers without a programming background. Selenium WebDriver on the other hand allows testers with a programming background (or developers) to write their own automated tests, which can be run from JUnit tests. <strong>But while using these tools myself I found myself easily annoyed at how fragile these tests actually are.</strong> In order to understand why this is so you have to understand how Selenium works.</p>

<p>When you use Selenium IDE to record a session of interactions with your browser it&rsquo;s actually looking at what events you are triggering on which elements in the Document Object Model (DOM). Most of these elements can be identified with a <code>name</code>, <code>id</code> or <code>class</code> attribute. Take for example the login screen of Google&rsquo;s Gmail service. The username and password field are aptly named &ldquo;Email&rdquo; and &ldquo;Passwd&rdquo; in the DOM using the <code>id</code> attribute. So when we make a recording using Selenium IDE, it will know that I&rsquo;m typing in those two fields, and store the interaction together with that id attribute.</p>

<p><img src="http://blog.minicom.nl/images/gmail-login.png" alt="img" /> <img src="http://blog.minicom.nl/images/selenium-ide.png" alt="img" /></p>

<p>Although this is a fairly cheap way to construct a test, there are several problems with this approach:</p>

<ul>
<li><strong>Tests are not necessarily repeatable</strong>: Web application frameworks like ExtJS and Google Web Toolkit produce different unique <code>id</code> attribute values on every page serve. This means we need an alternative way of selecting the correct element in the DOM. There are several options like CSS and XPath selectors but these can get fairly complicated very quickly, which in turn negatively affects readability.</li>
<li><strong>Tests are easily broken</strong>: When a developer or designer changes the structure of the DOM, there is a chance that certain CSS and XPath selectors will no longer work. Even worse, in case the name of the <code>id</code> attribute is changed the test will definitely fail and it will have to be fixed.</li>
<li><strong>Repairing tests means re-recording the interaction between user and browser</strong>: The cost of recording is relatively small, but if tests break regularly this can become prohibitively expensive. Imaging that most of your tests are done somewhere in a secured area of your web application. In order to get there, the browser first has to login into the web application. If this login page is modified, all of these tests will have to be either re-recorded or fixed manually.</li>
<li><strong>Test setup and tear down is expensive</strong>: In case you want to test a feature in your web application you are first required to log in, navigate to the appropriate page, perform the actual test, and log out. Such a test case can take considerable time to record, while the only important section is the actual test.</li>
</ul>


<p>There is an obvious advantage in having an automated test suite which tests your web application from a user&rsquo;s perspective, but these issues are making automated tests unwieldy and costly. The most sensible conclusion is that the approach of working on a DOM level is simply too low-level for creating and maintaining an extensive automated test suite. But now let&rsquo;s approach this from a tester&rsquo;s point of view. Most of the testers I know are very capable at writing test cases for manual regression testing. These test cases are well documented and manually executed before pushing software out to customers. These regression test documents usually have the following structure:</p>

<blockquote><p><strong>Preconditions</strong>: Your browser is displaying the login screen.</p>

<p><strong>Procedure</strong></p>

<ol>
<li>Enter &ldquo;admin&rdquo; in the &ldquo;Username&rdquo; field.</li>
<li>Enter &ldquo;test&rdquo; in the &ldquo;Password&rdquo; field.</li>
<li>Click on the &ldquo;Login&rdquo; button.</li>
</ol>


<p><strong>Postcondition</strong>: You are now logged in, and your browser should be displaying the user&rsquo;s dashboard.</p></blockquote>

<p>These regression test documents have a clear structure and are easy to read and maintain for testers. The reason that this works for human testers is that they can reason about what&rsquo;s displayed in the browser. They know that the &ldquo;Username&rdquo; field is the field that has the text &ldquo;Username&rdquo; displayed right next to it. They know that the &ldquo;Login&rdquo; button is the element that&rsquo;s clickable and contains the text &ldquo;Login&rdquo;. They know what the user&rsquo;s dashboard looks like. My ideal test automation tool for web applications would be a tool which is capable of interpreting test documents like to one above, and execute them fully automatic.</p>

<p>If you iterate through all the issues I named earlier, you&rsquo;ll notice that such a tool could easily address the first three issues in the list. Since test documents don&rsquo;t specifically name attributes of elements in the DOM, it doesn&rsquo;t really matter what the <code>id</code> attribute of any element says. Modifications to the DOM on a structural level don&rsquo;t affect the interpretation of the presented web page to the same degree. And finally because this type of &lsquo;recording&rsquo; is much more resilient to changes in the DOM, it should require only a minimal amount of test case maintenance, whilst the test cases themselves remain human-readable.</p>

<p>I&rsquo;m very interested to see if it would be possible to create such a tool and to see if it indeed holds so much value for testers as I believe that it does&hellip;</p>
]]></content>
  </entry>
  
</feed>