Implement deserialize #138

peterzhu2118 · 2020-12-16T21:02:25Z

Based on #137.

This PR implements Liquid::Template.load which reads and deserializes a parse liquid template.

I added a SerializeParseContext class that is a subclass of ParseContext that is used to store the information required for deserialization. The actual deserialization work is done in the function block_body_parse_from_serialize that is called during Liquid::BlockBody.parse when a SerializeParseContext is passed in.

peterzhu2118 · 2020-12-16T21:07:17Z

test/unit/template_test.rb

+require 'test_helper'
+
+class TemplateTest < MiniTest::Test
+  def test_serialize


I'm not sure how to (or need to) further improve test coverage.

One way to stress test it would be if we could run the tests with a monkey patch that automatically serializes and deserializes on Liquid::Template.parse. Even if we didn't always run CI with that, it would help us find missing tests by adding a test for any failure that it turns up.

I think if we add this to the benchmarks, and run those as part of CI it would add coverage for free.

We also need to make sure Liquid::Template.load(serialize).render! is, and will remain faster than Liquid::Template.parse(source).render!.

I implemented benchmarks in Shopify/liquid#1385. Deserialize is about 20% faster than parsing from source.

tobi · 2020-12-17T17:14:29Z

Great work Peter.

peterzhu2118 · 2020-12-17T17:42:12Z

Thank you Tobi! ♥

macournoyer

Wow! That was a big piece. Nice work 🙇

The only potential issue I see is w/ the assert(RTEST(tag_class));

ext/liquid_c/block.c

macournoyer · 2021-01-07T15:17:53Z

ext/liquid_c/block.c

+            VALUE markup = rb_utf8_str_new(tag_markup_header_markup(current_tag), current_tag->markup_len);
+
+            VALUE tag_class = rb_funcall(tag_registry, intern_square_brackets, 1, tag_name);
+            assert(RTEST(tag_class));


I think this should raise? Seems technically possible that tag could get removed between the time the template is parsed, cached (for a long time) & deserialized.

Maybe using Block.unknown_tag?

👍 Good point. On a unrelated note, should block.c be renamed to block_body.c since it actually contains the code for Liquid::C::BlockBody?

yes, block_body.c would be a more appropriate name for this file.

ext/liquid_c/serialize_parse_context.c

dylanahsmith · 2020-12-18T23:18:29Z

test/unit/template_test.rb

+require 'test_helper'
+
+class TemplateTest < MiniTest::Test
+  def test_serialize


One way to stress test it would be if we could run the tests with a monkey patch that automatically serializes and deserializes on Liquid::Template.parse. Even if we didn't always run CI with that, it would help us find missing tests by adding a test for any failure that it turns up.

dylanahsmith · 2020-12-18T23:20:10Z

test/unit/template_test.rb

+      dump_load_eval('{% if test %}goodbye {% else %}hello {% endif %}world', 'test' => false))
+    assert_equal('hello world', dump_load_eval('{% if true %}hello {% endif %}{% if true %}world{% endif %}'))
+    assert_equal('123', dump_load_eval('{% for i in (1..10) %}{{i}}{% if i == 3 %}{% break %}{% endif %}{% endfor %}'))
+  end


We could use a unit test that ensures that line numbers are being set properly. I think Liquid::ParseContext#line_number is being inherited by Liquid::C::SerializeParseContext without it ever being set on deserialization.

Is e3f403c the kind of test you were thinking of?

ext/liquid_c/template.c

dylanahsmith · 2021-01-05T14:25:59Z

ext/liquid_c/template.c

+    const char *data = RSTRING_PTR(source);
+
+    document_body_header_t *header = (document_body_header_t *)data;
+


We should at least have a serialization format version number in the header to check against in order be able to make backwards incompatible changes. We will need to actually raise an exception for that check to safely handle the incompatible version error by loading and parsing the template source.

Another check we could use would be an endianness check. Little-endian seems fairly standard these days, so if we don't want to extend the header and add a runtime check, we could have a compile-time check for it in extconf.rb. We can do the same if we add any more platform specific dependencies.

We should also document that this method needs to be passed data serialized by this library to be used safely, since we probably don't want to add significant overhead to provide runtime safety.

Fixed in 005b088. I added a Liquid::C::DeserializationError class that is raised for all serialization incompatibility reasons.

ext/liquid_c/serialize_parse_context.c

ext/liquid_c/block.c

ext/liquid_c/tag_markup.c

Co-Authored-By: Dylan Thacker-Smith <Dylan.Smith@shopify.com>

…ext_tag Co-Authored-By: Dylan Thacker-Smith <Dylan.Smith@shopify.com>

Co-Authored-By: Dylan Thacker-Smith <Dylan.Smith@shopify.com>

…ion has completed

peterzhu2118 · 2021-01-11T15:54:59Z

@dylanahsmith I added a rake task that runs the liquid integration tests for serialization in a65f10e and it's now running as part of CI. The commits after that commit are fixes I made from bugs discovered when running the integration tests.

peterzhu2118 requested a review from dylanahsmith December 16, 2020 21:02

peterzhu2118 force-pushed the pz-deserialize branch from 809932a to ed940df Compare December 16, 2020 21:04

peterzhu2118 commented Dec 16, 2020

View reviewed changes

dylanahsmith requested a review from macournoyer December 18, 2020 22:31

dylanahsmith mentioned this pull request Jan 4, 2021

Fixes for serialization Shopify/liquid#1380

Merged

peterzhu2118 mentioned this pull request Jan 6, 2021

Implement benchmarks for deserialization Shopify/liquid#1385

Closed

macournoyer approved these changes Jan 7, 2021

View reviewed changes

peterzhu2118 force-pushed the pz-serialize branch from 986bff5 to bafe96f Compare January 7, 2021 16:11

peterzhu2118 force-pushed the pz-deserialize branch 3 times, most recently from 0b504b7 to 07ffbfb Compare January 7, 2021 16:37

dylanahsmith reviewed Jan 7, 2021

View reviewed changes

dylanahsmith mentioned this pull request Jan 7, 2021

Make Liquid::C::Tokenizer#shift private so it is only used for testing #140

Merged

macournoyer mentioned this pull request Jan 7, 2021

Implement serialize #137

Open

peterzhu2118 force-pushed the pz-serialize branch from bafe96f to 0e8d1c2 Compare January 8, 2021 19:32

peterzhu2118 force-pushed the pz-deserialize branch from 07ffbfb to a406547 Compare January 8, 2021 19:34

peterzhu2118 and others added 12 commits January 11, 2021 10:43

Implement deserialize

624cec1

Raise unknown tag in deserialization

fde0cc2

Remove unnecessary offset fields in tag_markup_header_t

116920d

Co-Authored-By: Dylan Thacker-Smith <Dylan.Smith@shopify.com>

Rename tags to tag_markups on the vm_assembler_t to avoid confusion

ce68924

Co-Authored-By: Dylan Thacker-Smith <Dylan.Smith@shopify.com>

Remove serialize_parse_context.current_tag and split tag_markup_get_n…

3ff9c15

…ext_tag Co-Authored-By: Dylan Thacker-Smith <Dylan.Smith@shopify.com>

Remove block_body_parse_from_serialize exception handler

84301b5

Co-Authored-By: Dylan Thacker-Smith <Dylan.Smith@shopify.com>

Fix rebase conflicts

997b213

Co-Authored-By: Dylan Thacker-Smith <Dylan.Smith@shopify.com>

Add Liquid::C::DeserializationError

2d1aabc

Pass options to SerializeParseContext

3bf7485

Rerfactor source to serialized_data

378ae55

Add assertion

1a83ab7

Add test for serialization with line numbers

532d60f

peterzhu2118 added 6 commits January 11, 2021 10:43

Run liquid integration tests for serialization

a65f10e

Let SerializeParseContext behave like ParseContext after deserializat…

1cf94ca

…ion has completed

Keep track of the current tag in the SerializeParseContext

8ed1eda

Set the line numbers during deserialize

10e9294

Fix BlockBody#blank?

dd35354

Fix rebase issues

5b1f9d8

peterzhu2118 force-pushed the pz-serialize branch from 0e8d1c2 to f26621c Compare January 11, 2021 15:51

peterzhu2118 force-pushed the pz-deserialize branch from e3f403c to 5b1f9d8 Compare January 11, 2021 15:51

peterzhu2118 requested a review from dylanahsmith January 11, 2021 15:55

peterzhu2118 mentioned this pull request Jan 11, 2021

Implement benchmarks for deserialize, and deserialize & render #141

Open

		const char *data = RSTRING_PTR(source);

		document_body_header_t header = (document_body_header_t )data;

Implement deserialize #138

Are you sure you want to change the base?

Implement deserialize #138

Uh oh!

Conversation

peterzhu2118 commented Dec 16, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobi commented Dec 17, 2020

Uh oh!

peterzhu2118 commented Dec 17, 2020

Uh oh!

macournoyer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

peterzhu2118 commented Jan 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants