Description
When dotAI generates embeddings for content, Story Block (Block Editor) fields are currently extracted by converting the field to HTML and then running it through Tika to strip the markup down to plain text.
In com.dotcms.ai.util.ContentToStringUtil:
private Optional<String> parseBlockEditor(@NotNull String val) {
final StoryBlockMap storyBlockMap = new StoryBlockMap(val);
return parseHTML(storyBlockMap.toHtml()); // → Tika strips all markup to plain text
}The problem: Tika (and the JSoup/regex fallbacks in parseHTML) flatten the content to plain text, which destroys structure that matters for embedding quality — tables, code blocks, ordered/unordered lists, headings. The model gets a worse, less-structured representation of the content than it should.
What we want
Embed Story Block fields as Markdown instead of HTML-stripped-to-text. Markdown is already clean (no markup to strip) and preserves the structure — tables, code fences, lists, and headings survive into the text we embed, giving the model a far better representation.
How this is now feasible
PR #35728 (issue #35727) adds StoryBlockMap.toMarkdown() (and $markdownTool.blockToMarkdown(json)), a Tiptap-JSON → Markdown converter that handles paragraphs, headings, blockquotes, lists, code blocks (with language), tables, images, links, etc. This gives us a direct JSON → Markdown path that doesn't lose structure.
The fix is essentially: in parseBlockEditor, use storyBlockMap.toMarkdown() and return that markdown directly (no Tika/HTML round-trip needed, since markdown is already plain text).
Related work
Depends on #35728 being merged (or StoryBlockMap.toMarkdown() being available).
Acceptance Criteria
Priority
Medium
Additional Context
ContentToStringUtil already imports MarkdownTool and StoryBlockMap, so the wiring is local to parseBlockEditor. The existing HTML path (toHtml() + Tika) can remain for other field types; only the Block Editor branch changes.
Description
When dotAI generates embeddings for content, Story Block (Block Editor) fields are currently extracted by converting the field to HTML and then running it through Tika to strip the markup down to plain text.
In
com.dotcms.ai.util.ContentToStringUtil:The problem: Tika (and the JSoup/regex fallbacks in
parseHTML) flatten the content to plain text, which destroys structure that matters for embedding quality — tables, code blocks, ordered/unordered lists, headings. The model gets a worse, less-structured representation of the content than it should.What we want
Embed Story Block fields as Markdown instead of HTML-stripped-to-text. Markdown is already clean (no markup to strip) and preserves the structure — tables, code fences, lists, and headings survive into the text we embed, giving the model a far better representation.
How this is now feasible
PR #35728 (issue #35727) adds
StoryBlockMap.toMarkdown()(and$markdownTool.blockToMarkdown(json)), a Tiptap-JSON → Markdown converter that handles paragraphs, headings, blockquotes, lists, code blocks (with language), tables, images, links, etc. This gives us a direct JSON → Markdown path that doesn't lose structure.The fix is essentially: in
parseBlockEditor, usestoryBlockMap.toMarkdown()and return that markdown directly (no Tika/HTML round-trip needed, since markdown is already plain text).Related work
feat(tiptap): convert Story Block content to MarkdownAcceptance Criteria
StoryBlockMap.toMarkdown()), not HTML-stripped-via-Tika.Priority
Medium
Additional Context
ContentToStringUtilalready importsMarkdownToolandStoryBlockMap, so the wiring is local toparseBlockEditor. The existing HTML path (toHtml()+ Tika) can remain for other field types; only the Block Editor branch changes.