Merged
Conversation
…ers and TokenProcessor optimizations
--- updated-dependencies: - dependency-name: Roslynator.CodeFixes dependency-version: 4.14.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
--- updated-dependencies: - dependency-name: Roslynator.Analyzers dependency-version: 4.14.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
- Replaced legacy string-based Markdown rendering with a modern Markdig-based renderer in MarkdownRenderer.cs for improved performance and reduced VT escape sequence issues. - Removed unused legacy rendering code and streamlined the Render method. - Updated Rows class to use an empty array initializer. - Enhanced StandardRenderer to utilize StringBuilder pooling for better memory management. - Refactored TextMateProcessor to support cancellation tokens and progress reporting in batch processing methods. - Updated TokenProcessor to handle escaping of markup characters and improved debugging capabilities. - Added comprehensive unit tests for StandardRenderer, TextMateProcessor, and TokenProcessor to ensure functionality and performance. - Cleaned up CacheManager tests to verify caching behavior and grammar retrieval. - Updated project file to generate documentation.
* Improved `Debug.cs` with detailed XML documentation for `TextMateDebug` class. * Refactored `DebugTextMate` and `DebugTextMateTokens` methods for better readability and performance. * Updated `Helpers.cs` to include XML documentation for `TextMateHelper` class and its members. * Enhanced `ImageFile.cs` with partial methods for regex and improved image source normalization. * Streamlined `StringBuilderPool.cs` for concise string builder management. * Updated `TextMateResolver.cs` with improved token resolution logic. * Refined `VTConversion.cs` for better parsing of VT escape sequences and style application. * Optimized `CacheManager.cs` for efficient caching of themes and grammars. * Updated project dependencies in `PSTextMate.csproj` for better compatibility. * Added a new demo script `demo-textmate.ps1` for showcasing TextMate functionality.
* Updated `Onigwrap` package version to `1.0.*` for compatibility. * Changed `TextMateSharp` package version to `2.0.*` in both project files. * Refactored tests in `MarkdownRendererTests`, `StandardRendererTests`, and `TextMateProcessorTests` to directly assert on the result instead of accessing `Renderables`. * Simplified batch processing in `TextMateProcessorTests` to improve readability. * Added new `HighlightedText` class to encapsulate rendering logic and improve type safety. * Introduced `ImageBlockRenderer` for handling image rendering with various layouts. * Removed unnecessary `Core.Rows` wrapper to streamline rendering with `Spectre.Console.Rows`.
* Refactored `InputObject` and `Path` properties to support nullable types. * Added `_sourceExtensionHint` and `_sourceBaseDirectory` for improved source detection. * Updated logic in `ProcessRecord` to handle input more robustly. * Introduced `GetSourceHint` method to extract file extension and base directory from the pipeline.
* Enhanced the logic for determining spacing between visual and non-visual blocks. * Added methods to check for visual styling and standalone images. * Refactored existing methods for clarity and performance. fix(SpanOptimizedStringExtensions): ✨ Format method for readability * Adjusted method formatting for `ContainsAnyOptimized` for better readability. fix(StringBuilderExtensions): ✨ Format method for readability * Adjusted method formatting for `AppendLink` and `AppendWithStyle` for better readability. fix(StringExtensions): ✨ Format method for readability * Adjusted method formatting for `AllIsNullOrEmpty` for better readability. fix(ImageFile): ✨ Simplify path resolution logic * Removed redundant namespace references for `Path` in `ImageFile` class.
- Added new renderers for various Markdown elements including `CodeBlock`, `Heading`, `HorizontalRule`, `HtmlBlock`, `List`, `Paragraph`, `Quote`, and `Table`. - Refactored existing renderers to utilize the new visitor pattern for better extensibility and maintainability. - Enhanced `ParagraphRenderer` to skip empty literals and handle line breaks conditionally. - Introduced `SpectreTextMateStyler` for applying TextMate styles to text, with caching for performance. - Added unit tests for Markdown pipelines and styling to ensure functionality and correctness.
* Introduced `SpectreTextMateStyler` for caching and applying styles based on TextMate scopes. * Added `StringBuilderExtensions` for optimized string building with Spectre.Console markup. * Implemented `StringBuilderPool` for efficient reuse of `StringBuilder` instances. * Created `StringExtensions` for modern string manipulation using spans. * Developed `TextMateResolver` to resolve language tokens and file extensions. * Added `ThemeExtensions` for converting TextMate themes to Spectre.Console styles. * Implemented `TokenStyleProcessor` for processing tokens and applying styles. * Created `VTParser` for parsing VT escape sequences into Spectre.Console objects. * Added test markdown files to validate spacing and rendering behavior.
* Introduced `ProcessTokensToParagraph` method for efficient token processing. * Updated various renderers (e.g., `BlockRenderer`, `HtmlBlockRenderer`, `ImageBlockRenderer`) to utilize `Text` objects instead of `Markup`, improving performance and reducing markup parsing overhead. * Enhanced `ParagraphRenderer` to build `Text` segments directly, avoiding extra spacing in terminal output. * Added integration tests to ensure inline code, links, and image text are rendered correctly. * Refactored inline content processing in `ListRenderer` and `QuoteRenderer` for consistency with new rendering approach.
…xtMate languages * Introduced `Test-SupportedTextMate` cmdlet for validating language support based on file extension, language ID, or file path. * Added `Get-SupportedTextMate` cmdlet to retrieve all supported TextMate languages and their configurations. * Updated `CmdletsToExport` in the module manifest to reflect the new cmdlets. * Enhanced string manipulation methods in `StringExtensions` for optimized performance using Span operations. * Updated dependencies in the project file for improved compatibility.
- Implemented cmdlets for formatting C#, Markdown, and PowerShell with TextMate syntax highlighting. - Introduced cmdlets for retrieving supported TextMate languages and testing support for languages and extensions. - refactoredbuild script
… and Markdown * Introduced tests for `Format-CSharp`, `Format-PowerShell`, and `Format-Markdown` functions. * Added new markdown test files to validate rendering features. * Updated build script to handle native runtime libraries more effectively. * Refactored module manifest and project file for improved structure.
…t container * Introduced a new `-Lines` switch parameter in `ShowTextMateCmdlet` and `TextMateCmdletBase` to control output format. * Updated processing logic to conditionally output either a collection of renderables or a single HighlightedText object. * Enhanced tests for `Format-CSharp`, `Format-Markdown`, `Format-PowerShell`, and `Show-TextMate` to validate new functionality.
* Refactor tests to remove the -Lines parameter and ensure they validate rendered output correctly. * Enhance test descriptions for clarity and maintainability. * Add new test cases for incomplete tasks in Markdown.
* added sixel support from new PwshSpectreConsole classes. * added LineNumbers parameter.
* Removed unused variables and redundant code in `TextMateCmdletBase`. * Simplified conditional statements in `HighlightedText` and `StandardRenderer`. * Consolidated logic in `Grapheme` for better readability. * Added new test script `TestMeasure.ps1` for measuring string properties. * Replaced old test script with a new `harness.ps1` for improved loading.
- Introduced `Format-CSharp`, `Format-Markdown`, and `Format-PowerShell` cmdlets for syntax highlighting and formatting of respective code snippets. - Added `Get-SupportedTextMate` cmdlet to retrieve supported TextMate languages and grammar metadata. - Implemented `Show-TextMate` for displaying syntax-highlighted text using TextMate grammars. - Created `Test-SupportedTextMate` to verify support for languages, extensions, or files. - Each cmdlet includes detailed examples, parameters, and usage notes to enhance user understanding.
There was a problem hiding this comment.
Pull request overview
Large refactor/release update that modernizes PSTextMate’s rendering pipeline (moving away from markup-string building toward Spectre.Console object rendering), updates the PowerShell cmdlet surface, and switches the test/build tooling to a PowerShell-first workflow.
Changes:
- Replaces/rewires Markdown rendering to build Spectre renderables directly (block-based renderers, table/list/heading/image/HTML handling).
- Introduces new cmdlets and output type (
HighlightedText) and reworks module manifest exports/aliases. - Replaces prior C# test project coverage with Pester tests and updates build scripts to an InvokeBuild-based pipeline.
Reviewed changes
Copilot reviewed 105 out of 107 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/test-line-calc.cs | Adds a Markdig diagnostic script for block line/gap calculations. |
| tools/diagnostic-trivia.cs | Adds a Markdig “trivia tracking” diagnostic script to compare line counts. |
| tools/analyze-block-lines.cs | Adds a Markdig block span/line analysis helper script. |
| tools/TestMeasure.ps1 | Adds PowerShell measurement experiments for wide/ANSI/VT strings. |
| tests/testhelper.psm1 | Adds Spectre render-to-string helpers for Pester assertions. |
| tests/test-markdown.md | Updates markdown fixture content (lists/code/table sections). |
| tests/test-markdown-rendering.md | Tweaks rendering fixture formatting (line breaks/spacing). |
| tests/Test-SupportedTextMate.tests.ps1 | Adds Pester coverage for Test-SupportedTextMate. |
| tests/Show-TextMate.tests.ps1 | Adds Pester coverage for Show-TextMate. |
| tests/PSTextMate.Tests.csproj | Removes prior C# test project. |
| tests/Integration/TaskListReflectionRemovalTests.cs | Removes prior xUnit integration tests for task lists. |
| tests/Integration/TaskListIntegrationTests.cs | Removes prior xUnit integration tests for task lists. |
| tests/GlobalUsings.cs | Removes xUnit global usings. |
| tests/Get-SupportedTextMate.tests.ps1 | Adds Pester coverage for Get-SupportedTextMate. |
| tests/Format-PowerShell.tests.ps1 | Adds Pester coverage for Format-PowerShell. |
| tests/Format-Markdown.tests.ps1 | Adds Pester coverage for Format-Markdown. |
| tests/Format-CSharp.tests.ps1 | Adds Pester coverage for Format-CSharp. |
| tests/Extensions/StringBuilderExtensionsTests.cs | Removes prior xUnit tests for StringBuilder extensions. |
| tests/Core/Markdown/MarkdownRendererTests.cs | Removes prior xUnit tests for Markdown renderer. |
| src/Utilities/TokenStyleProcessor.cs | Adds a token→Spectre renderables helper (currently has nullability issues). |
| src/Utilities/ThemeExtensions.cs | Moves/updates Theme extension utilities under new namespace/layout. |
| src/Utilities/TextMateResolver.cs | Updates resolver namespace/formatting and token heuristics. |
| src/Utilities/StringBuilderPool.cs | Adds pooled StringBuilder utility for allocation reduction. |
| src/Utilities/StringBuilderExtensions.cs | Adds optimized StringBuilder helpers (links/styled text). |
| src/Utilities/SpectreTextMateStyler.cs | Adds cached Spectre style computation for TextMate scopes. |
| src/Utilities/MarkdownPatterns.cs | Adds pattern detection helpers (e.g., standalone image paragraphs). |
| src/Utilities/InlineTextExtractor.cs | Adds reusable Markdig inline text extraction utility. |
| src/Utilities/ITextMateStyler.cs | Introduces an abstraction for applying styles to token text. |
| src/Utilities/Helpers.cs | Rehomes/extends TextMateHelper and adds line normalization. |
| src/Utilities/Completers.cs | Rehomes/updates argument completers + validate-set generators. |
| src/Utilities/AssemblyInfo.cs | Adds InternalsVisibleTo for test/proxy assemblies. |
| src/Rendering/TableRenderer.cs | Adds a Spectre object-based Markdown table renderer (has nullability/flow issues). |
| src/Rendering/QuoteRenderer.cs | Adds a Spectre object-based Markdown quote renderer. |
| src/Rendering/MarkdownRenderer.cs | Adds a Markdig-based renderer producing renderables and preserving source blank-line gaps. |
| src/Rendering/ListRenderer.cs | Adds a Spectre object-based list/task-list renderer (including nested list formatting). |
| src/Rendering/ImageBlockRenderer.cs | Adds image block layout helpers (panel/grid/table/side captions). |
| src/Rendering/HtmlBlockRenderer.cs | Updates HTML rendering to return renderables rather than markup strings. |
| src/Rendering/HorizontalRuleRenderer.cs | Updates horizontal rule renderer namespace/style. |
| src/Rendering/HeadingRenderer.cs | Adds a Spectre object-based heading renderer with theme fallbacks. |
| src/Rendering/BlockRenderer.cs | Adds block dispatch to the new per-block renderers (including standalone images). |
| src/PSTextMate.csproj | Updates analyzers settings and bumps dependency versions (wildcards in several refs). |
| src/Helpers/Helpers.cs | Removes older helper location (moved to Utilities). |
| src/Helpers/Debug.cs | Removes older debug helper code. |
| src/Helpers/Completers.cs | Removes older completers location (moved to Utilities). |
| src/Extensions/StringExtensions.cs | Removes older string extension helpers. |
| src/Extensions/StringBuilderExtensions.cs | Removes older StringBuilder extensions location (moved to Utilities). |
| src/Core/Validation/MarkdownInputValidator.cs | Removes markdown input validator utilities. |
| src/Core/TokenDebugInfo.cs | Removes debug info type used by old debug cmdlets. |
| src/Core/TextMateProcessor.cs | Refactors processing output from Rows to IRenderable[] and updates rendering paths. |
| src/Core/StyleHelper.cs | Updates namespace/formatting for style conversion utilities. |
| src/Core/StandardRenderer.cs | Refactors to build Spectre renderables (Paragraph/Rows) instead of markup strings. |
| src/Core/MarkdownToken.cs | Refactors Markdown token type (primary-constructor style). |
| src/Core/MarkdownRenderer.cs | Replaces old markdown renderer with a facade to the new Markdig-based renderer. |
| src/Core/MarkdownLinkFormatter.cs | Removes legacy markdown link markup formatter. |
| src/Core/Markdown/Types/MarkdownTypes.cs | Removes legacy markdown render result/types/options. |
| src/Core/Markdown/Renderers/QuoteRenderer.cs | Removes legacy markdown quote renderer. |
| src/Core/Markdown/Renderers/ListRenderer.cs | Removes legacy markdown list renderer. |
| src/Core/Markdown/Renderers/HeadingRenderer.cs | Removes legacy markdown heading renderer. |
| src/Core/Markdown/Renderers/BlockRenderer.cs | Removes legacy markdown block dispatcher. |
| src/Core/Markdown/README.md | Removes legacy markdown architecture doc. |
| src/Core/Markdown/Optimizations/SpanOptimizedMarkdownProcessor.cs | Removes legacy markdown span optimization utilities. |
| src/Core/Markdown/MarkdownRenderer.cs | Removes legacy markdown orchestrator. |
| src/Core/Markdown/InlineProcessor.cs | Removes legacy inline processor. |
| src/Core/MarkdigTextMateScopeMapper.cs | Updates namespace/formatting for Markdig→TextMate scope mapping. |
| src/Core/MarkdigSpectreMarkdownRenderer.cs | Removes legacy wrapper type. |
| src/Core/HighlightedText.cs | Adds new primary output type implementing IRenderable (with optional line numbers). |
| src/Core/CacheManager.cs | Updates namespace/formatting for theme/grammar caching. |
| src/Compatibility/Converter.cs | Removes legacy compatibility wrapper returning Rows. |
| src/Cmdlets/TestSupportedTextMate.cs | Adds/rewrites Test-SupportedTextMate cmdlet in new namespace/layout. |
| src/Cmdlets/SupportCmdlets.cs | Removes older combined support cmdlets file. |
| src/Cmdlets/ShowTextMateCmdlet.cs | Refactors Show-TextMate to return HighlightedText via a base cmdlet. |
| src/Cmdlets/GetSupportedTextMate.cs | Adds Get-SupportedTextMate cmdlet. |
| src/Cmdlets/FormatPowershell.cs | Adds Format-PowerShell cmdlet. |
| src/Cmdlets/FormatMarkdown.cs | Adds Format-Markdown cmdlet (with -Alternate). |
| src/Cmdlets/FormatCSharp.cs | Adds Format-CSharp cmdlet. |
| src/Cmdlets/DebugCmdlets.cs | Removes debug cmdlets previously present. |
| harness.ps1 | Adds a helper harness to load modules and run a quick markdown render. |
| docs/en-us/Test-SupportedTextMate.md | Adds cmdlet documentation. |
| docs/en-us/Show-TextMate.md | Adds cmdlet documentation. |
| docs/en-us/Get-SupportedTextMate.md | Adds cmdlet documentation. |
| docs/en-us/Format-PowerShell.md | Adds cmdlet documentation. |
| docs/en-us/Format-Markdown.md | Adds cmdlet documentation. |
| docs/en-us/Format-CSharp.md | Adds cmdlet documentation. |
| debug.ps1 | Removes older debug script. |
| build.ps1 | Refactors build entrypoint to InvokeBuild-based workflow. |
| README.md | Rewrites README for new cmdlet surface and usage examples. |
| PSTextMate.slnx | Adds new solution file format. |
| PSTextMate.generated.sln | Removes generated .sln. |
| PSTextMate.build.ps1 | Adds InvokeBuild task definitions (build/help/test). |
| Module/PSTextMate.psd1 | Updates manifest exports/aliases and module layout (RootModule path). |
| Module/PSTextMate.format.ps1xml | Updates formatting views (removes old debug views, adds Grapheme view). |
| .gitignore | Updates ignore rules (Output/tools/ref/etc). |
| .github/copilot-instructions.md | Removes repo Copilot instruction file. |
| .github/agents/csharp-dotnet-trackd.agent.md | Adds an agent configuration doc. |
src/Utilities/TokenStyleProcessor.cs
Outdated
Comment on lines
+59
to
+62
| foreach ((IToken[]? tokens, string? line) in tokenizedLines) { | ||
| // Process each line | ||
| IRenderable[] lineRenderables = ProcessTokens(tokens, line, theme, styler); | ||
|
|
There was a problem hiding this comment.
ProcessLines iterates with nullable tuple elements (IToken[]? tokens, string? line) but then passes them to ProcessTokens(IToken[] tokens, string line, ...). This won't compile with nullable reference types enabled. Iterate with non-nullables (or add null guards and make ProcessTokens accept nullables) to keep the API consistent.
Comment on lines
+40
to
+46
| // Add headers if present | ||
| (bool isHeader, List<TableCellContent> cells) headerRow = allRows.FirstOrDefault(r => r.isHeader); | ||
| if (headerRow.cells?.Count > 0) { | ||
| for (int i = 0; i < headerRow.cells.Count; i++) { | ||
| TableCellContent cell = headerRow.cells[i]; | ||
| // Use constructor to set header text; this is the most compatible way | ||
| var column = new TableColumn(cell.Text); |
There was a problem hiding this comment.
FirstOrDefault on a tuple returns the default value when no header row exists; that default has cells == null. The subsequent if (headerRow.cells?.Count > 0) doesn't convince the compiler that headerRow.cells is non-null, so headerRow.cells.Count/indexing will produce nullable warnings (and can become errors with warnings-as-errors). Consider using an explicit headerRowIndex/FindIndex, or check headerRow.cells is not null and assign to a local non-null cells variable before indexing.
src/Rendering/TableRenderer.cs
Outdated
Comment on lines
+59
to
+63
| else { | ||
| // No explicit headers, use first row as headers | ||
| (bool isHeader, List<TableCellContent> cells) = allRows.FirstOrDefault(); | ||
| if (cells?.Count > 0) { | ||
| for (int i = 0; i < cells.Count; i++) { |
There was a problem hiding this comment.
In the fallback header path, the deconstruction (bool isHeader, List<TableCellContent> cells) = allRows.FirstOrDefault(); introduces an unused isHeader local (likely triggering warnings-as-errors) and cells is still nullable from FirstOrDefault(). Prefer deconstructing into a discard (e.g., (_, var cells) = ...) and/or avoid FirstOrDefault by indexing when allRows.Count > 0 is already known.
src/Rendering/TableRenderer.cs
Outdated
Comment on lines
+81
to
+87
| foreach ((bool isHeader, List<TableCellContent>? cells) in allRows.Where(r => !r.isHeader)) { | ||
| if (cells?.Count > 0) { | ||
| var rowCells = new List<IRenderable>(); | ||
| foreach (TableCellContent? cell in cells) { | ||
| Style cellStyle = GetCellStyle(theme); | ||
| rowCells.Add(new Text(cell.Text, cellStyle)); | ||
| } |
There was a problem hiding this comment.
cells is a non-null list, but the loop declares elements as nullable (foreach (TableCellContent? cell in cells)) and then dereferences cell.Text. With nullable reference types and warnings-as-errors, this can fail the build. Iterate with a non-null element type (or add a null check) to avoid nullability warnings.
src/Cmdlets/TestSupportedTextMate.cs
Outdated
Comment on lines
+50
to
+54
| FileInfo filePath = new(GetUnresolvedProviderPathFromPSPath(File)); | ||
| if (!filePath.Exists) { | ||
| WriteError(new ErrorRecord(null, "TestSupportedTextMateCmdlet", ErrorCategory.ObjectNotFound, File)); | ||
| } | ||
| WriteObject(TextMateExtensions.IsSupportedFile(filePath.FullName)); |
There was a problem hiding this comment.
new ErrorRecord(null, ...) will throw because the ErrorRecord constructor requires a non-null Exception. Also, after writing the error, the cmdlet still calls IsSupportedFile and writes an output value, which contradicts the docs that say existence is validated first. Construct an appropriate exception (e.g., FileNotFoundException) and return after WriteError (or use ThrowTerminatingError) to avoid emitting a misleading result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lots of changes..