diff --git a/.github/workflows/upload_sources_to_crowdin.yml b/.github/workflows/upload_sources_to_crowdin.yml index 2ab0d8ab..87de1f10 100644 --- a/.github/workflows/upload_sources_to_crowdin.yml +++ b/.github/workflows/upload_sources_to_crowdin.yml @@ -7,6 +7,28 @@ on: description: Optional docs-relative paths, comma or newline separated. required: false type: string + paths_to_delete: + description: Optional docs-relative paths to remove from docs-structure, comma or newline separated. + required: false + type: string + workflow_call: + inputs: + paths_to_upload: + description: Optional docs-relative paths, comma or newline separated. + required: false + type: string + paths_to_delete: + description: Optional docs-relative paths to remove from docs-structure, comma or newline separated. + required: false + type: string + secrets: + CROWDIN_PROJECT_ID: + required: true + CROWDIN_PERSONAL_TOKEN: + required: true + +permissions: + contents: write jobs: upload-sources-to-crowdin: @@ -26,12 +48,27 @@ jobs: run: npm ci - name: Generate docs structure manifest + env: + PATHS_TO_UPLOAD: ${{ inputs.paths_to_upload || '' }} + PATHS_TO_DELETE: ${{ inputs.paths_to_delete || '' }} run: npm run generate_doc_structure + - name: Commit docs structure manifest + run: | + if [ -n "$(git status --porcelain -- docs-structure.json)" ]; then + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + git add docs-structure.json + git commit -m "chore: update docs structure manifest" + git push + else + echo "No docs-structure.json changes to commit." + fi + - name: Generate crowdin file id: extract_files env: - PATHS_TO_UPLOAD: ${{ github.event.inputs.paths_to_upload || '' }} + PATHS_TO_UPLOAD: ${{ inputs.paths_to_upload || '' }} run: npm run generate_file - name: crowdin action diff --git a/src/README.md b/src/README.md index eeb397f3..3c2e735c 100644 --- a/src/README.md +++ b/src/README.md @@ -1,36 +1,138 @@ -# ita_documentation +# devportal-docs – source scripts -## Translation manifest +This directory contains the TypeScript scripts that power the `upload_sources_to_crowdin` GitHub Actions workflow. They are responsible for keeping a `docs-structure.json` manifest up to date and for regenerating the `crowdin.yml` configuration file before every upload. -Before uploading sources to Crowdin, the workflow now generates a `docs-structure.json` manifest. +--- -The manifest is meant to be translated together with the markdown sources and later consumed by the downstream repository to rename translated directories and files. +## High-level workflow + +``` +Trigger (manual / called workflow) + └─ generate_doc_structure → writes / updates docs-structure.json + └─ git commit + push (only when the file actually changed) + └─ generate_file → writes crowdin.yml + └─ crowdin-action → uploads sources to Crowdin +``` + +The two workflow inputs that control behaviour are: + +- `paths_to_upload` – optional list of docs-relative paths whose sources should be added/refreshed. +- `paths_to_delete` – optional list of docs-relative paths whose nodes should be removed from the manifest. + +Both inputs accept a JSON array, a comma-separated list, or one path per line. + +--- + +## Scripts + +### `npm run generate_doc_structure` + +Entry point: [`generateDocStructure.ts`](generateDocStructure.ts) + +Reads `PATHS_TO_UPLOAD` and `PATHS_TO_DELETE` from the environment and writes `docs-structure.json`. + +- **Full scan** (no paths provided): walks the entire `docs/` tree and rebuilds the manifest from scratch. +- **Incremental update** (`PATHS_TO_UPLOAD` set): loads the existing manifest and merges only the selected nodes into it, creating any missing intermediate directory nodes. +- **Deletion** (`PATHS_TO_DELETE` set): loads the existing manifest and removes the targeted nodes. + +The script exits with a non-zero code if `docs/` does not exist or if the manifest cannot be written. + +### `npm run generate_file` + +Entry point: [`generateCrowdinConfig.ts`](generateCrowdinConfig.ts) + +Reads `PATHS_TO_UPLOAD` from the environment and writes `crowdin.yml`. + +- When `PATHS_TO_UPLOAD` is set, only the `.md` files under the selected paths are included. +- When it is empty, all `.md` files under `docs/` are included. +- `docs-structure.json` is always prepended to the files list so translators can translate folder/file labels. +- When running on GitHub Actions (`GITHUB_OUTPUT` is set), the script also exposes a `found_files` step output containing a JSON array of the collected markdown paths. + +--- + +## Shared helpers – `docsStructure.ts` + +All tree-walking, merging, and path-normalisation logic lives in [`docsStructure.ts`](docsStructure.ts). Key exports: + +| Export | Purpose | +|---|---| +| `buildDirectoryNode` | Recursively walks a directory and builds a node tree. Dashes and underscores in names are converted to spaces for the `label`. | +| `collectDocsData` | Full scan of `docs/`; returns the manifest tree and the flat list of all `.md` paths. | +| `collectSelectedMarkdownFiles` | Expands a list of selected paths to individual `.md` files, recursing into directories and skipping `.gitbook`. | +| `readDocsStructureManifest` | Loads `docs-structure.json`; falls back to an empty root node when the file is missing or malformed. | +| `writeDocsStructureManifest` | Orchestrates reading, merging, and writing the manifest. | +| `mergeManifestWithSelectedPaths` | Inserts or updates nodes for selected paths in the existing tree. | +| `deleteSelectedNode` | Removes a node (and its subtree) from the manifest. | +| `buildCrowdinFileEntries` | Maps source `.md` paths to Crowdin `source`/`translation` pairs, injecting `%locale%` after `docs/`. | +| `parseRequestedDocsPaths` | Parses a JSON array, CSV, or newline-separated string into a `string[]`. | + +--- + +## Translation manifest (`docs-structure.json`) + +The manifest is uploaded to Crowdin alongside the Markdown sources so translators can provide localised names for each directory and file. A downstream repository consumes the translated copy to rename directories and files when assembling the localised site. ### Schema -The schema keeps the source folder and file names in object keys so they remain stable across locales. Only the `label` values are meant to be translated. +Object keys are the original filesystem names (stable across locales). Only `label` values are translated. + +**Directory node** + +```jsonc +"soluzioni": { + "label": "soluzioni", // translatable + "directory": true, + "children": { /* nested nodes */ } +} +``` -Directory nodes expose: +**File node** -- `label`: the source name to translate. -- `directory`: `true`. -- `children`: nested directory and file entries. +```jsonc +"README.md": { + "label": "README", // translatable + "directory": false +} +``` -File nodes expose: +**Full example** -- `label`: the source basename to translate. -- `directory`: `false`. +```jsonc +{ + "version": 1, + "tree": { + "docs": { + "label": "docs", + "directory": true, + "children": { + "soluzioni": { + "label": "soluzioni", + "directory": true, + "children": { + "asilo-nido": { + "label": "asilo nido", + "directory": true, + "children": { + "README.md": { "label": "README", "directory": false } + } + } + } + } + } + } + } +} +``` -### Scripts +--- -- `npm run generate_doc_structure`: generates `docs-structure.json`. -- `npm run generate_file`: regenerates `crowdin.yml`, including the manifest upload entry. +## Workflow dispatch inputs -### Workflow dispatch input +When triggering `.github/workflows/upload_sources_to_crowdin.yml` manually from the GitHub UI: -When manually running `.github/workflows/upload_sources_to_crowdin.yml`, you can optionally set `paths_to_upload` to limit the uploaded markdown files. +| Input | Description | +|---|---| +| `paths_to_upload` | Paths to add/refresh. Resolved from `docs/`, so `app-io/guide/1.0` expands to all `.md` files under `docs/app-io/guide/1.0`. Explicit `.md` files are also accepted with or without the `docs/` prefix. | +| `paths_to_delete` | Paths whose nodes should be removed from the manifest. Same format as `paths_to_upload`. | -- Paths are resolved from `docs/` by default, so `app-io/manuale-gruppi-io/1.0` expands to every `.md` file under `docs/app-io/manuale-gruppi-io/1.0`. -- Explicit markdown files are still supported, with or without the `docs/` prefix. -- Provide one path per line or a comma-separated list. -- `.gitbook` directories are ignored. +Provide values as one path per line or a comma-separated list. `.gitbook` directories are always ignored. diff --git a/src/docsStructure.ts b/src/docsStructure.ts index 34e35184..cd1f1914 100644 --- a/src/docsStructure.ts +++ b/src/docsStructure.ts @@ -24,6 +24,12 @@ export interface DocsStructureManifest { tree: Record; } +export interface WriteDocsStructureManifestOptions { + selectedPaths?: string[]; + pathsToDelete?: string[]; + existingManifestPath?: string; +} + function toPosixPath(filePath: string): string { return filePath.replace(/\\/g, '/'); } @@ -41,6 +47,24 @@ function isPathWithinRoot(rootPath: string, candidatePath: string): boolean { return relativePath === '' || (!relativePath.startsWith('..') && !path.isAbsolute(relativePath)); } +function safeStatSync(absolutePath: string, rootPath: string, displayPath: string): fs.Stats { + const lstat = fs.lstatSync(absolutePath); + + if (lstat.isSymbolicLink()) { + const realPath = fs.realpathSync(absolutePath); + + if (!isPathWithinRoot(rootPath, realPath)) { + throw new Error( + `The path "${displayPath}" is a symlink that resolves outside "${DOCS_DIR}/".`, + ); + } + + return fs.statSync(realPath); + } + + return lstat; +} + function toCrowdinSourcePath(entryPath: string, rootDir: string): string { const rootPath = path.resolve(rootDir); const absoluteEntryPath = path.resolve(entryPath); @@ -49,10 +73,12 @@ function toCrowdinSourcePath(entryPath: string, rootDir: string): string { } function normalizeSelectedPath(selectedPath: string): string { - const normalizedPath = toPosixPath(selectedPath.trim()).replace(/^\.\//, ''); + const normalizedPath = toPosixPath(selectedPath.trim()).replace(/^\.\//, '').replace(/\/+$/, ''); - if (normalizedPath === DOCS_DIR) { - return ''; + if (normalizedPath === '' || normalizedPath === DOCS_DIR) { + throw new Error( + `The path "${selectedPath}" refers to the "${DOCS_DIR}/" root. Specify a Markdown file or subdirectory, or omit selected paths to perform a full scan.`, + ); } if (normalizedPath.startsWith(`${DOCS_DIR}/`)) { @@ -62,6 +88,10 @@ function normalizeSelectedPath(selectedPath: string): string { return normalizedPath; } +function toLabel(entryName: string): string { + return entryName.replace(/[-_]+/g, ' '); +} + function includesIgnoredDirectory(candidatePath: string): boolean { const segments = toPosixPath(candidatePath).split('/'); return segments.some((segment) => IGNORED_DIRECTORY_NAMES.has(segment)); @@ -121,18 +151,318 @@ function buildDirectoryNode( mdFiles.push(toCrowdinSourcePath(entryPath, rootDir)); children[entry.name] = { - label: path.basename(entry.name, '.md').replace(/[-_]+/g, ' '), + label: toLabel(path.basename(entry.name, '.md')), directory: false, }; } return { - label: path.basename(dirPath).replace(/[-_]+/g, ' '), + label: toLabel(path.basename(dirPath)), directory: true, children, }; } +function createEmptyRootNode(rootDir: string): DocsStructureNode { + return { + label: toLabel(path.basename(rootDir)), + directory: true, + children: {}, + }; +} + +function ensureDirectoryNode(node: DocsStructureNode): DocsStructureNode { + if (!node.directory) { + return { + label: node.label, + directory: true, + children: {}, + }; + } + + if (!node.children) { + return { + ...node, + children: {}, + }; + } + + return node; +} + +function cloneNode(node: DocsStructureNode): DocsStructureNode { + if (!node.directory) { + return { + label: node.label, + directory: false, + }; + } + + const childrenEntries = Object.entries(node.children ?? {}).map(([childName, childNode]) => [ + childName, + cloneNode(childNode), + ]); + + return { + label: node.label, + directory: true, + children: Object.fromEntries(childrenEntries), + }; +} + +function mergeNodes(baseNode: DocsStructureNode, incomingNode: DocsStructureNode): DocsStructureNode { + if (!incomingNode.directory) { + return cloneNode(incomingNode); + } + + const normalizedBaseNode = ensureDirectoryNode(baseNode); + const mergedChildren: Record = { + ...(normalizedBaseNode.children ?? {}), + }; + + for (const [childName, incomingChildNode] of Object.entries(incomingNode.children ?? {})) { + const existingChildNode = mergedChildren[childName]; + + if (!existingChildNode) { + mergedChildren[childName] = cloneNode(incomingChildNode); + continue; + } + + mergedChildren[childName] = mergeNodes(existingChildNode, incomingChildNode); + } + + return { + label: incomingNode.label || normalizedBaseNode.label, + directory: true, + children: mergedChildren, + }; +} + +function createNodeFromSelectedEntry(selectedAbsolutePath: string, rootDir: string): DocsStructureNode { + const rootPath = path.resolve(rootDir); + const selectedPathStat = safeStatSync(selectedAbsolutePath, rootPath, selectedAbsolutePath); + + if (selectedPathStat.isDirectory()) { + return buildDirectoryNode(selectedAbsolutePath, rootDir, []); + } + + return { + label: toLabel(path.basename(selectedAbsolutePath, '.md')), + directory: false, + }; +} + +function insertSelectedNode( + rootNode: DocsStructureNode, + relativeSegments: string[], + selectedNode: DocsStructureNode, +) { + const normalizedRootNode = ensureDirectoryNode(rootNode); + + if (relativeSegments.length === 0) { + normalizedRootNode.children = { + ...(selectedNode.children ?? {}), + }; + normalizedRootNode.label = selectedNode.label; + return; + } + + let cursorNode = normalizedRootNode; + + for (let index = 0; index < relativeSegments.length - 1; index += 1) { + const segment = relativeSegments[index]; + const existingChild = cursorNode.children?.[segment]; + + const nextNode = existingChild + ? ensureDirectoryNode(existingChild) + : { + label: toLabel(segment), + directory: true, + children: {}, + }; + + if (!cursorNode.children) { + cursorNode.children = {}; + } + + cursorNode.children[segment] = nextNode; + cursorNode = nextNode; + } + + const targetSegment = relativeSegments[relativeSegments.length - 1]; + const existingTargetNode = cursorNode.children?.[targetSegment]; + const mergedTargetNode = existingTargetNode + ? mergeNodes(existingTargetNode, selectedNode) + : cloneNode(selectedNode); + + if (!cursorNode.children) { + cursorNode.children = {}; + } + + cursorNode.children[targetSegment] = mergedTargetNode; +} + +function deleteSelectedNode(rootNode: DocsStructureNode, relativeSegments: string[]) { + const normalizedRootNode = ensureDirectoryNode(rootNode); + + if (relativeSegments.length === 0) { + normalizedRootNode.children = {}; + return; + } + + let cursorNode = normalizedRootNode; + + for (let index = 0; index < relativeSegments.length - 1; index += 1) { + const segment = relativeSegments[index]; + const nextNode = cursorNode.children?.[segment]; + + if (!nextNode || !nextNode.directory) { + return; + } + + cursorNode = ensureDirectoryNode(nextNode); + } + + const targetSegment = relativeSegments[relativeSegments.length - 1]; + + if (!cursorNode.children?.[targetSegment]) { + return; + } + + delete cursorNode.children[targetSegment]; +} + +function mergeManifestWithSelectedPaths( + existingManifest: DocsStructureManifest, + selectedPaths: string[], + pathsToDelete: string[], + rootDir: string, +): DocsStructureManifest { + const rootPath = path.resolve(rootDir); + const rootName = path.basename(rootDir); + const baseRootNode = ensureDirectoryNode( + cloneNode(existingManifest.tree[rootName] ?? createEmptyRootNode(rootDir)), + ); + + for (const rawSelectedPath of selectedPaths) { + const normalizedPath = normalizeSelectedPath(rawSelectedPath); + + if (includesIgnoredDirectory(normalizedPath)) { + continue; + } + + const absoluteSelectedPath = path.resolve(rootPath, normalizedPath); + + if (!isPathWithinRoot(rootPath, absoluteSelectedPath)) { + throw new Error(`The path "${rawSelectedPath}" must stay within "${DOCS_DIR}/".`); + } + + if (!fs.existsSync(absoluteSelectedPath)) { + throw new Error(`The path "${rawSelectedPath}" does not exist within "${DOCS_DIR}/".`); + } + + const selectedPathStat = safeStatSync(absoluteSelectedPath, rootPath, rawSelectedPath); + + if (!selectedPathStat.isDirectory() && (!selectedPathStat.isFile() || !isMarkdownFile(path.basename(absoluteSelectedPath)))) { + throw new Error(`The path "${rawSelectedPath}" must be a Markdown file or a directory.`); + } + + const relativePath = toPosixPath(path.relative(rootPath, absoluteSelectedPath)); + const relativeSegments = relativePath ? relativePath.split('/') : []; + const selectedNode = createNodeFromSelectedEntry(absoluteSelectedPath, rootDir); + insertSelectedNode(baseRootNode, relativeSegments, selectedNode); + } + + for (const rawPathToDelete of pathsToDelete) { + const normalizedPath = normalizeSelectedPath(rawPathToDelete); + + if (includesIgnoredDirectory(normalizedPath)) { + continue; + } + + const absolutePathToDelete = path.resolve(rootPath, normalizedPath); + + if (!isPathWithinRoot(rootPath, absolutePathToDelete)) { + throw new Error(`The path "${rawPathToDelete}" must stay within "${DOCS_DIR}/".`); + } + + const relativePath = toPosixPath(path.relative(rootPath, absolutePathToDelete)); + const relativeSegments = relativePath ? relativePath.split('/') : []; + deleteSelectedNode(baseRootNode, relativeSegments); + } + + return { + version: 1, + tree: { + [rootName]: baseRootNode, + }, + }; +} + +export function parseRequestedDocsPaths(rawValue: string | undefined): string[] { + if (!rawValue) { + return []; + } + + const trimmedValue = rawValue.trim(); + + if (!trimmedValue) { + return []; + } + + try { + const parsedValue = JSON.parse(trimmedValue) as unknown; + + if (Array.isArray(parsedValue)) { + return parsedValue + .map((entry) => `${entry}`.trim()) + .filter((entry) => entry.length > 0); + } + } catch { + // Fall back to simple text parsing for workflow_dispatch input values. + } + + return trimmedValue + .split(/\r?\n|,/) + .map((entry) => entry.trim()) + .filter((entry) => entry.length > 0); +} + +export function readDocsStructureManifest( + manifestPath: string = DOCS_STRUCTURE_FILE, + rootDir: string = DOCS_DIR, +): DocsStructureManifest { + const rootName = path.basename(rootDir); + const fallbackManifest: DocsStructureManifest = { + version: 1, + tree: { + [rootName]: createEmptyRootNode(rootDir), + }, + }; + + if (!fs.existsSync(manifestPath)) { + return fallbackManifest; + } + + try { + const fileContent = fs.readFileSync(manifestPath, 'utf8'); + const parsedManifest = JSON.parse(fileContent) as Partial; + const tree = parsedManifest.tree ?? {}; + const parsedRootNode = tree[rootName]; + + return { + version: 1, + tree: { + [rootName]: parsedRootNode + ? ensureDirectoryNode(cloneNode(parsedRootNode)) + : createEmptyRootNode(rootDir), + }, + }; + } catch { + return fallbackManifest; + } +} + export function collectDocsData(rootDir: string = DOCS_DIR): { manifest: DocsStructureManifest; mdFiles: string[]; @@ -170,7 +500,7 @@ export function collectSelectedMarkdownFiles( for (const rawSelectedPath of selectedPaths) { const normalizedPath = normalizeSelectedPath(rawSelectedPath); - if (!normalizedPath || includesIgnoredDirectory(normalizedPath)) { + if (includesIgnoredDirectory(normalizedPath)) { continue; } @@ -184,7 +514,7 @@ export function collectSelectedMarkdownFiles( throw new Error(`The path "${rawSelectedPath}" does not exist within "${DOCS_DIR}/".`); } - const selectedPathStat = fs.statSync(absoluteSelectedPath); + const selectedPathStat = safeStatSync(absoluteSelectedPath, rootPath, rawSelectedPath); if (selectedPathStat.isDirectory()) { collectMarkdownFilesFromDirectory(absoluteSelectedPath, rootDir, collectedFiles); @@ -219,8 +549,19 @@ export function buildCrowdinFileEntries(mdFiles: string[]): CrowdinFileEntry[] { export function writeDocsStructureManifest( outputPath: string = DOCS_STRUCTURE_FILE, rootDir: string = DOCS_DIR, + options: WriteDocsStructureManifestOptions = {}, ): DocsStructureManifest { - const { manifest } = collectDocsData(rootDir); + const selectedPaths = options.selectedPaths ?? []; + const pathsToDelete = options.pathsToDelete ?? []; + const manifest = selectedPaths.length > 0 || pathsToDelete.length > 0 + ? mergeManifestWithSelectedPaths( + readDocsStructureManifest(options.existingManifestPath ?? outputPath, rootDir), + selectedPaths, + pathsToDelete, + rootDir, + ) + : collectDocsData(rootDir).manifest; + fs.writeFileSync(outputPath, `${JSON.stringify(manifest, null, 2)}\n`, 'utf8'); return manifest; } diff --git a/src/generateCrowdinConfig.ts b/src/generateCrowdinConfig.ts index 33b63eb8..5b530cb9 100644 --- a/src/generateCrowdinConfig.ts +++ b/src/generateCrowdinConfig.ts @@ -8,6 +8,7 @@ import { collectSelectedMarkdownFiles, CONFIG_FILE, DOCS_DIR, + parseRequestedDocsPaths, type CrowdinFileEntry, } from './docsStructure'; @@ -17,35 +18,6 @@ interface CrowdinConfig { files: CrowdinFileEntry[]; } -function parseRequestedDocsPaths(rawValue: string | undefined): string[] { - if (!rawValue) { - return []; - } - - const trimmedValue = rawValue.trim(); - - if (!trimmedValue) { - return []; - } - - try { - const parsedValue = JSON.parse(trimmedValue) as unknown; - - if (Array.isArray(parsedValue)) { - return parsedValue - .map((entry) => `${entry}`.trim()) - .filter((entry) => entry.length > 0); - } - } catch { - // Fall back to simple text parsing for workflow_dispatch input values. - } - - return trimmedValue - .split(/\r?\n|,/) - .map((entry) => entry.trim()) - .filter((entry) => entry.length > 0); -} - function generateCrowdinConfig() { console.log(`🔍 Scanning the "${DOCS_DIR}" directory...`); diff --git a/src/generateDocStructure.ts b/src/generateDocStructure.ts index 26bf6cb9..51bf1379 100644 --- a/src/generateDocStructure.ts +++ b/src/generateDocStructure.ts @@ -2,6 +2,7 @@ import * as fs from 'fs'; import { DOCS_DIR, DOCS_STRUCTURE_FILE, + parseRequestedDocsPaths, writeDocsStructureManifest, } from './docsStructure'; @@ -13,13 +14,29 @@ function generateDocStructure() { process.exit(1); } + const selectedPaths = parseRequestedDocsPaths(process.env.PATHS_TO_UPLOAD); + const pathsToDelete = parseRequestedDocsPaths(process.env.PATHS_TO_DELETE); + let manifest: ReturnType; try { - manifest = writeDocsStructureManifest(); + manifest = writeDocsStructureManifest(DOCS_STRUCTURE_FILE, DOCS_DIR, { + selectedPaths, + pathsToDelete, + existingManifestPath: DOCS_STRUCTURE_FILE, + }); } catch (error) { console.error('❌ Error while generating the manifest:', error); process.exit(1); } + + if (selectedPaths.length > 0) { + console.log(`🎯 Incrementally updated manifest from ${selectedPaths.length} selected path(s).`); + } + + if (pathsToDelete.length > 0) { + console.log(`🗑️ Removed ${pathsToDelete.length} selected path(s) from manifest.`); + } + console.log(`✅ Updated ${DOCS_STRUCTURE_FILE}.`); if (!manifest.tree.docs) { diff --git a/tsconfig.json b/tsconfig.json index 0f881e47..5cdde441 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -8,10 +8,5 @@ "skipLibCheck": true, "types": ["node"] }, - "include": [ - "*.ts", - "src/docsStructure.ts", - "src/generateCrowdinConfig.ts", - "src/generateDocStructure.ts" - ] + "include": ["*.ts", "src/**/*.ts"] } diff --git a/upload_sources_to_crowdin_workflow.md b/upload_sources_to_crowdin_workflow.md new file mode 100644 index 00000000..6f721b5d --- /dev/null +++ b/upload_sources_to_crowdin_workflow.md @@ -0,0 +1,270 @@ +# `upload_sources_to_crowdin` Workflow + +This document explains the [.github/workflows/upload_sources_to_crowdin.yml](.github/workflows/upload_sources_to_crowdin.yml) workflow, the scripts it runs, and how data flows between them. + +The workflow uploads Markdown source files from `docs/` (plus a `docs-structure.json` manifest) to Crowdin so translators can work on them. It supports both a full upload (every `.md` under `docs/`) and an incremental upload limited to user-selected paths, as well as deletions from the structure manifest. + +--- + +## 1. High-level overview + +```mermaid +flowchart TD + A[Trigger: workflow_dispatch or workflow_call] --> B[Checkout repo] + B --> C[Setup Node.js] + C --> D[Cache + install npm deps] + D --> E[generate_doc_structure
updates docs-structure.json] + E --> F[Commit docs-structure.json
if changed] + F --> G[generate_file
writes crowdin.yml] + G --> H["crowdin/github-action@v2
uploads sources"] +``` + +Two inputs drive the behavior: + +- `paths_to_upload` — optional list of docs-relative paths to add/refresh. +- `paths_to_delete` — optional list of docs-relative paths to remove from the manifest. + +Both accept comma-separated, newline-separated, or JSON array values (see [`parseRequestedDocsPaths`](src/docsStructure.ts#L385-L412)). + +--- + +## 2. Trigger and inputs + +```mermaid +flowchart LR + M[Manual run
workflow_dispatch] --> J[Job: upload-sources-to-crowdin] + W[Called by another workflow
workflow_call] --> J + J --> S[(Secrets:
CROWDIN_PROJECT_ID
CROWDIN_PERSONAL_TOKEN)] +``` + +- `workflow_dispatch` lets a maintainer run the workflow from the GitHub UI and optionally fill `paths_to_upload` / `paths_to_delete`. +- `workflow_call` lets other workflows reuse this one and pass the same inputs plus the required Crowdin secrets. +- `permissions: contents: write` is required because the job commits `docs-structure.json` back to the branch. + +--- + +## 3. Step-by-step breakdown + +### 3.1 Checkout + Node setup + dependency cache + +```yaml +- uses: actions/checkout@v6 +- uses: ./.github/actions/setup-node +- uses: actions/cache@v5 + with: + path: ~/.npm + key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }} +- run: npm ci +``` + +Standard preparation: + +1. Check out the branch that triggered the workflow. +2. Run the local composite action `setup-node` (Node version pinned for the project). +3. Restore the npm cache keyed on `package-lock.json`, so unchanged lockfiles skip a full install. +4. `npm ci` installs the exact dependency tree required by the TypeScript scripts (`ts-node`, `typescript`, `js-yaml`, `@types/*`). + +### 3.2 Generate the docs structure manifest + +```yaml +- name: Generate docs structure manifest + env: + PATHS_TO_UPLOAD: ${{ inputs.paths_to_upload || '' }} + PATHS_TO_DELETE: ${{ inputs.paths_to_delete || '' }} + run: npm run generate_doc_structure +``` + +`npm run generate_doc_structure` is defined in [package.json](package.json#L8-L11) as: + +```text +ts-node-script --project tsconfig.json generateDocStructure.ts +``` + +It runs [generateDocStructure.ts](generateDocStructure.ts), which: + +1. Verifies `docs/` exists. +2. Parses `PATHS_TO_UPLOAD` and `PATHS_TO_DELETE` via [`parseRequestedDocsPaths`](docsStructure.ts#L385-L412) (accepts JSON array, CSV, or newline list). +3. Calls [`writeDocsStructureManifest`](docsStructure.ts#L519-L538) with those lists. +4. Logs a summary and exits non-zero on failure. + +#### What `writeDocsStructureManifest` does + +```mermaid +flowchart TD + A[writeDocsStructureManifest] --> B{selected or
delete paths
present?} + B -- No --> F[collectDocsData
full rescan of docs/] + B -- Yes --> C[readDocsStructureManifest
load existing JSON] + C --> D[mergeManifestWithSelectedPaths] + D --> E[insert/update selected nodes
+ delete requested nodes] + F --> G[Build manifest object
version + tree.docs] + E --> G + G --> H[Write docs-structure.json] +``` + +Key helpers in [docsStructure.ts](docsStructure.ts): + +- [`buildDirectoryNode`](docsStructure.ts#L112-L142) — recursively walks a directory and produces a node tree where each entry has a human-readable `label` (dashes/underscores turned into spaces) and a `directory` flag. Markdown filenames are added to a flat `mdFiles` list. +- [`collectDocsData`](docsStructure.ts#L443-L461) — full scan: returns both the manifest and the list of all `.md` paths. +- [`readDocsStructureManifest`](docsStructure.ts#L411-L441) — loads the existing JSON, falling back to an empty root node when the file is missing or malformed. +- [`mergeManifestWithSelectedPaths`](docsStructure.ts#L319-L382) — for each selected path: + - Normalizes the path, strips a leading `docs/`, rejects empty / ignored (`.gitbook`) entries. + - Validates the path stays inside `docs/` and points to a `.md` file or a directory. + - Builds a node from the filesystem ([`createNodeFromSelectedEntry`](docsStructure.ts#L212-L222)) and inserts it into the existing tree via [`insertSelectedNode`](docsStructure.ts#L224-L268), creating missing intermediate directory nodes on the fly and merging children via [`mergeNodes`](docsStructure.ts#L185-L210). + - For deletions, [`deleteSelectedNode`](docsStructure.ts#L270-L296) walks down the tree and removes the targeted child. +- Finally `JSON.stringify(manifest, null, 2)` is written to `docs-structure.json` with a trailing newline. + +#### Manifest shape + +```jsonc +{ + "version": 1, + "tree": { + "docs": { + "label": "docs", + "directory": true, + "children": { + "soluzioni": { + "label": "soluzioni", + "directory": true, + "children": { + "asilo-nido": { + "label": "asilo nido", + "directory": true, + "children": { + "README.md": { "label": "README", "directory": false } + } + } + } + } + } + } + } +} +``` + +### 3.3 Commit the manifest (if changed) + +```bash +if [ -n "$(git status --porcelain -- docs-structure.json)" ]; then + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + git add docs-structure.json + git commit -m "chore: update docs structure manifest" + git push +fi +``` + +The workflow only commits and pushes when `docs-structure.json` has actually changed, avoiding empty commits. The `contents: write` permission grants the necessary push access. + +### 3.4 Generate `crowdin.yml` + +```yaml +- name: Generate crowdin file + id: extract_files + env: + PATHS_TO_UPLOAD: ${{ inputs.paths_to_upload || '' }} + run: npm run generate_file +``` + +`npm run generate_file` runs [generateCrowdinConfig.ts](src/generateCrowdinConfig.ts): + +```mermaid +flowchart TD + A[Start] --> B{PATHS_TO_UPLOAD
provided?} + B -- Yes --> C[collectSelectedMarkdownFiles
expand dirs to .md leaves] + B -- No --> D[collectDocsData
all .md under docs/] + C --> E[Compose files list:
docs-structure.json
+ buildCrowdinFileEntries] + D --> E + E --> F[yaml.dump → crowdin.yml] + F --> G{GITHUB_OUTPUT set?} + G -- Yes --> H[Append found_files=JSON
step output] + G -- No --> I[Skip output
local run] +``` + +Highlights: + +- [`collectSelectedMarkdownFiles`](docsStructure.ts#L463-L505) walks each selected path: if it points to a directory it recurses (skipping `.gitbook`), if it points to a `.md` file it just adds it. Paths are normalized so users can pass `docs/foo/bar.md` or `foo/bar.md` interchangeably. +- [`buildCrowdinFileEntries`](docsStructure.ts#L507-L517) maps each source path to a translation path by injecting `%locale%` after `docs/`. The result looks like: + + ```yaml + - source: docs/soluzioni/asilo-nido/README.md + translation: docs/%locale%/soluzioni/asilo-nido/README.md + ``` + +- An extra entry is prepended unconditionally for the manifest itself: + + ```yaml + - source: docs-structure.json + translation: docs/%locale%/_meta/docs-structure.json + ``` + + This is what gives translators access to the readable `label` strings for each folder/file. + +- The whole config is serialized with `js-yaml` (`lineWidth: -1` to avoid line wrapping) into [crowdin.yml](crowdin.yml). +- When running on GitHub Actions, the script also appends `found_files=` to `$GITHUB_OUTPUT`, exposing it as `steps.extract_files.outputs.found_files` for downstream steps. Locally that env var is absent and the script just logs an info message. + +### 3.5 Upload to Crowdin + +```yaml +- uses: crowdin/github-action@v2 + with: + upload_sources: true + upload_translations: false + auto_approve_imported: true + download_translations: false + create_pull_request: false + base_url: 'https://pagopa.crowdin.com' + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }} + CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }} +``` + +The official Crowdin action reads the freshly generated `crowdin.yml` and: + +- Uploads every file listed under `files:` as a source string set. +- Auto-approves imported strings so existing translations get reused immediately. +- Does **not** download translations, open PRs, or push translations back in this workflow — that responsibility lives in a separate workflow. +- Authenticates against the PagoPA Crowdin instance using the two repository secrets. + +--- + +## 4. End-to-end data flow + +```mermaid +sequenceDiagram + participant U as User / Caller workflow + participant GA as GitHub Actions runner + participant DS as generateDocStructure.ts + participant FS as Filesystem (docs/) + participant G as Git remote + participant CC as generateCrowdinConfig.ts + participant CR as Crowdin + + U->>GA: Trigger (optionally with paths_to_upload / paths_to_delete) + GA->>GA: Checkout + Node + npm ci + GA->>DS: npm run generate_doc_structure + DS->>FS: Read docs/ (full or selected paths) + DS->>FS: Read existing docs-structure.json + DS->>FS: Write merged docs-structure.json + GA->>G: Commit + push (only if file changed) + GA->>CC: npm run generate_file + CC->>FS: Collect .md files (full or selected) + CC->>FS: Write crowdin.yml + CC->>GA: Set found_files step output + GA->>CR: crowdin-action uploads sources from crowdin.yml +``` + +--- + +## 5. Reference + +| Concern | Location | +| --- | --- | +| Workflow definition | [.github/workflows/upload_sources_to_crowdin.yml](.github/workflows/upload_sources_to_crowdin.yml) | +| Manifest entry point | [generateDocStructure.ts](src/generateDocStructure.ts) | +| Crowdin config entry point | [generateCrowdinConfig.ts](src/generateCrowdinConfig.ts) | +| Shared helpers (tree walk, merge, parsing) | [docsStructure.ts](src/docsStructure.ts) | +| Generated manifest | `docs-structure.json` | +| Generated Crowdin config | [crowdin.yml](crowdin.yml) | +| npm scripts | [package.json](package.json#L8-L11) |