Skip to content

README.md

Latest commit

 

History

History
507 lines (342 loc) · 6.37 KB

README.md

File metadata and controls

507 lines (342 loc) · 6.37 KB

TestLink → BrowserStack Migration Framework

A scalable migration framework for migrating data from TestLink into BrowserStack Test Management.

Supports:

  • MySQL / MariaDB
  • Docker & On-Prem installations
  • Native DB backups
  • Attachment extraction
  • Config extraction
  • Canonical NDJSON exports
  • BrowserStack API imports
  • Resumable migrations

Problem Statement

Organizations using TestLink often need to migrate:

  • projects
  • test suites
  • test cases
  • executions
  • attachments
  • plans
  • custom fields

into BrowserStack Test Management.

However, TestLink data is distributed across:

  • relational databases
  • filesystem attachments
  • config files
  • execution history

This framework was designed to handle large-scale migrations safely and efficiently.

Architecture

The system follows a two-phase hybrid migration architecture.

Architecture Diagram

Phase 1 — Extraction & Snapshot

Extracts:

  • native DB backup
  • canonical NDJSON exports
  • attachments
  • config files

Outputs a reusable migration bundle.

Phase 2 — BrowserStack Import

Reads migration bundle and imports:

  • projects
  • folders
  • testcases
  • plans
  • results
  • attachments

using BrowserStack APIs.

Why Hybrid Architecture?

The migration system combines:

  • native DB backups
  • NDJSON canonical exports
  • zipped asset bundles
  • decoupled API importer

This provides:

  • resumability
  • scalability
  • portability
  • low memory usage
  • failure recovery
  • large dataset support

Folder Structure

.
├── browserstack/
├── extractor/
├── orchestrator/
├── utils/
├── mappings/
├── output/
├── requirements.txt
└── index.py

Core Components

index.py

Main entrypoint.

Starts complete migration flow.

orchestrator/runner.py

Master orchestration layer.

Handles:

  • user prompts
  • DB connection
  • extraction
  • migration flow

orchestrator/extraction.py

Canonical entity exporter.

Exports:

  • projects
  • suites
  • testcases
  • plans
  • executions
  • custom fields

into NDJSON format.

orchestrator/migration.py

BrowserStack import coordinator.

Handles:

  • project import
  • folder import
  • testcase import
  • plan import
  • results import
  • attachment upload

Database Extraction

Supported DBs

  • MySQL
  • MariaDB
  • PostgreSQL (planned)
  • MSSQL (planned)

Native Backup Commands

MySQL / MariaDB

mysqldump \
  -h localhost \
  -P 3306 \
  -u testlink \
  -p testlink \
  --single-transaction \
  --quick \
  --routines \
  --triggers \
  --events \
  > backup.sql

Compressed:

mysqldump ... | gzip > backup.sql.gz

PostgreSQL

pg_dump \
  -h localhost \
  -U postgres \
  -d testlink \
  -Fc \
  -f backup.dump

MSSQL

sqlcmd -Q "
BACKUP DATABASE testlink
TO DISK='backup.bak'
"

Config Extraction

Extracts:

  • config_db.inc.php
  • custom_config.inc.php

Docker example:

docker cp \
  testlink:/var/www/html/testlink/config_db.inc.php .

Compressed into:

configs.zip

Attachment Extraction

Extracts assets from:

upload_area/

Supports:

  • Docker
  • on-prem filesystem
  • future S3 support

Docker example:

docker cp \
  testlink:/var/testlink/upload_area .

Compressed into:

attachments.zip

Canonical NDJSON Export

The framework exports entities into:

NDJSON

Example:

{"id":1,"name":"Login Test"}
{"id":2,"name":"Checkout Test"}

Benefits:

  • streaming friendly
  • low memory usage
  • resumable imports
  • chunk processing
  • scalable for large datasets

BrowserStack Import Layer

Uses BrowserStack Test Management APIs for importing:

  • projects
  • folders
  • testcases
  • plans
  • results
  • attachments

Mappings are maintained for:

TestLink ID -> BrowserStack ID

Mapping Files

Generated under:

mappings/

Examples:

  • projects.json
  • testcases.json
  • attachments.json

Used for:

  • resumability
  • attachment linking
  • execution linking

Current Simplification Strategy

Instead of recreating exact TestLink hierarchy, all suites are currently mapped into a single BrowserStack root folder:

TestLink Migrated

This improves:

  • migration reliability
  • API compatibility
  • large dataset handling

while preserving original metadata.

Limitations

Current limitations include:

  • simplified suite hierarchy
  • partial BrowserStack API restrictions
  • PostgreSQL/MSSQL extractors still WIP
  • attachment linking currently focused on testcase entities

Explored Approaches

1. Direct DB → API

Rejected because:

  • tightly coupled
  • difficult retries
  • repeated DB access

2. CSV/XML Export

Rejected because:

  • weak hierarchy support
  • attachment loss
  • limited resumability

3. Native .bak Only

Rejected because:

  • BrowserStack APIs still require transformed entities

4. Final Chosen Approach

Hybrid architecture combining:

  • native backup
  • canonical NDJSON
  • zipped assets
  • decoupled importer

provided the best balance.

Strengths of Current System

  • scalable
  • resumable
  • Docker compatible
  • low memory usage
  • attachment aware
  • API-driven
  • mapping based
  • reusable migration bundles
  • production-safe extraction flow

Future Improvements

Planned enhancements:

  • PostgreSQL extractor
  • MSSQL extractor
  • S3 attachment support
  • rate limiting
  • checkpoint recovery
  • parallel imports
  • exact suite hierarchy recreation
  • incremental migrations

Setup

Create and activate a virtual environment:

python3 -m venv venv
source venv/bin/activate

Install Python dependencies:

pip3 install -r requirements.txt

Install the MySQL client:

brew install mysql-client

Run migration:

python3 index.py

Output Structure

output/
├── attachments/
├── configs/
├── db_backup/
├── entities/
└── logs/

Author Notes

This project evolved from a simple DB backup script into a scalable migration framework by progressively solving:

  • DB portability
  • attachment handling
  • hierarchy mapping
  • resumability
  • API integration
  • large dataset processing

The final architecture separates:

  • extraction
  • transformation
  • transport
  • import
  • mapping

into independent layers for better maintainability and operational safety.