Refactoring Code

[AndyRae]

Is this the right issue type?

• Yes, I'm planning work for this project team.

Summary

To group refactoring issues.

Problems

1. tools/ Directory is for everything

   • 20 files with no organisation
   • Mixes domain logic, infrastructure, utilities, and application code
   • Makes codebase hard to navigate, maintain, develop
   • Given we don't support sqlite we have a lot of code for it

2. cli/ should be thin

   • should parse arguments and call services
   • currently processes significant business logic
   • presentation layer doing application work

3. No Clear Architecture

   • Missing standard Python package layers (domain, application, infrastructure)
   • Tight coupling between modules
   • Mixed abstraction levels
   • Configuration mixed with code

4. Poor Documentation

   • No architecture documentation
   • Minimal inline documentation

---

Structure

plausible structure ?

carrottransform/
├── domain/          # Business logic & models
│   ├── omop/       # OMOP CDM domain
│   ├── mapping/    # Mapping rules & concepts
│   └── person/     # Person domain logic
│
├── application/     # Application services
│   ├── orchestrator.py
│   ├── processor.py
│   └── record_builder.py
│
├── infrastructure/  # External concerns
│   ├── storage/    # File I/O, database, S3
│   └── sources/    # Data source adapters
│
└── shared/         # Shared utilities
    ├── utils/      # Pure utilities
    └── logging/    # Logging setup

Tests

should mirror new structure

tests/
    - unit/
        - domain/
        - application/
        - infrastructure/
    - integration/
    - fixtures/
    - data/
        - {test-case-name}/
            - source/ # .csv files
            - output/ # .tsv files
            - v1.json  # if available
            - v2.json # if available, maybe expect only one?

• don't test docker stuff
• move external dependencies (e.g db, to services)
• fix the markers
• document what we are actually testing and how any of this works

Confirm creation

• Ready to create this epic?

[CodeByKarthik]

Following our discussions about the architectural challenges, I've few ideas for extending on top of Andy’s foundation to address the core structural problems.

Architecture Overview

carrottransform/
├── domain/              # Business logic & models
│   ├── omop/            # OMOP CDM domain
│   ├── mapping/         # Mapping rules & concepts
│   ├── person/          # Person domain logic
│   ├── validators/      # Validation Business rules
│   └── interfaces/      # Package interfaces or abstraction
│
│
├── application/                  # Application services
│   ├── services/
│   ├── dto/                      # Data Transfer Objects
│   ├── interface/                # Package interfaces or abstraction
│
│
├── infrastructure/       # External concerns
│   ├── storage/          # File I/O, database, S3
│   └── sources/          # Data source adapters
│
├── cli/  
│   ├── commands/         # Command-line interface
│
│
├── config/               # Configuration management
│   ├── settings.py       # Runtime configuration
│   ├── schemas/  
│   └── defaults/
│       └── config.json
│
│
├── shared/                # Shared utilities
|   ├── utils/             # Pure utilities
|   ├── logging/           # Logging setup
|   ├── metrics/
|   ├── Types/             # Type definitions and hints
|   └── Exceptions/        # Centralized exception hierarchy
|
│
├── public/
		├── images             # images
        └── docs               # Public documentation

[CodeByKarthik]

Implementation Strategy

In terms of Implementation strategy, I have some plans on how to approach the refactor.

• Directory Structure

  • Follow the architecture layout (eg: Domain → Application → Infrastructure).
  • Maintain clear separation of concerns between layers.
  • Dependency rules (outer layers depend on inner but never the reverse).
  • Make sure each modules belong to their respective layers.

• Object Design

  • Keep classes small and single-purpose.
  • Avoid bloated classes which leads to Objects Explosion.
  • If a class grows too large, split by responsibilities.

• Dependency Injection

  • All the implementation will be based on dependency injection.
  • Dependencies are injected in the initialiser.

• Interfaces & Abstractions

  • Utilise interfaces or abstract within packages.
  • Define interfaces in Domain and Application layers.
  • Abstract for infrastructure is optional yet use it when necessary.

• Object Creation and Behaviours

  • Objects can be created using Factory or an Abstract Factory.
  • Some places requires strategy pattern.

• Object Structuring

  • Objects can be structured using adapter or facade or a combination of registry and facade.
  • Some Objects might require composite pattern to grow them individually.

• Lazy Initialisation

  • Object are created until needed to optimise CLI
  • Since Carrot Transform supports CLI tooling

• Will use type hinting, enums and custom exception errors

This approach aims to structure the codebase and decouple the existing modules and packages for more modular, scalable, maintainable, testable and efficient bug tracking architecture.