Feature/test directory structure (#488)

VENKAT-AVVARI-190825 · web-flow · commit 055e4ae9862d · 2026-02-06T12:18:03.000Z
* Add unit and integration test structure for phase_polars - Created test directory structure mirroring phase_polars folder - Added empty test files for all load and transform phases - Unit tests: tests/unit/phase_polars/load/ and tests/unit/phase_polars/transform/ - Integration tests: tests/integration/phase_polars/load/ and tests/integration/phase_polars/transform/ - Ready for implementation once phase_polars code is developed * Update test files to empty placeholder structure - Simplified test files to contain only comment headers - Removed detailed test implementations as phase_polars code not yet implemented - Updated .gitignore to exclude virtual environments - Ready for actual test implementation once phase_polars modules are developed * Add comprehensive testing documentation and guidelines Fixes #479 - Created TESTING.md with detailed test structure documentation - Added directory structure explanation mirroring phase_polars organization - Documented test organization principles and naming conventions - Included running instructions for different test scenarios - Updated README.md with testing section and quick commands - Established clear separation between unit, integration, and e2e tests - Documented current implementation status and future considerations
diff --git a/.gitignore b/.gitignore
@@ -14,6 +14,7 @@ demodata/
 .eggs
 *.gfs
 .venv
+venv*/
 .direnv
 var/cache
 /collection
diff --git a/README.md b/README.md
@@ -52,6 +52,23 @@ Development requires Python 3.6.2 or later, we recommend using a [virtual enviro
     make
     python -m digital-land --help
 
+## Testing
+
+This repository follows a structured testing approach. See [TESTING.md](TESTING.md) for detailed testing guidelines and structure documentation.
+
+### Quick Test Commands
+
+```bash
+# Run all tests
+pytest
+
+# Run phase_polars tests
+pytest tests/unit/phase_polars/ tests/integration/phase_polars/
+
+# Run with coverage
+pytest --cov=digital_land
+```
+
 
 ## Commands Guide
 
diff --git a/TESTING.md b/TESTING.md
@@ -0,0 +1,94 @@
+# Testing Guidelines
+
+## Test Structure
+
+This repository follows a structured testing approach that mirrors the codebase organization to ensure maintainability and clarity.
+
+### Directory Structure
+
+```
+tests/
+├── unit/                    # Unit tests for individual components
+│   ├── phase/              # Tests for legacy phase modules
+│   └── phase_polars/       # Tests for new phase_polars framework
+│       ├── load/           # Load phase unit tests
+│       └── transform/      # Transform phase unit tests
+├── integration/            # Integration tests for component interactions
+│   ├── phase/              # Legacy phase integration tests
+│   └── phase_polars/       # Phase_polars integration tests
+│       ├── load/           # Load phase integration tests
+│       └── transform/      # Transform phase integration tests
+├── e2e/                    # End-to-end tests
+├── acceptance/             # Acceptance tests
+└── performance/            # Performance tests
+```
+
+### Test Organization Principles
+
+1. **Mirror Source Structure**: Test directories should mirror the source code structure
+2. **Separation of Concerns**: Unit, integration, and e2e tests are clearly separated
+3. **Framework Isolation**: Legacy `phase/` and new `phase_polars/` tests are isolated
+
+### Phase_polars Test Structure
+
+The `phase_polars` test structure follows the modular design of the new ETL framework:
+
+#### Unit Tests (`tests/unit/phase_polars/`)
+- **Load Tests**: `load/test_save_database.py`, `load/test_save_file.py`
+- **Transform Tests**: Individual test files for each transform phase:
+  - `transform/test_convert.py`
+  - `transform/test_normalise.py`
+  - `transform/test_filter.py`
+  - `transform/test_validate.py`
+  - And 14 additional transform phase test files
+
+#### Integration Tests (`tests/integration/phase_polars/`)
+- **Load Integration**: `load/test_load_integration.py`
+- **Transform Integration**: `transform/test_transform_integration.py`
+- **Individual Phase Integration**: Separate files for each transform phase
+
+### Naming Conventions
+
+- Test files: `test_<module_name>.py`
+- Test classes: `Test<ClassName>`
+- Test methods: `test_<functionality>`
+- Integration test files: `test_<component>_integration.py`
+
+### Running Tests
+
+```bash
+# Run all tests
+pytest
+
+# Run unit tests only
+pytest tests/unit/
+
+# Run phase_polars tests only
+pytest tests/unit/phase_polars/ tests/integration/phase_polars/
+
+# Run specific test file
+pytest tests/unit/phase_polars/transform/test_convert.py
+
+# Run with verbose output
+pytest -v tests/unit/phase_polars/
+```
+
+### Test Implementation Status
+
+Currently, the phase_polars test files contain placeholder structures ready for implementation once the corresponding phase modules are developed. This ensures:
+
+1. **Consistent Structure**: All test files follow the same organizational pattern
+2. **Ready for Development**: Test structure is prepared for parallel development
+3. **Clear Separation**: Unit and integration tests are clearly distinguished
+
+### Dependencies
+
+- **pytest**: Primary testing framework
+- **pytest-mock**: For mocking dependencies
+- **coverage**: For test coverage reporting
+
+### Future Considerations
+
+- Tests will be implemented as phase_polars modules are developed
+- Integration tests will validate data flow between phases
+- Performance tests may be added for large dataset processing
diff --git a/tests/integration/phase_polars/__init__.py b/tests/integration/phase_polars/__init__.py
diff --git a/tests/integration/phase_polars/load/__init__.py b/tests/integration/phase_polars/load/__init__.py
diff --git a/tests/integration/phase_polars/load/test_load_integration.py b/tests/integration/phase_polars/load/test_load_integration.py
@@ -0,0 +1 @@
+# Integration tests for load phases
diff --git a/tests/integration/phase_polars/transform/__init__.py b/tests/integration/phase_polars/transform/__init__.py
diff --git a/tests/integration/phase_polars/transform/test_concat_field.py b/tests/integration/phase_polars/transform/test_concat_field.py
@@ -0,0 +1 @@
+# Integration tests for concat_field transform phase
diff --git a/tests/integration/phase_polars/transform/test_entity_lookup.py b/tests/integration/phase_polars/transform/test_entity_lookup.py
@@ -0,0 +1 @@
+# Integration tests for entity_lookup transform phase
diff --git a/tests/integration/phase_polars/transform/test_entity_reference.py b/tests/integration/phase_polars/transform/test_entity_reference.py
@@ -0,0 +1 @@
+# Integration tests for entity_reference transform phase
diff --git a/tests/integration/phase_polars/transform/test_fact_hash.py b/tests/integration/phase_polars/transform/test_fact_hash.py
@@ -0,0 +1 @@
+# Integration tests for fact_hash transform phase
diff --git a/tests/integration/phase_polars/transform/test_field_prune.py b/tests/integration/phase_polars/transform/test_field_prune.py
@@ -0,0 +1 @@
+# Integration tests for field_prune transform phase
diff --git a/tests/integration/phase_polars/transform/test_flatten.py b/tests/integration/phase_polars/transform/test_flatten.py
@@ -0,0 +1 @@
+# Integration tests for flatten transform phase
diff --git a/tests/integration/phase_polars/transform/test_map.py b/tests/integration/phase_polars/transform/test_map.py
@@ -0,0 +1 @@
+# Integration tests for map transform phase
diff --git a/tests/integration/phase_polars/transform/test_migrate.py b/tests/integration/phase_polars/transform/test_migrate.py
@@ -0,0 +1 @@
+# Integration tests for migrate transform phase
diff --git a/tests/integration/phase_polars/transform/test_parse.py b/tests/integration/phase_polars/transform/test_parse.py
@@ -0,0 +1 @@
+# Integration tests for parse transform phase
diff --git a/tests/integration/phase_polars/transform/test_patch.py b/tests/integration/phase_polars/transform/test_patch.py
@@ -0,0 +1 @@
+# Integration tests for patch transform phase
diff --git a/tests/integration/phase_polars/transform/test_pivot.py b/tests/integration/phase_polars/transform/test_pivot.py
@@ -0,0 +1 @@
+# Integration tests for pivot transform phase
diff --git a/tests/integration/phase_polars/transform/test_priority.py b/tests/integration/phase_polars/transform/test_priority.py
@@ -0,0 +1 @@
+# Integration tests for priority transform phase
diff --git a/tests/integration/phase_polars/transform/test_resolve_organisation.py b/tests/integration/phase_polars/transform/test_resolve_organisation.py
@@ -0,0 +1 @@
+# Integration tests for resolve_organisation transform phase
diff --git a/tests/integration/phase_polars/transform/test_set_default.py b/tests/integration/phase_polars/transform/test_set_default.py
@@ -0,0 +1 @@
+# Integration tests for set_default transform phase
diff --git a/tests/integration/phase_polars/transform/test_transform_integration.py b/tests/integration/phase_polars/transform/test_transform_integration.py
@@ -0,0 +1 @@
+# Integration tests for transform phases
diff --git a/tests/unit/phase_polars/__init__.py b/tests/unit/phase_polars/__init__.py
diff --git a/tests/unit/phase_polars/load/__init__.py b/tests/unit/phase_polars/load/__init__.py
diff --git a/tests/unit/phase_polars/load/test_save_database.py b/tests/unit/phase_polars/load/test_save_database.py
@@ -0,0 +1 @@
+# Unit tests for save_database phase
diff --git a/tests/unit/phase_polars/load/test_save_file.py b/tests/unit/phase_polars/load/test_save_file.py
@@ -0,0 +1 @@
+# Unit tests for save_file phase
diff --git a/tests/unit/phase_polars/transform/__init__.py b/tests/unit/phase_polars/transform/__init__.py
diff --git a/tests/unit/phase_polars/transform/test_concat_field.py b/tests/unit/phase_polars/transform/test_concat_field.py
@@ -0,0 +1 @@
+# Unit tests for concat_field transform phase
diff --git a/tests/unit/phase_polars/transform/test_convert.py b/tests/unit/phase_polars/transform/test_convert.py
@@ -0,0 +1 @@
+# Unit tests for convert transform phase
diff --git a/tests/unit/phase_polars/transform/test_entity_lookup.py b/tests/unit/phase_polars/transform/test_entity_lookup.py
@@ -0,0 +1 @@
+# Unit tests for entity_lookup transform phase
diff --git a/tests/unit/phase_polars/transform/test_entity_reference.py b/tests/unit/phase_polars/transform/test_entity_reference.py
@@ -0,0 +1 @@
+# Unit tests for entity_reference transform phase
diff --git a/tests/unit/phase_polars/transform/test_fact_hash.py b/tests/unit/phase_polars/transform/test_fact_hash.py
@@ -0,0 +1 @@
+# Unit tests for fact_hash transform phase
diff --git a/tests/unit/phase_polars/transform/test_field_prune.py b/tests/unit/phase_polars/transform/test_field_prune.py
@@ -0,0 +1 @@
+# Unit tests for field_prune transform phase
diff --git a/tests/unit/phase_polars/transform/test_filter.py b/tests/unit/phase_polars/transform/test_filter.py
@@ -0,0 +1 @@
+# Unit tests for filter transform phase
diff --git a/tests/unit/phase_polars/transform/test_flatten.py b/tests/unit/phase_polars/transform/test_flatten.py
@@ -0,0 +1 @@
+# Unit tests for flatten transform phase
diff --git a/tests/unit/phase_polars/transform/test_map.py b/tests/unit/phase_polars/transform/test_map.py
@@ -0,0 +1 @@
+# Unit tests for map transform phase
diff --git a/tests/unit/phase_polars/transform/test_migrate.py b/tests/unit/phase_polars/transform/test_migrate.py
@@ -0,0 +1 @@
+# Unit tests for migrate transform phase
diff --git a/tests/unit/phase_polars/transform/test_normalise.py b/tests/unit/phase_polars/transform/test_normalise.py
@@ -0,0 +1 @@
+# Unit tests for normalise transform phase
diff --git a/tests/unit/phase_polars/transform/test_parse.py b/tests/unit/phase_polars/transform/test_parse.py
@@ -0,0 +1 @@
+# Unit tests for parse transform phase
diff --git a/tests/unit/phase_polars/transform/test_patch.py b/tests/unit/phase_polars/transform/test_patch.py
@@ -0,0 +1 @@
+# Unit tests for patch transform phase
diff --git a/tests/unit/phase_polars/transform/test_pivot.py b/tests/unit/phase_polars/transform/test_pivot.py
@@ -0,0 +1 @@
+# Unit tests for pivot transform phase
diff --git a/tests/unit/phase_polars/transform/test_priority.py b/tests/unit/phase_polars/transform/test_priority.py
@@ -0,0 +1 @@
+# Unit tests for priority transform phase
diff --git a/tests/unit/phase_polars/transform/test_resolve_organisation.py b/tests/unit/phase_polars/transform/test_resolve_organisation.py
@@ -0,0 +1 @@
+# Unit tests for resolve_organisation transform phase
diff --git a/tests/unit/phase_polars/transform/test_set_default.py b/tests/unit/phase_polars/transform/test_set_default.py
@@ -0,0 +1 @@
+# Unit tests for set_default transform phase
diff --git a/tests/unit/phase_polars/transform/test_validate.py b/tests/unit/phase_polars/transform/test_validate.py
@@ -0,0 +1 @@
+# Unit tests for validate transform phase

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+# Integration tests for concat_field transform phase`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+# Integration tests for entity_lookup transform phase`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+# Integration tests for entity_reference transform phase`