Skip to content

Commit 18cbddf

Browse files
nadeem4nadeem4
committed
feat(docs): Revise contributing guide and enhance API documentation structure
- Updated the CONTRIBUTING.md file to improve clarity on project setup, testing, and contribution workflow. - Reorganized mkdocs.yml to better structure API documentation, adding detailed sections for Core and REST APIs. - Introduced new documentation files for various API components, including Datasource, LLM, Indexing, and Query APIs. - Enhanced README.md with installation instructions and versioning policy for better user onboarding. - Removed outdated getting-started.md and integrated relevant content into the new documentation structure. - Added configuration documentation for datasources and LLMs to facilitate user setup.
1 parent 2c9d256 commit 18cbddf

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

65 files changed

+2071
-1274
lines changed

‎.github/workflows/publish_pypi.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,21 @@ jobs:
2727
- name: Build Adapter SDK
2828
run: python -m build packages/adapter-sdk
2929

30+
- name: Build Adapter SQLAlchemy
31+
run: python -m build packages/adapter-sqlalchemy
32+
33+
- name: Build Adapter MSSQL
34+
run: python -m build packages/adapters/mssql
35+
36+
- name: Build Adapter MySQL
37+
run: python -m build packages/adapters/mysql
38+
39+
- name: Build Adapter Postgres
40+
run: python -m build packages/adapters/postgres
41+
42+
- name: Build Adapter SQLite
43+
run: python -m build packages/adapters/sqlite
44+
3045
- name: Build Core
3146
run: python -m build packages/core
3247

‎CONTRIBUTING.md‎

Lines changed: 58 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -1,94 +1,102 @@
11
# Contributing to NL2SQL
22

3-
Welcome to the `nl2sql` monorepo! This project is a production-grade Natural Language to SQL engine.
3+
Thanks for contributing to the `nl2sql` monorepo. This guide covers local setup,
4+
tests, documentation, and adapter development.
45

5-
## 🏗️ Monorepo Structure
6+
## Monorepo layout
67

7-
* `packages/core`: The main query engine (LangGraph, core Logic).
8-
* `packages/adapter-sdk`: The interface definition for database adapters.
9-
* `packages/adapters/*`: Implementation of database drivers (Postgres, MSSQL, MySQL, etc.).
8+
- `packages/core`: Core engine and pipeline.
9+
- `packages/api`: FastAPI REST service.
10+
- `packages/adapter-sdk`: Adapter interfaces and contracts.
11+
- `packages/adapter-sqlalchemy`: SQLAlchemy adapter base.
12+
- `packages/adapters/*`: Database adapter implementations.
13+
- `docs/`: MkDocs documentation.
1014

11-
## 🚀 Getting Started
15+
## Prerequisites
1216

13-
### Prerequisites
17+
- Python 3.10+
18+
- Docker (required for integration tests that spin up databases)
1419

15-
* Python 3.10+
16-
* Docker & Docker Compose (for integration tests)
20+
## Local setup
1721

18-
### Installation
22+
1. Clone the repository.
23+
2. Create and activate a virtual environment.
24+
3. Install editable packages you plan to work on.
1925

20-
1. **Clone the repo**:
26+
Example (PowerShell):
2127

22-
```bash
23-
git clone https://github.com/nadeem4/nl2sql.git
24-
cd nl2sql
25-
```
26-
27-
2. **Create a Virtual Environment**:
28-
29-
```bash
30-
python -m venv venv
31-
.\venv\Scripts\activate # Windows
32-
source venv/bin/activate # Linux/Mac
33-
```
34-
35-
3. **Install Editable Packages**:
36-
37-
```bash
38-
pip install -r requirements.txt
39-
```
28+
```powershell
29+
python -m venv venv
30+
.\venv\Scripts\activate
31+
python -m pip install -e packages/adapter-sdk
32+
python -m pip install -e packages/core
33+
python -m pip install -e packages/adapter-sqlalchemy
34+
python -m pip install -e packages/adapters/postgres
35+
```
4036

41-
## 🧪 Testing
37+
## Running tests
4238

43-
### Running Unit Tests
39+
Unit tests:
4440

4541
```bash
4642
pytest packages/core/tests/unit
4743
```
4844

49-
### Running Integration Tests
50-
51-
This requires Docker. It will spin up REAL databases and run the Compliance Suite against them.
45+
Integration tests (requires Docker):
5246

5347
```powershell
5448
./scripts/test_integration.ps1
5549
```
5650

57-
## 🤝 Contribution Workflow
51+
## Documentation
5852

59-
1. Create a branch `feat/your-feature`.
60-
2. Make changes in the relevant package.
61-
3. **Run Tests** to ensure no regression.
62-
4. Submit a Pull Request.
53+
Docs are built with MkDocs. To run locally:
6354

64-
## 🧩 Creating a New Adapter
55+
```bash
56+
python -m pip install -r requirements-docs.txt
57+
mkdocs serve
58+
```
59+
60+
## Contribution workflow
61+
62+
1. Create a feature branch (e.g., `feat/my-change`).
63+
2. Make changes and run relevant tests.
64+
3. Open a pull request with a clear summary and test plan.
6565

66-
We have two base classes for adapters. Choose the right one to keep dependencies minimal:
66+
## Creating a new adapter
6767

68-
| Base Class | Package | Use Case | Dependencies |
69-
| :--- | :--- | :--- | :--- |
70-
| `BaseSQLAlchemyAdapter` | `adapter-sqlalchemy` | **Relational Databases** (Postgres, Snowflake, Oracle, etc.) | High (`SQLAlchemy`) |
71-
| `DatasourceAdapter` (Protocol) | `adapter-sdk` | **Everything Else** (REST APIs, Mongo, CSV, GraphDBs) | None |
68+
Choose the base class that matches your datasource:
7269

73-
### 1. SQL Database Adapter
70+
| Base | Package | Use Case | Dependencies |
71+
| --- | --- | --- | --- |
72+
| `BaseSQLAlchemyAdapter` | `adapter-sqlalchemy` | Relational databases | SQLAlchemy |
73+
| `DatasourceAdapter` (protocol) | `adapter-sdk` | Non-SQL or custom sources | None |
7474

75-
Inherit from `BaseSQLAlchemyAdapter`. It handles `fetch_schema` and `execute` for you.
75+
### SQL adapter
76+
77+
Implement `BaseSQLAlchemyAdapter` to inherit schema fetch and execution.
7678

7779
```python
7880
from nl2sql_sqlalchemy_adapter import BaseSQLAlchemyAdapter
7981

8082
class MyDbAdapter(BaseSQLAlchemyAdapter):
8183
def connect(self, config):
82-
# ... implementation ...
84+
...
8385
```
8486

85-
### 2. General Adapter (API/NoSQL)
87+
### Non-SQL adapter
8688

8789
Implement the `DatasourceAdapter` protocol directly.
8890

8991
```python
9092
from nl2sql_adapter_sdk import DatasourceAdapter
9193

9294
class MyApiAdapter(DatasourceAdapter):
93-
# You MUST implement fetch_schema, execute, etc. yourself
95+
...
9496
```
97+
98+
## Where to look
99+
100+
- Architecture and system behavior: `docs/architecture/`
101+
- Core API reference: `docs/api/core/`
102+
- REST API reference: `docs/api/rest/`

‎README.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,19 @@ flowchart TD
9999

100100
### 1. Installation
101101

102+
```bash
103+
# Install core only
104+
pip install nl2sql-core
105+
106+
# Install core with selected adapters
107+
pip install nl2sql-core[mysql,mssql]
108+
109+
# Install core with all adapters
110+
pip install nl2sql-core[all]
111+
```
112+
113+
For local development:
114+
102115
```bash
103116
git clone https://github.com/nadeem4/nl2sql.git
104117
cd nl2sql
@@ -124,6 +137,12 @@ result = run_with_graph(ctx, "Top 5 customers by revenue last quarter?")
124137
print(result.get("final_answer"))
125138
```
126139

140+
## 🔖 Versioning Policy
141+
142+
NL2SQL uses unified versioning across the monorepo. Core, adapters, API, and CLI
143+
share the same version number and are released together. Internal dependencies
144+
are pinned to the same version to avoid mismatches.
145+
127146
## 📚 Documentation
128147

129148
- **[System Architecture](docs/architecture/high-level.md)**: runtime topology and core flows

‎docs/api/core/auth.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Auth API
2+
3+
## Purpose
4+
Enforce role-based access control (RBAC) for datasource and table access.
5+
6+
## Responsibilities
7+
- Check permissions for a user-context against policy rules.
8+
- Return allowed datasources and tables for a user-context.
9+
10+
## Key Modules
11+
- `packages/core/src/nl2sql/api/auth_api.py`
12+
- `packages/core/src/nl2sql/auth/models.py`
13+
- `packages/core/src/nl2sql/auth/rbac.py`
14+
15+
## Public Surface
16+
17+
### UserContext
18+
19+
Source:
20+
`packages/core/src/nl2sql/auth/models.py`
21+
22+
Fields:
23+
| name | type | required | meaning |
24+
| --- | --- | --- | --- |
25+
| `user_id` | `Optional[str]` | no | Unique identifier for the user. |
26+
| `tenant_id` | `Optional[str]` | no | Tenant/organization identifier. |
27+
| `roles` | `List[str]` | no | Assigned RBAC roles. |
28+
29+
### RolePolicy
30+
31+
Source:
32+
`packages/core/src/nl2sql/auth/models.py`
33+
34+
Fields:
35+
| name | type | required | meaning |
36+
| --- | --- | --- | --- |
37+
| `description` | `str` | yes | Human-readable role description. |
38+
| `role` | `str` | yes | Role ID for auditing/logging. |
39+
| `allowed_datasources` | `List[str]` | no | Allowed datasource IDs or `*`. |
40+
| `allowed_tables` | `List[str]` | no | Allowed tables in `datasource.table` format. |
41+
42+
### AuthAPI.check_permissions
43+
44+
Source:
45+
`packages/core/src/nl2sql/api/auth_api.py`
46+
47+
Signature:
48+
`check_permissions(user_context: UserContext, datasource_id: str, table: str) -> bool`
49+
50+
Parameters:
51+
| name | type | required | meaning |
52+
| --- | --- | --- | --- |
53+
| `user_context` | `UserContext` | yes | User identity + roles. |
54+
| `datasource_id` | `str` | yes | Datasource ID being accessed. |
55+
| `table` | `str` | yes | Table name (`datasource.table` format enforced by policy). |
56+
57+
Returns:
58+
`bool` indicating whether access is allowed.
59+
60+
Raises:
61+
None directly. Policy validation errors can surface at load time.
62+
63+
Side Effects:
64+
None.
65+
66+
Idempotency:
67+
Yes.
68+
69+
### AuthAPI.get_allowed_resources
70+
71+
Source:
72+
`packages/core/src/nl2sql/api/auth_api.py`
73+
74+
Signature:
75+
`get_allowed_resources(user_context: UserContext) -> dict`
76+
77+
Parameters:
78+
| name | type | required | meaning |
79+
| --- | --- | --- | --- |
80+
| `user_context` | `UserContext` | yes | User identity + roles. |
81+
82+
Returns:
83+
`dict` with keys `datasources` and `tables`.
84+
85+
Raises:
86+
None.
87+
88+
Side Effects:
89+
None.
90+
91+
Idempotency:
92+
Yes.
93+
94+
## Behavioral Contracts
95+
- Policies enforce table namespacing (`datasource.table` or `datasource.*`).
96+
- Unknown roles yield empty permissions.

0 commit comments

Comments
 (0)