Skip to content

Commit e9d19b4

Browse files
author
Thordata
committed
release: v1.8.3
1 parent 29e92ec commit e9d19b4

14 files changed

Lines changed: 787 additions & 707 deletions

‎.env.example‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -25,19 +25,18 @@ THORDATA_UNLIMITED_PASSWORD=
2525
# Optional: Other proxy products
2626
THORDATA_DATACENTER_USERNAME=
2727
THORDATA_DATACENTER_PASSWORD=
28+
2829
THORDATA_MOBILE_USERNAME=
2930
THORDATA_MOBILE_PASSWORD=
31+
3032
THORDATA_ISP_HOST=
3133
THORDATA_ISP_USERNAME=
3234
THORDATA_ISP_PASSWORD=
3335

3436
# --- Proxy Endpoint (Optional; defaults per product) ---
35-
# These apply when not overridden by THORDATA_<PRODUCT>_PROXY_* (e.g. THORDATA_RESIDENTIAL_PROXY_PORT).
36-
# Default ports by product: residential 9999, mobile 5555, datacenter 7777, isp 6666.
37-
# Protocol: http, https, socks5, or socks5h (remote DNS). http is most compatible.
3837
THORDATA_PROXY_HOST=
39-
THORDATA_PROXY_PORT=9999
40-
THORDATA_PROXY_PROTOCOL=http
38+
THORDATA_PROXY_PORT=
39+
THORDATA_PROXY_PROTOCOL=
4140

4241
# --- Scraping Browser (Optional) ---
4342
# If different from residential; used by get_browser_connection_url()

‎.github/workflows/integration-proxy.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Integration - Proxy Products (Overseas Runner)
2+
3+
on:
4+
workflow_dispatch:
5+
# Optional: run on PRs but this will require secrets to be available.
6+
# pull_request:
7+
8+
jobs:
9+
proxy-integration:
10+
runs-on: ubuntu-latest
11+
timeout-minutes: 25
12+
13+
steps:
14+
- name: Checkout
15+
uses: actions/checkout@v4
16+
17+
- name: Setup Python
18+
uses: actions/setup-python@v5
19+
with:
20+
python-version: "3.12"
21+
22+
- name: Install
23+
run: |
24+
python -m pip install -U pip
25+
python -m pip install -e .
26+
python -m pip install pytest
27+
28+
- name: Run proxy integration tests
29+
env:
30+
THORDATA_INTEGRATION: "true"
31+
32+
THORDATA_SCRAPER_TOKEN: ${{ secrets.THORDATA_SCRAPER_TOKEN }}
33+
THORDATA_PUBLIC_TOKEN: ${{ secrets.THORDATA_PUBLIC_TOKEN }}
34+
THORDATA_PUBLIC_KEY: ${{ secrets.THORDATA_PUBLIC_KEY }}
35+
36+
THORDATA_PROXY_HOST: ${{ secrets.THORDATA_PROXY_HOST }}
37+
THORDATA_PROXY_PORT: ${{ secrets.THORDATA_PROXY_PORT }}
38+
THORDATA_PROXY_PROTOCOL: ${{ secrets.THORDATA_PROXY_PROTOCOL }}
39+
40+
THORDATA_RESIDENTIAL_USERNAME: ${{ secrets.THORDATA_RESIDENTIAL_USERNAME }}
41+
THORDATA_RESIDENTIAL_PASSWORD: ${{ secrets.THORDATA_RESIDENTIAL_PASSWORD }}
42+
43+
THORDATA_MOBILE_USERNAME: ${{ secrets.THORDATA_MOBILE_USERNAME }}
44+
THORDATA_MOBILE_PASSWORD: ${{ secrets.THORDATA_MOBILE_PASSWORD }}
45+
46+
THORDATA_DATACENTER_USERNAME: ${{ secrets.THORDATA_DATACENTER_USERNAME }}
47+
THORDATA_DATACENTER_PASSWORD: ${{ secrets.THORDATA_DATACENTER_PASSWORD }}
48+
49+
THORDATA_ISP_HOST: ${{ secrets.THORDATA_ISP_HOST }}
50+
THORDATA_ISP_USERNAME: ${{ secrets.THORDATA_ISP_USERNAME }}
51+
THORDATA_ISP_PASSWORD: ${{ secrets.THORDATA_ISP_PASSWORD }}
52+
53+
run: |
54+
python -m pytest tests/test_integration_proxy_protocols.py -v -s --tb=short

‎ACCEPTANCE_COMPLETE.md‎

Lines changed: 0 additions & 88 deletions
This file was deleted.

‎CLEANUP_PLAN.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# Test and Example Files Cleanup Plan
2+
3+
## Analysis of Redundancy
4+
5+
### 1. Test Files Analysis
6+
7+
#### Redundant/Similar Test Files:
8+
1. **`test_examples.py`** vs **`test_client.py`** + **`test_async_client.py`**
9+
- `test_examples.py`: Tests example scripts execution (integration-style)
10+
- `test_client.py`/`test_async_client.py`: Unit tests for client methods
11+
- **Status**: NOT redundant - different purposes (integration vs unit tests)
12+
13+
2. **`test_tools.py`** vs **`test_tools_coverage.py`**
14+
- `test_tools.py`: Tests tool classes and serialization
15+
- `test_tools_coverage.py`: Tests all tool classes for contract compliance
16+
- **Status**: POTENTIALLY redundant - both test tool serialization
17+
- **Recommendation**: Merge into single comprehensive test file
18+
19+
3. **`test_client_errors.py`** vs **`test_async_client_errors.py`**
20+
- Both test error handling
21+
- **Status**: NOT redundant - sync vs async versions needed
22+
23+
4. **`test_task_status_and_wait.py`**
24+
- Tests task status and waiting logic
25+
- **Status**: Could be merged into `test_client.py` for better organization
26+
27+
### 2. Example Files Analysis
28+
29+
#### Redundant/Similar Example Files:
30+
1. **`quick_start.py`** vs **`full_acceptance_test.py`**
31+
- `quick_start.py`: Quick validation of core features
32+
- `full_acceptance_test.py`: Comprehensive acceptance test suite
33+
- **Status**: NOT redundant - different scopes (quick vs comprehensive)
34+
35+
2. **`demo_web_scraper_api.py`** vs **`demo_web_scraper_multi_spider.py`**
36+
- `demo_web_scraper_api.py`: Single spider workflow demo
37+
- `demo_web_scraper_multi_spider.py`: Multiple spiders test
38+
- **Status**: NOT redundant - different use cases
39+
40+
3. **`demo_universal.py`** vs **`quick_start.py`** (Universal section)
41+
- Both demonstrate Universal API
42+
- **Status**: MINOR redundancy - `demo_universal.py` is more focused
43+
- **Recommendation**: Keep both (focused demo vs quick start)
44+
45+
4. **`validate_env.py`**
46+
- Validates environment variables
47+
- **Status**: Unique utility, keep
48+
49+
5. **`diagnose_network.py`**
50+
- Network diagnostics utility
51+
- **Status**: Unique utility, keep
52+
53+
### 3. Integration Test Files
54+
55+
#### Important Integration Tests:
56+
1. **`test_integration_proxy_protocols.py`**
57+
- Tests proxy protocol connectivity (HTTPS, SOCKS5h)
58+
- **Status**: IMPORTANT - validates proxy network functionality
59+
- **Requires**: `THORDATA_INTEGRATION=true`, proxy credentials
60+
61+
2. **`test_tools_coverage.py`** (with integration flag)
62+
- Tests all tool classes with real API calls
63+
- **Status**: IMPORTANT - validates tool contracts
64+
- **Requires**: `THORDATA_INTEGRATION=true`
65+
66+
## Cleanup Recommendations
67+
68+
### High Priority (Remove/Merge)
69+
70+
1. **Merge `test_tools.py` into `test_tools_coverage.py`**
71+
- Both test tool serialization
72+
- `test_tools_coverage.py` is more comprehensive
73+
- **Action**: Merge functionality, remove `test_tools.py`
74+
75+
2. **Merge `test_task_status_and_wait.py` into `test_client.py`**
76+
- Task-related tests should be in main client test file
77+
- **Action**: Move tests, remove separate file
78+
79+
### Medium Priority (Review/Consolidate)
80+
81+
3. **Review example file organization**
82+
- Consider grouping by category:
83+
- `examples/basic/` - quick_start.py, validate_env.py
84+
- `examples/demos/` - demo_*.py files
85+
- `examples/tools/` - tool-specific examples (already exists)
86+
- **Action**: Reorganize for better discoverability
87+
88+
### Low Priority (Keep but Document)
89+
90+
4. **Keep all demo files**
91+
- Each serves a specific purpose
92+
- **Action**: Add clear docstrings explaining when to use each
93+
94+
## Integration Test Execution Plan
95+
96+
### Required Environment Variables:
97+
```bash
98+
THORDATA_INTEGRATION=true
99+
THORDATA_SCRAPER_TOKEN=...
100+
THORDATA_PUBLIC_TOKEN=...
101+
THORDATA_PUBLIC_KEY=...
102+
THORDATA_PROXY_HOST=pr.thordata.net
103+
THORDATA_RESIDENTIAL_USERNAME=...
104+
THORDATA_RESIDENTIAL_PASSWORD=...
105+
THORDATA_INTEGRATION_HTTP=true # Optional: test HTTP protocol too
106+
THORDATA_INTEGRATION_STRICT=true # Optional: fail on any error
107+
```
108+
109+
### Test Execution:
110+
1. Proxy Protocol Integration Test
111+
2. Tools Coverage Integration Test (if enabled)
112+
3. All other integration tests
113+
114+
## File Structure After Cleanup
115+
116+
```
117+
tests/
118+
├── test_client.py # Main sync client tests (includes task tests)
119+
├── test_async_client.py # Main async client tests
120+
├── test_client_errors.py # Sync error handling
121+
├── test_async_client_errors.py # Async error handling
122+
├── test_tools_coverage.py # All tool tests (merged from test_tools.py)
123+
├── test_integration_proxy_protocols.py # Proxy integration test
124+
├── test_examples.py # Example scripts integration test
125+
├── test_browser.py # Browser tests
126+
├── test_unlimited.py # Unlimited namespace tests
127+
├── test_batch_creation.py # Batch creation tests
128+
├── test_models.py # Model tests
129+
├── test_exceptions.py # Exception tests
130+
├── test_retry.py # Retry logic tests
131+
├── test_utils.py # Utility function tests
132+
├── test_enums.py # Enum tests
133+
├── test_env.py # Environment tests
134+
├── test_user_agent.py # User agent tests
135+
└── test_spec_parity.py # Spec parity tests
136+
137+
examples/
138+
├── quick_start.py # Quick validation
139+
├── full_acceptance_test.py # Comprehensive acceptance
140+
├── validate_env.py # Environment validation
141+
├── diagnose_network.py # Network diagnostics
142+
├── demo_serp_api.py # SERP demo
143+
├── demo_universal.py # Universal demo
144+
├── demo_web_scraper_api.py # Web Scraper single workflow
145+
├── demo_web_scraper_multi_spider.py # Multi-spider test
146+
├── demo_proxy_network.py # Proxy demo
147+
├── demo_browser_api.py # Browser demo
148+
├── demo_scraping_browser.py # Scraping browser demo
149+
├── demo_account_and_usage.py # Account demo
150+
├── async_high_concurrency.py # Async concurrency demo
151+
└── tools/ # Tool-specific examples
152+
├── amazon_scraper.py
153+
├── google_maps_scraper.py
154+
├── social_media_scraper.py
155+
└── youtube_downloader.py
156+
```
157+
158+
## Implementation Steps
159+
160+
1. **Phase 1: Merge Test Files**
161+
- Merge `test_tools.py` into `test_tools_coverage.py`
162+
- Merge `test_task_status_and_wait.py` into `test_client.py`
163+
- Run full test suite to ensure nothing breaks
164+
165+
2. **Phase 2: Run Integration Tests**
166+
- Set up environment variables
167+
- Run `test_integration_proxy_protocols.py`
168+
- Run `test_tools_coverage.py` with integration flag
169+
- Verify 100% pass rate
170+
171+
3. **Phase 3: Reorganize Examples (Optional)**
172+
- Group examples by category
173+
- Update documentation references
174+
175+
4. **Phase 4: Documentation**
176+
- Add clear docstrings to all example files
177+
- Update README with file organization
178+
179+
## Notes
180+
181+
- All integration tests are IMPORTANT for validating real-world functionality
182+
- Integration tests require real credentials and may consume quota
183+
- Integration tests should be run before major releases
184+
- Consider adding CI/CD integration test runs (with proper credential management)

‎README_INTEGRATION.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Integration Testing
2+
3+
## Quick Start
4+
5+
```bash
6+
# Set in .env
7+
THORDATA_INTEGRATION=true
8+
THORDATA_PROXY_HOST=vpn9wq0d.pr.thordata.net
9+
THORDATA_RESIDENTIAL_USERNAME=your_username
10+
THORDATA_RESIDENTIAL_PASSWORD=your_password
11+
THORDATA_UPSTREAM_PROXY=socks5://127.0.0.1:7898 # For TUN mode
12+
13+
# Run test
14+
pytest tests/test_integration_proxy_protocols.py -v
15+
```
16+
17+
## Protocol Support
18+
19+
SDK supports: **HTTP**, **HTTPS**, **SOCKS5**, **SOCKS5h**
20+
21+
See `PROXY_PROTOCOLS.md` for details.
22+
23+
## TUN Mode
24+
25+
With TUN mode (Clash Verge), the test:
26+
- Tests **HTTPS** protocol (most reliable)
27+
- Skips HTTP (often times out)
28+
- Skips SOCKS5h (TUN mode limitation)
29+
30+
This is expected behavior.

0 commit comments

Comments
 (0)