Investigate data availability: Cooperative AI Foundation

[PipFoweraker]

Source

https://www.cooperativeai.com/foundation

Objective

Research and document what funding data can be extracted from Cooperative AI Foundation.

Tasks

• Explore the website structure and available data
• Identify accessible funding information (grants, amounts, dates, recipients)
• Document data format and structure
• Assess feasibility of automated data extraction
• Note any API availability or terms of use
• Create sample data extract if possible

Deliverables

• Documentation of available data fields
• Assessment of extraction method (manual, scraping, API)
• Sample data if available

[PipFoweraker]

Investigation Plan: Funding Data Source Research

🎯 Investigation Objectives

Determine for this funding source:

1. What data is available? (grants, amounts, dates, recipients, descriptions)
2. How can we access it? (API, web scraping, manual download)
3. What format is it in? (JSON, CSV, HTML, PDF)
4. How complete is it? (historical depth, update frequency)
5. Any restrictions? (terms of use, rate limits, auth)
6. Extraction method? (automated vs manual)

---

📋 Investigation Checklist

1. Initial Reconnaissance (30 min)

• Explore website and available pages
• Look for grant database/portfolio pages
• Check for public API or downloadable datasets
• Review Terms of Service / robots.txt
• Note auth requirements

2. Data Inventory (45 min)

• List all available data fields
• Note data format for each field
• Check date range (oldest/newest)
• Estimate total record count
• Identify unique identifiers
• Check for related data

3. Technical Assessment (45 min)

• Inspect page source / DevTools
• Check for structured data (JSON-LD)
• Look for AJAX/API calls in Network tab
• Test pagination
• Check for anti-scraping measures
• Assess JavaScript requirements

4. Sample Extraction (1 hour)

• Extract 5-10 sample records
• Document extraction method
• Save in standardized format
• Note extraction difficulties
• Time the process
• Assess repeatability

5. Documentation (30 min)

• Complete investigation report
• Create sample data file
• Document extraction process
• List blockers/issues
• Recommend strategy

Total Time: ~3-4 hours

---

📝 Investigation Report Template

Create: data/raw/funding_sources/[source_name]/INVESTIGATION_REPORT.md

Sections to complete:

Summary

2-3 sentence overview and recommendation

Data Availability

• Table of available fields with completeness
• Date range coverage
• Total record estimate
• Update frequency

Technical Details

• Access method (public/auth/API/download)
• Data format (HTML/JSON/CSV/PDF)
• Technical characteristics (JS required, pagination, rate limits)
• Example URLs

Extraction Strategy

• Recommended method (automated scraping/manual/API/hybrid)
• Implementation approach (pseudocode)
• Estimated extraction time
• Required dependencies

Sample Data

• 5-10 records in JSON format
• Saved to: data/raw/funding_sources/[source]/manual/sample_YYYYMMDD.json

Challenges & Limitations

• Identified issues
• Data quality concerns
• Legal/ethical considerations
• Technical blockers

Next Steps

• Immediate actions
• Questions to resolve
• Ready for implementation? (Yes/No/Caveats)

---

🛠️ Investigation Tools

Quick Site Inspector

python scripts/investigation/inspect_site.py [URL]

Checks for:

• Structured data (JSON-LD)
• Data tables
• API hints
• Download links
• robots.txt rules

Sample Extractor

python scripts/investigation/extract_sample.py [URL] [source_name] [output_dir]

Extracts small sample and saves to JSON.

---

📊 Data Fields to Look For

Standard funding event fields:

• Grant ID / unique identifier
• Recipient name and type
• Grant amount and currency
• Grant date / announcement date
• Grant type / category
• Focus area
• Description
• Source URL
• Any additional metadata

---

✅ Completion Criteria

• Investigation report completed
• Sample data extracted (5-10 records)
• Extraction method documented
• Screenshots captured
• Technical assessment complete
• Recommendation provided
• GitHub issue updated with findings
• Source README updated

---

🚀 Getting Started

1. Setup workspace:

mkdir -p data/raw/funding_sources/[source_name]/{manual,screenshots}

2. Run inspection:

python scripts/investigation/inspect_site.py [SOURCE_URL]

3. Manual exploration:

• Browse site, take screenshots
• Try search features
• Check DevTools Network tab

4. Extract sample:

python scripts/investigation/extract_sample.py [URL] [source_name] [output_dir]

5. Complete report and update this issue

---

📝 Deliverables

1. INVESTIGATION_REPORT.md - Completed investigation
2. manual/sample_YYYYMMDD.json - Sample data
3. screenshots/ - Key screenshots
4. extraction_prototype.py - Prototype script (if automated feasible)
5. GitHub issue comment with summary

---

🔄 Integration

• Issue Create dump spaces for funding event data from multiple sources #2: Use dump space structure for sample data
• Issue Set up data directory structure with raw/transformed/serveable zones #1: Investigation informs migration script requirements
• Future: Findings guide automated extraction development

---

See full investigation template with detailed examples in: issues_3_9_plan.md