[Proposed Feature] Intelligent Analytics for Deeper End-User Understanding

[diegoripley]

Motivation

This proposal is primarily motivated by the challenges discussed in The Storm Events Database Explorer (Great Data Products). As highlighted by Brad Andrich and Quinn Kiter, understanding user needs currently requires intensive manual effort, such as the 20-question surveys used to identify workflows for meteorologists and emergency managers [00:14:36]. Furthermore, there is a critical need to serve "underserved" personas—such as local government planners—who require high-value geospatial insights but lack the "army of data scientists" or SQL expertise to interrogate raw datasets [00:17:02]. By building analytics directly into each data product, Source Cooperative can automate this discovery process for publishers and lower the barrier for non-technical users.

Description of Feature

Add built-in analytics for geospatial data repositories to automatically analyze server access logs and generate insights on most-viewed geospatial areas (e.g., via converting PMTiles range requests to Z/X/Y tile requests and their geospatial extent or GeoParquet spatial filter patterns converted to bounding box extents) and user locations (city-level from client IPs using GeoIP lookups). Additionally, you could use the user-agent for more granular information about the end-users (ex. QGIS) so that Source Cooperative can prioritize work accordingly.

The implementation could include privacy controls to respect Do Not Track (DNT) headers by skipping enrichment and anonymizing IPs when DNT=1. Output aggregated metrics (e.g., view counts per tile extent, spatial query bounds, or city) would be sent to a queryable backend for visualization in tools like Grafana GeoMap panels.

Technical Implementation

The system could leverage OpenTelemetry Collector for log ingestion and Vector for parsing, enrichment, and aggregation as potential open-source components.

If executed effectively, these analytics could be exported as a native GeoParquet file for each data product. This would allow the feature to be expanded to the wider community, enabling partners to aggregate usage data cross-platform to better serve users, acknowledging that Source Cooperative is one of several initiatives serving the geospatial ecosystem.

Value to Source Cooperative

1. Actionable Insights for Publishers

This feature transforms Source from a static data hosting utility into an intelligent platform that reveals usage patterns without requiring publishers to conduct manual user surveys [00:14:18]. Publishers gain instant feedback on:

• Geospatial Hotspots: Identifying which specific regions (e.g., specific waterways, urban zones, or weather corridors) are being accessed most frequently.
• User Geography: Understanding city-level demand from sectors like meteorology and emergency management.
• Strategic Optimization: Enabling data prioritization and regional server optimizations based on real-world demand.

2. Community & Ecosystem Growth

By providing these insights as GeoParquet files, Source Cooperative facilitates a "network effect" for open data:

• Serving the Technical Gap: As discussed in the motivator video, many users (like mid-sized city planners) need the data to "just work" [00:17:02]. Analytics help publishers tailor their data structures (like pre-calculated GeoParquet files) to these specific needs.
• Cross-Initiative Aggregation: Partners can combine Source Cooperative usage data with metrics from other platforms to form a holistic view of global geospatial data demand.
• Standardization: Using GeoParquet for analytics metadata reinforces modern standards, making it easier for users to analyze "data about the data" using tools like DuckDB or Apache Sedona.

3. Platform Differentiation & Trust

For the cooperative, this increases retention by providing high-value analytics as a core utility, differentiating Source from basic S3-like storage providers. It aligns with the mission of making planetary data accessible while fostering community trust through transparent, privacy-respecting aggregation where individual logs are not stored, only the high-level spatial trends.

Let me know if you need any further clarification on the proposed feature, I think this could be a really amazing feature.

[alukach]

Thanks for this well-written ticket! Strong agree on the value of understanding the geographic regions (@tylere suggested the same thing recently). And I think a DNT=1 opt-out of tracking also seems like a great suggestion.

We are currently tracking some details via Workers Analytics Engine:

data.source.coop/src/analytics.rs

Lines 46 to 57 in 07a6230

	let result = AnalyticsEngineDataPointBuilder::new()
	.indexes([index.as_str()])
	.add_blob(event.account_id) // blob1
	.add_blob(event.product_id) // blob2
	.add_blob(file_path) // blob3
	.add_blob(event.method) // blob4
	.add_blob(event.user_id) // blob5
	.add_blob(event.country) // blob6
	.add_blob(event.content_type) // blob7
	.add_double(event.bytes_sent) // double1
	.add_double(event.status_code) // double2
	.write_to(&dataset);

However, we are not tracking either the range or client_ip. Beyond that, it's not immediately clear to me how to convert range requests to geolocations. This will be interesting to experiment with in the coming future.

This conversation should probably be tracked in source-cooperative/source.coop#257 and source-cooperative/source.coop#259 (I typically use this codebase for very specific backend issues and use https://github.com/source-cooperative/source.coop for more "big picture" concepts + frontend/api issues)