Skip to content

feat: dynamic geoip db support#940

Merged
sdsantos merged 18 commits intomainfrom
feat/dynamic-geoipdb
Feb 3, 2026
Merged

feat: dynamic geoip db support#940
sdsantos merged 18 commits intomainfrom
feat/dynamic-geoipdb

Conversation

@aanorbel
Copy link
Copy Markdown
Member

@aanorbel aanorbel commented Oct 6, 2025

  • Implement dynamic GeoIP database updates by fetching the latest MMDB files from a GitHub repository
  • Perform daily version checks to avoid unnecessary downloads
  • Download new versions when available and clean up old database files to save space
  • Integrate the downloaded path into engine preferences for use in network tests

Closes ooni/probe#2868

@aanorbel aanorbel marked this pull request as draft October 6, 2025 10:41
Comment thread composeApp/src/desktopMain/kotlin/org/ooni/engine/DesktopOonimkallBridge.kt Outdated
Comment thread composeApp/src/commonMain/kotlin/org/ooni/probe/App.kt Outdated
Comment thread composeApp/src/commonMain/kotlin/org/ooni/probe/net/Http.kt Outdated
Comment thread composeApp/src/androidMain/kotlin/org/ooni/probe/net/Http.android.kt Outdated
Comment thread composeApp/src/desktopMain/kotlin/org/ooni/probe/net/Http.desktop.kt Outdated
val ip: String?,
val asn: String?,
val countryCode: String?,
val geoIpdb: String?,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to get the geoIpDb from the engine?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To determine which version of geoIPDB the engine is currently working with.

Comment thread composeApp/src/commonMain/kotlin/org/ooni/engine/models/EnginePreferences.kt Outdated
Comment thread composeApp/src/commonMain/kotlin/org/ooni/engine/Engine.kt Outdated
Comment thread composeApp/src/commonMain/kotlin/org/ooni/probe/domain/GetEnginePreferences.kt Outdated
Comment thread composeApp/src/commonMain/kotlin/org/ooni/probe/di/Dependencies.kt Outdated
@aanorbel aanorbel force-pushed the feat/dynamic-geoipdb branch 2 times, most recently from af697b7 to eef3dbc Compare October 28, 2025 16:11
@aanorbel aanorbel force-pushed the feat/dynamic-geoipdb branch from eef3dbc to b5322c8 Compare November 4, 2025 13:22
@aanorbel aanorbel requested a review from sdsantos November 5, 2025 08:20
@aanorbel aanorbel changed the title chore: draft for using geoipdb from path feat: dynamic geoip db support Nov 5, 2025
@aanorbel aanorbel marked this pull request as ready for review November 5, 2025 08:23
Copy link
Copy Markdown
Contributor

@sdsantos sdsantos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some other issues:
a) We're not cleaning old version of the geoIp database after we download an update
b) GeoIP databases will count for storage usaged, but they are not cleared when the clean storage (maybe this is what we want, but just wanted to make sure)
c) The fetch is running on app start, and therefore on any instrumented test we have. It should be disabled for most tests I imagine.

Comment thread composeApp/src/androidMain/kotlin/org/ooni/probe/net/Http.android.kt Outdated
Comment thread composeApp/src/androidMain/kotlin/org/ooni/probe/net/Http.android.kt Outdated
Comment thread composeApp/src/iosMain/kotlin/org/ooni/probe/net/Http.ios.kt Outdated
Comment on lines +29 to +30
const val GEOIP_DB_VERSION_DEFAULT: String = "20250801"
const val GEOIP_DB_REPO: String = "aanorbel/oomplt-mmdb"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These constants could be private.

Comment thread composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated
Comment thread composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated
Comment thread composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated
return Success(null)
} else {
val url = buildGeoIpDbUrl(latestVersion)
val target = "$cacheDir/$latestVersion.mmdb"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the cacheDir the place we really want for these files? Is some systems, cache directories can be deleted periodically.

@aanorbel aanorbel force-pushed the feat/dynamic-geoipdb branch from b1d98df to 305a640 Compare November 12, 2025 13:57
Comment thread composeApp/src/commonMain/kotlin/org/ooni/probe/domain/FetchGeoIpDbUpdates.kt Outdated
Comment on lines +43 to +47
val currentTimeMillis = Clock.System.now().toEpochMilliseconds()
val oneDayInMillis = 24 * 60 * 60 * 1000L
val timeSinceLastCheck = currentTimeMillis - lastCheckMillis

if (timeSinceLastCheck < oneDayInMillis) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can take advantage of kotlin.time.Duration to make this more clear:

if (Clock.System.now() - Instant.fromEpochMilliseconds(lastCheckMillis) < 1.days) {
    
}

@aanorbel aanorbel force-pushed the feat/dynamic-geoipdb branch from f0d6556 to 4ccfdd8 Compare January 22, 2026 16:19
@sdsantos sdsantos merged commit fd26642 into main Feb 3, 2026
10 of 12 checks passed
@sdsantos sdsantos deleted the feat/dynamic-geoipdb branch February 3, 2026 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support dynamic fetching of GeoIP data in apps

2 participants