Migrate CKAN from 2.9 to 2.11

[mosoriob]

Summary

Upgrade CKAN from version 2.9 to 2.11 to take advantage of latest features, security updates, and bug fixes.

Background

CKAN 2.11 includes significant improvements over 2.9, but also introduces breaking changes that require careful migration planning.

⚠️ Critical Breaking Changes

1. SECRET_KEY Configuration (Required)

• Breaking: CKAN 2.11 requires the SECRET_KEY configuration option to start
• Action: Add SECRET_KEY = your_secret_key_here to ckan.ini
• Generate with: python -c "import secrets; print(secrets.token_urlsafe(20))"

2. Session Handling Refactored

• Breaking: Replaced Beaker sessions with Flask-Session
• Action: Update all session configuration keys in ckan.ini
• Users will be logged out when server sessions are cleared

3. Activity Plugin Extraction

• Breaking: Activities moved to separate plugin
• Action: Add activity to plugins list: ckan.plugins = ... activity ...

4. Authentication Changes (Flask-Login)

• Breaking: Replaced repoze.who with Flask-Login
• Different session persistence behavior

5. Configuration Validation (Strict Mode)

• Breaking: Invalid config options will prevent CKAN from starting
• Action: Review and fix all configuration options

🔒 Security Changes

6. Upload Type Restrictions

• Breaking: File types for user/group images now restricted by default
• Action: Configure ckan.upload.user.mimetypes if additional formats needed

7. Public Registration Disabled

• Breaking: User registration disabled by default
• Action: Set ckan.auth.create_user_via_web = true to re-enable

8. API Token Authentication Required

• Breaking: Extensions require valid API tokens
• Action: Configure API tokens for DataPusher/Xloader

🗄️ Database & Infrastructure

9. Database Upgrade Required

• Run ckan db upgrade and ckan datastore upgrade
• Minimum PostgreSQL version: 12
• Python 3.9-3.12 required

10. Tracking Extension

• Breaking: Tracking moved to core extension
• Action: Add tracking to plugins list

🎨 UI Changes

11. Bootstrap 5 Migration

• Breaking: Default templates now use Bootstrap 5
• Fallback: Configure Bootstrap 3 templates if needed

🗑️ Removed Features

12. Recline View Plugins Removed

• recline_view, recline_grid_view, recline_map_view no longer available
• Action: Use datatables_view or community alternatives

13. Legacy API Keys Removed

• Only API tokens supported for authentication

Migration Tasks

Pre-Migration

• Backup database completely
• Backup configuration files
• Test migration in development environment
• Verify Python 3.9+ and PostgreSQL 12+

Configuration Updates

• Add SECRET_KEY to configuration
• Update session configuration keys (Beaker → Flask-Session)
• Add activity plugin to plugin list
• Add tracking plugin if using tracking
• Review and fix invalid configuration options
• Update API token configurations for extensions
• Configure upload mimetypes if needed
• Set user registration policy

Custom Extension Compatibility

• Test ckanext-tacc_theme compatibility

• Test ckanext-dso_scheming compatibility

• Test ckanext-tapisfilestore compatibility

• Test ckanext-spatial compatibility

• Test ckanext-ouath2 compatibility

• Update any extensions using removed features

Database Migration

• Run ckan db upgrade
• Run ckan datastore upgrade
• Clear Redis sessions if using Redis

Docker Configuration Updates

• Update Dockerfile and docker-compose files
• Update environment variable mappings
• Test Docker build and deployment

Post-Migration Testing

• Test user authentication and sessions
• Verify OAuth2 integration still works
• Test file upload functionality
• Check activity streams
• Test API token authentication
• Verify all enabled plugins work correctly
• Test Tapis file storage integration
• Test frontend UI (Bootstrap 5 changes)
• Monitor logs for configuration errors

Documentation Updates

• Update deployment documentation
• Update development setup instructions
• Document any configuration changes

Considerations

• This is a major version upgrade with significant architectural changes
• Thorough testing is essential before deploying to production
• Session management behavior has changed - users may experience different logout behavior
• Some extensions may need updates to work with CKAN 2.11
• Allow adequate time for testing and validation

Acceptance Criteria

• CKAN successfully upgraded to version 2.11
• All custom extensions working properly
• OAuth2 authentication functioning
• File storage via Tapis working correctly
• All existing datasets and users preserved
• Performance is maintained or improved
• All breaking changes addressed
• Configuration validation passes
• No errors in application logs

Resources

• CKAN 2.11 Full Changelog
• CKAN Configuration Guide
• [Migration requires careful planning due to breaking changes]