NIFI-15355 Refactor ConnectorRepository API for extension point

[kevdoran]

Summary

NIFI-15355

This PR refactors the Connector API to create clean extension points for third-party implementations. The primary goal is to allow external developers to implement custom ConnectorRepository and ConnectorLifecycleManager instances without requiring changes to NiFi core.

Motivation

The original ConnectorRepository interface in nifi-framework-core-api was entangled with internal framework dependencies (like FlowManager) that cannot be exposed as a stable public API. This refactoring creates a layered architecture where:

• Public extension points live in nifi-framework-api with minimal, stable dependencies
• Internal orchestration remains in nifi-framework-core-api and nifi-framework-core

Updated Internal Arch Diagram

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              nifi-web-api                                       │
│  ┌───────────────────┐                                                          │
│  │ ConnectorResource │  ◄── REST API endpoints (/connectors/*)                  │
│  └─────────┬─────────┘                                                          │
│            │                                                                    │
│            ▼                                                                    │
│  ┌─────────────────────────┐                                                    │
│  │StandardNiFiServiceFacade│  ◄── Service layer / orchestration                 │
│  └─────────┬───────────────┘                                                    │
│            │                                                                    │
│            ▼                                                                    │
│  ┌─────────────────────┐                                                        │
│  │ StandardConnectorDAO│  ◄── Data Access Object                                │
│  └─────────┬───────────┘                                                        │
└────────────┼────────────────────────────────────────────────────────────────────┘
             │
             ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                            nifi-framework-core                                  │
│  ┌───────────────┐                                                              │
│  │FlowController │  ◄── Central controller, owns FlowManager & ConnectorManager │
│  └───────┬───────┘                                                              │
│          │                                                                      │
│          ├──────────────────────┐                                               │
│          ▼                      ▼                                               │
│  ┌──────────────────┐    ┌─────────────────────────┐                            │
│  │   FlowManager    │    │ StandardConnectorManager│ ◄── Internal orchestrator  │
│  │                  │    │    (ConnectorManager)   │                            │
│  │ -createConnector │    └───────────┬─────────────┘                            │
│  │ -removeConnector │                │                                          │
│  └──────────────────┘                │ delegates to                             │
│                           ┌──────────┴──────────┐                               │
│                           ▼                     ▼                               │
│            ┌──────────────────────┐  ┌─────────────────────────────┐            │
│            │FlowJsonConnectorRepo │  │StandardConnectorLifecycleMgr│            │
│            │   (default impl)     │  │       (default impl)        │            │
│            └──────────┬───────────┘  └──────────────┬──────────────┘            │
└───────────────────────┼─────────────────────────────┼───────────────────────────┘
                        │ implements                  │ implements
┌───────────────────────┼─────────────────────────────┼───────────────────────────┐
│                       ▼                             ▼      nifi-framework-api   │
│  ┌─────────────────────────────┐    ┌────────────────────────────────┐          │
│  │     ConnectorRepository     │    │   ConnectorLifecycleManager    │          │
│  │      «interface»            │    │        «interface»             │          │
│  │                             │    │                                │          │
│  │ + save(record)              │    │ + startConnector(node)         │          │
│  │ + load(id) / loadAll()      │    │ + stopConnector(node)          │          │
│  │ + delete(id)                │    │ + restartConnector(node)       │          │
│  │ + prepareForUpdate(id)      │    │ + getConnectorState(id)        │          │
│  │ + supportsExternalMod()     │    │                                │          │
│  └─────────────────────────────┘    └────────────────────────────────┘          │
│               ▲                                    ▲                            │
│               │                                    │                            │
│     ┌─────────┴─────────┐                ┌────────┴────────┐                    │
│     │  EXTENSION POINT  │                │ EXTENSION POINT │                    │
│     │ (3rd party impl)  │                │ (3rd party impl)│                    │
│     └───────────────────┘                └─────────────────┘                    │
│                                                                                 │
│  Examples:                                                                      │
│   - S3ConnectorRepository         - Custom lifecycle with                       │
│   - DatabaseConnectorRepository     external orchestration                      │
└─────────────────────────────────────────────────────────────────────────────────┘

Data Flow for Creating a Connector:

REST Request → ConnectorResource → ServiceFacade → ConnectorDAO 
            → FlowManager.createConnector() → ConnectorManager.addConnector()
            → ConnectorRepository.save()

Data Flow for Updating a Connector:

REST Request → ... → ConnectorDAO → ConnectorManager.applyUpdate()
            → ConnectorRepository.prepareForUpdate()  [can reject → 409 Conflict]
            → ConnectorLifecycleManager.stopConnector()
            → apply changes
            → ConnectorLifecycleManager.startConnector()
            → ConnectorRepository.completeUpdate()

Key Changes

New Public Extension Points (nifi-framework-api)

Interface	Purpose
ConnectorRepository	Persistence of connector configuration (save, load, delete)
ConnectorLifecycleManager	Runtime lifecycle management (start, stop, restart)
SecretsProvider	Pluggable secrets management
PersistedConnectorRecord	Data transfer object for persisted connector state
ConnectorRepositoryContext	Initialization context for repository implementations
ConnectorLifecycleContext	Initialization context for lifecycle implementations

Internal Framework Changes

• Renamed the internal ConnectorRepository to ConnectorManager in nifi-framework-core-api
• ConnectorManager now orchestrates between the pluggable ConnectorRepository and ConnectorLifecycleManager
• Created FlowJsonConnectorRepository as the default implementation (embeds connectors in flow.json.gz)
• Created StandardConnectorLifecycleManager as the default lifecycle implementation

External Modification Support

Third-party repositories can support external changes to connector configuration (e.g., files modified directly in S3 or records updated in a database):

• supportsExternalModification() - indicates if external changes are supported
• prepareForUpdate() / completeUpdate() - transactional update semantics allowing the repository to "vote" on updates
• ConnectorUpdateRejectedException - returns HTTP 409 Conflict when updates are rejected

Configuration

Implementations are configurable via nifi.properties:

nifi.connector.repository.implementation=org.apache.nifi.components.connector.FlowJsonConnectorRepository
nifi.connector.lifecycle.manager.implementation=org.apache.nifi.components.connector.StandardConnectorLifecycleManager

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
• Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

[kevdoran]

• Remove CONNECTOR_MANAGER_IMPLEMENTATION as that is no longer intended to be configurable via properties