refactor/use in memory cache for createLocalisedResourceDocJsonCached…

.github/workflows/build_container_develop_branch.yml

@@ -69,9 +69,6 @@ jobs:
           echo COUNTERPARTY_OTP_INSTRUCTION_TRANSPORT=dummy >> obp-api/src/main/resources/props/test.default.props
           echo SEPA_CREDIT_TRANSFERS_OTP_INSTRUCTION_TRANSPORT=dummy >> obp-api/src/main/resources/props/test.default.props
 
-          echo kafka.akka.timeout = 9 >> obp-api/src/main/resources/props/test.default.props
-          echo remotedata.timeout = 10 >> obp-api/src/main/resources/props/test.default.props
-
           echo allow_oauth2_login=true >> obp-api/src/main/resources/props/test.default.props
           echo oauth2.jwk_set.url=https://www.googleapis.com/oauth2/v3/certs >> obp-api/src/main/resources/props/test.default.props
 

.github/workflows/build_container_non_develop_branch.yml

@@ -69,9 +69,6 @@ jobs:
           echo COUNTERPARTY_OTP_INSTRUCTION_TRANSPORT=dummy >> obp-api/src/main/resources/props/test.default.props
           echo SEPA_CREDIT_TRANSFERS_OTP_INSTRUCTION_TRANSPORT=dummy >> obp-api/src/main/resources/props/test.default.props
 
-          echo kafka.akka.timeout = 9 >> obp-api/src/main/resources/props/test.default.props
-          echo remotedata.timeout = 10 >> obp-api/src/main/resources/props/test.default.props
-
           echo allow_oauth2_login=true >> obp-api/src/main/resources/props/test.default.props
           echo oauth2.jwk_set.url=https://www.googleapis.com/oauth2/v3/certs >> obp-api/src/main/resources/props/test.default.props
 

.github/workflows/build_pull_request.yml

@@ -65,8 +65,6 @@ jobs:
           echo COUNTERPARTY_OTP_INSTRUCTION_TRANSPORT=dummy >> obp-api/src/main/resources/props/test.default.props
           echo SEPA_CREDIT_TRANSFERS_OTP_INSTRUCTION_TRANSPORT=dummy >> obp-api/src/main/resources/props/test.default.props
 
-          echo kafka.akka.timeout = 9 >> obp-api/src/main/resources/props/test.default.props
-          echo remotedata.timeout = 10 >> obp-api/src/main/resources/props/test.default.props
 
           echo allow_oauth2_login=true >> obp-api/src/main/resources/props/test.default.props
           echo oauth2.jwk_set.url=https://www.googleapis.com/oauth2/v3/certs >> obp-api/src/main/resources/props/test.default.props

README.md

@@ -374,17 +374,6 @@ We use 9 to run the API in production mode.
 
 Most internal OBP model data access now occurs over Akka. This is so the machine that has JDBC access to the OBP database can be physically separated from the OBP API layer. In this configuration we run two instances of OBP-API on two different machines and they communicate over Akka. Please see README.Akka.md for instructions.
 
-## Using SSL Encryption with kafka
-
-For SSL encryption we use JKS keystores. Note that both the keystore and the truststore (and all keys within) must have the same password for unlocking, for which the API will stop at boot up and ask for. 
-
-* Edit your props file(s) to contain:
-
-   ```
-   kafka.use.ssl=true
-   keystore.path=/path/to/api.keystore.jks
-   truststore.path=/path/to/api.truststore.jks
-   ```
 
 ## Using SSL Encryption with RabbitMq
 

cheat_sheet.md

@@ -24,8 +24,6 @@
 
 [Access Control](https://apiexplorersandbox.openbankproject.com/glossary#API.Access-Control)
 
-[OBP Kafka](https://apiexplorersandbox.openbankproject.com/glossary#Adapter.Kafka.Intro)
-
 [OBP Akka](https://apiexplorersandbox.openbankproject.com/glossary#Adapter.Akka.Intro)
 
 [API Explorer](https://github.com/OpenBankProject/API-Explorer/blob/develop/README.md)

completed_developments.md

@@ -278,10 +278,6 @@ Support for on premise OAuth2 provider e.g. MitreId. See the glossary.
 ### Message Docs (for Akka)
 Message Docs (which define Core Banking System Akka messages) are now available independent of the connector being used on the API instance. See [here](https://apiexplorersandbox.openbankproject.com/?ignoredefcat=true&tags=#v2_2_0-getMessageDocs)
 
-
-### Message Docs (for Kafka)
-Message Docs (which define Core Banking System Kafka messages) are now available independent of the connector being used on the API instance. See [here](https://apiexplorersandbox.openbankproject.com/?ignoredefcat=true&tags=#v2_2_0-getMessageDocs)
-
 ### Endpoint config and cleanup
 Endpoints can now be enabled / disabled explicitly using Props file.
 We removed old versions including v1.0, v1.1 and v.1.2.
@@ -302,15 +298,11 @@ We added Custom code folders so that bank specific forks can more easily git mer
 
 
 ### API Tester
-API Tester is a Python/Djano App for testing an OBP API instance from the outside. Partiularly useful when using a non-sandbox (e.g. kafka) connector. It supports a variety of authentication methods so you can test outside a gateway. You can configure different data profiles for specifying parameters such as bank_id, account_id etc. See [here](https://github.com/OpenBankProject/API-Tester) for the source code and installation instructions.
+API Tester is a Python/Djano App for testing an OBP API instance from the outside. Partiularly useful when using a non-sandbox (e.g. RabbitMq) connector. It supports a variety of authentication methods so you can test outside a gateway. You can configure different data profiles for specifying parameters such as bank_id, account_id etc. See [here](https://github.com/OpenBankProject/API-Tester) for the source code and installation instructions.
 
 ### Extend Swagger support
 We improved the information contained in the Swagger (and Resource Doc) endpoints. They are also available from the API Explorer. See [here](https://apiexplorersandbox.openbankproject.com/?ignoredefcat=true&tags=#v1_4_0-getResourceDocsSwagger)
 
-
-### Kafka versioning
-The built in kafka connectors now provide message versioning
-
 ### Akka Remote data (Three tier architechture)
 Most OBP data access now happens over Akka. This allows the API layer to be physically separated from the storage layer with the API layer only able to call a specified set of data access functions with only the storage layer having JDBC / SQL access.
 
@@ -428,13 +420,6 @@ is used to explore and interact with the OBP API. See [API Explorer on Sandbox](
 See [Resource Docs endpoint](https://api.openbankproject.com/obp/v1.4.0/resource-docs/obp)
 
 
-### Kafka connector
-
-*   Get transactions via Kafka bus and language neutral connector on the south side of the MQ
-
-See [Docker obp-full-kafka](https://hub.docker.com/r/openbankproject/obp-full-kafka/)
-
-
 ### Version 1.4.0
 
 This version is stable. For the spec see [here](https://github.com/OpenBankProject/OBP-API/wiki/REST-API-V1.4.0) or [here](https://apiexplorersandbox.openbankproject.com/?version=1.4.0&list-all-banks=false&core=&psd2=&obwg=&ignoredefcat=true)

obp-api/pom.xml

@@ -79,16 +79,6 @@
       <groupId>org.slf4j</groupId>
       <version>1.7.26</version>
     </dependency>
-    <dependency>
-      <groupId>org.apache.kafka</groupId>
-      <artifactId>kafka-clients</artifactId>
-      <version>${kafka.version}</version>
-    </dependency>
-    <dependency>
-      <groupId>org.apache.kafka</groupId>
-      <artifactId>connect-json</artifactId>
-      <version>${kafka.version}</version>
-    </dependency>
     <dependency>
       <groupId>org.bouncycastle</groupId>
       <artifactId>bcpg-jdk15on</artifactId>
@@ -253,11 +243,6 @@
       <artifactId>akka-remote_${scala.version}</artifactId>
       <version>${akka.version}</version>
     </dependency>
-    <dependency>
-      <groupId>com.typesafe.akka</groupId>
-      <artifactId>akka-stream-kafka_${scala.version}</artifactId>
-      <version>${akka-streams-kafka.version}</version>
-    </dependency>
     <dependency>
       <groupId>com.sksamuel.avro4s</groupId>
       <artifactId>avro4s-core_${scala.version}</artifactId>
@@ -512,14 +497,6 @@
       <version>1.20.3</version>
       <scope>test</scope>
     </dependency>
-    <!-- https://mvnrepository.com/artifact/org.testcontainers/kafka -->
-    <dependency>
-      <groupId>org.testcontainers</groupId>
-      <artifactId>kafka</artifactId>
-      <version>1.20.3</version>
-      <scope>test</scope>
-    </dependency>
-
   </dependencies>
 
   <build>

obp-api/src/main/resources/i18n/lift-core_es_ES.properties

@@ -947,15 +947,11 @@ Este error no puede ser mostrado al usuario, sólo para depuración.
 OBP-50004 = método (AuthUser.getCurrentUser) no puede encontrar el usuario actual en el contexto actual!
 OBP-50005 = ha producido un error interno o no especificado.
 OBP-50006 = interrumpió la excepción.
-OBP-50007 = de ejecución de Kafka.
-OBP-50008 = de tiempo de espera del flujo Kafka de Akka.
-OBP-50009 = desconocido de Kafka.
 OBP-50010 = devuelve la caja vacía a Liftweb.
 OBP-50012 = se puede obtener el objeto CallContext aquí.
 OBP-50013 = sistema bancario central devolvió un error o una respuesta no especificada.
 OBP-50014 = se puede actualizar el usuario.
 OBP-50015 = servidor encontró una condición inesperada que le impidió cumplir con la solicitud.
-OBP-50016 = servidor kafka no está disponible.
 OBP-50017 = punto final está prohibido en esta instancia de la API.
 OBP-50018 = de construcción.
 OBP-50019 = se puede conectar a la base de datos OBP.
@@ -981,7 +977,6 @@ OBP-50217 = no devolvió la transacción que solicitamos.
 OBP-50218 = conector no devolvió el conjunto de etiquetas de punto final que solicitamos.
 OBP-50219 = no devolvió las cuentas bancarias que solicitamos.
 #Excepciones del adaptador (OBP-6XXXX)
-#Reservado para mensajes del adaptador (al sur de Kafka)
 #También se utiliza para el conector == mapeado, y mostrarlo como los errores internos.
 OBP-60001 = excepción de transacción. 
 OBP-60002 = la Excepción de Valor de Carga. 

obp-api/src/main/resources/props/sample.props.template

@@ -31,7 +31,7 @@ connector=star
 #hikari.keepaliveTime=
 #hikari.maxLifetime=
 
-## if connector = star, then need to set which connectors will be used. For now, obp support rest, akka, kafka. If you set kafka, then you need to start the kafka server.
+## if connector = star, then need to set which connectors will be used. For now, obp support rest, akka. 
 starConnector_supported_types=mapped,internal
 
 ## whether export LocalMappedConnector methods as endpoints, it is just for develop, default is false
@@ -125,34 +125,13 @@ long_endpoint_timeout = 55000
 ## Scheduler will be disabled if delay is not set.
 #transaction_status_scheduler_delay=300
 
-## If using kafka, set the brokers
-#kafka.bootstrap_hosts=localhost:9092
-# WARNING: if this number does not match the partitions in Kafka config, you will SUFFER !
-#kafka.partitions=3
-
-#This is the api instance, we create kafka topic based on this number, each instance should have each own id. use it in load balancing + Kafka setup
-#This is also used for scheduler.
-#OBP set the default as the non-persistent UUID string.
-#api_instance_id=7uy8a7e4-6d02-40e3-a129-0b2bf89de8uh
-#If a value is set, OBP will concatenate the UUID string with the value.
-#api_instance_id=1
-#When the value is set to conclude with "final," OBP will preserve that value.
-#api_instance_id=1_final
-
-## DEPRECATED
-## Enable user authentication via kafka
-#kafka.user.authentication=true
-
 
 ## Enable user authentication via the connector
 #connector.user.authentication=true
 
 ## Enable SSL for JWT, if set to true must set paths for the keystore locations
 jwt.use.ssl=false
 
-## Enable SSL for kafka, if set to true must set paths for the keystore locations
-#kafka.use.ssl=true
-
 ## Enable SSL for rabbitmq, if set to true must set paths for the keystore locations
 #rabbitmq.use.ssl=false
 
@@ -710,9 +689,6 @@ autocomplete_at_login_form_enabled=false
 # To BYPASS this security features (for local development only), set this property to true to skip the email address validation.
 #authUser.skipEmailValidation=false
 
-# If using Kafka but want to get counterparties from OBP, set this to true
-#get_counterparties_from_OBP_DB=true
-
 # control the create and access to public views.
 # allow_public_views=false
 
@@ -756,7 +732,7 @@ dauth.host=127.0.0.1
 # }
 # When is enabled we show all messages in a chain. For instance:
 # {
-#   "error": "OBP-30001: Bank not found. Please specify a valid value for BANK_ID. <- Full(Kafka_TimeoutExceptionjava.util.concurrent.TimeoutException: The stream has not been completed in 1550 milliseconds.)"
+#   "error": "OBP-30001: Bank not found. Please specify a valid value for BANK_ID. <- Full(TimeoutExceptionjava.util.concurrent.TimeoutException: The stream has not been completed in 1550 milliseconds.)"
 # }
 display_internal_errors=false
 # -------------------------------------- Display internal errors --

obp-api/src/main/resources/props/test.default.props.template

@@ -18,7 +18,6 @@
 #which data connector to use
 
 #connector=rest
-#connector=kafka
 #connector=obpjvm
 ## proxy connector get data from LocalMappedConnector, and set the follow corresponding fields to be null: @optional, inbound.optional.fields props, outbound.optional.fields props
 #connector=proxy
@@ -28,17 +27,6 @@ starConnector_supported_types = mapped,internal
 # Connector cache time-to-live in seconds, caching disabled if not set
 #connector.cache.ttl.seconds=3
 
-# OBP-JVM transport type. currently supported: kafka, mock
-#obpjvm.transport=kafka
-
-#if using kafka, set zookeeper host and brokers
-#defaults to "localhost:2181" if not set
-#kafka.zookeeper_host=localhost:2181
-#kafka.bootstrap_hosts=localhost:9092
-
-#if using kafka, the following is mandatory
-#kafka.request_topic=Request
-#kafka.response_topic=Response
 
 #this is needed for oauth to work. it's important to access the api over this url, e.g.
 # if this is 127.0.0.1 don't use localhost to access it.

obp-api/src/main/scala/bootstrap/liftweb/Boot.scala

@@ -80,7 +80,6 @@ import code.endpointTag.EndpointTag
 import code.entitlement.{Entitlement, MappedEntitlement}
 import code.entitlementrequest.MappedEntitlementRequest
 import code.fx.{MappedCurrency, MappedFXRate}
-import code.kafka.{KafkaHelperActors, OBPKafkaConsumer}
 import code.kycchecks.MappedKycCheck
 import code.kycdocuments.MappedKycDocument
 import code.kycmedias.MappedKycMedia
@@ -540,13 +539,6 @@ class Boot extends MdcLoggable {
     }
 
 
-    if (connector.startsWith("kafka") || (connector == "star" && APIUtil.getPropsValue("starConnector_supported_types","").split(",").contains("kafka"))) {
-      logger.info(s"KafkaHelperActors.startLocalKafkaHelperWorkers( ${actorSystem} ) starting")
-      KafkaHelperActors.startLocalKafkaHelperWorkers(actorSystem)
-      // Start North Side Consumer if it's not already started
-      OBPKafkaConsumer.primaryConsumer.start()
-    }
-
     // API Metrics (logs of API calls)
     // If set to true we will write each URL with params to a datastore / log file
     if (APIUtil.getPropsAsBoolValue("write_metrics", false)) {

obp-api/src/main/scala/code/actorsystem/ObpLookupSystem.scala

@@ -28,34 +28,6 @@ trait ObpLookupSystem extends MdcLoggable {
     obpLookupSystem
   }
 
-  def getKafkaActor(actorName: String) = {
-
-    val actorPath: String = {
-      val hostname = ObpActorConfig.localHostname
-      val port = ObpActorConfig.localPort
-      val props_hostname = Helper.getHostname
-      if (port == 0) {
-        logger.error("Failed to connect to local Kafka actor")
-      }
-      s"akka.tcp://ObpActorSystem_${props_hostname}@${hostname}:${port}/user/${actorName}"
-    }
-
-    this.obpLookupSystem.actorSelection(actorPath)
-  }
-
-  def getKafkaActorChild(actorName: String, actorChildName: String) = {
-    val actorPath: String = {
-      val hostname = ObpActorConfig.localHostname
-      val port = ObpActorConfig.localPort
-      val props_hostname = Helper.getHostname
-      if (port == 0) {
-        logger.error("Failed to connect to local Kafka actor")
-      }
-      s"akka.tcp://ObpActorSystem_${props_hostname}@${hostname}:${port}/user/${actorName}/${actorChildName}"
-    }
-    this.obpLookupSystem.actorSelection(actorPath)
-  }
-
   def getActor(actorName: String) = {
 
     val actorPath: String = {