Using milvus vts to migrate from milvus to milvus.
source milvus : k8s distributed v2.5.11
target milvus : k8s distributed v2.5.11
### migration.conf:
env {
parallelism = 1
job.mode = "BATCH"
}
source {
Source configuration (e.g., Milvus, Elasticsearch, etc.)
Milvus {
url = "http://******:19530"
token = "root:Milvus"
database = "default"
# collections = [""]
batch_size = 1000
}
}
sink {
Target configuration
Milvus {
url = "http://***:30735"
token = "root:Milvus"
database = "default"
batch_size = 1000
}
}
### Here is the log:
root@NJS702IO-NEBULA-Monitor01:/opt/seatunnel/bin# ./seatunnel.sh --config ./migration.conf
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2025-09-09 09:43:11,833 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2025-09-09 09:43:11,838 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2025-09-09 09:43:11,841 INFO [o.a.s.e.c.c.SeaTunnelConfig ] [main] - seatunnel.home is /opt/seatunnel/bin
2025-09-09 09:43:11,943 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2025-09-09 09:43:11,943 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2025-09-09 09:43:12,311 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
2025-09-09 09:43:12,311 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast-client.yaml
2025-09-09 09:43:12,608 INFO [.c.i.s.ClientInvocationService] [main] - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2025-09-09 09:43:12,680 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2025-09-09 09:43:12,681 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2025-09-09 09:43:12,754 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2025-09-09 09:43:12,758 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:5801
2025-09-09 09:43:12,809 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2025-09-09 09:43:12,809 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:26c68d92-c16f-436c-99e1-138cf63084b9, server version: 5.1, local address: /127.0.0.1:2419
2025-09-09 09:43:12,811 INFO [c.h.i.d.Diagnostics ] [main] - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2025-09-09 09:43:12,833 INFO [c.h.c.i.s.ClientClusterService] [hz.client_1.event-2] - hz.client_1 [seatunnel] [5.1]
Members [1] {
Member [localhost]:5801 - 26c68d92-c16f-436c-99e1-138cf63084b9
}
2025-09-09 09:43:12,874 INFO [.c.i.s.ClientStatisticsService] [main] - Client statistics is enabled with period 5 seconds.
2025-09-09 09:43:13,146 INFO [o.a.s.c.s.u.ConfigBuilder ] [main] - Loading config file from path: ./migration.conf
2025-09-09 09:43:13,363 INFO [o.a.s.c.s.u.ConfigShadeUtils ] [main] - Load config shade spi: [base64]
2025-09-09 09:43:13,429 INFO [o.a.s.c.s.u.ConfigBuilder ] [main] - Parsed config file:
{
"sink" : [
{
"database" : "default",
"batch_size" : 10,
"plugin_name" : "Milvus",
"url" : "http://:30735",
"token" : ""
}
],
"source" : [
{
"database" : "",
"batch_size" : 100,
"plugin_name" : "Milvus",
"url" : "http://:30282",
"token" : "root:***"
}
],
"env" : {
"job.mode" : "BATCH",
"parallelism" : 1
}
}
2025-09-09 09:43:13,439 INFO [p.MultipleTableJobConfigParser] [main] - add common jar in plugins :[]
2025-09-09 09:43:13,456 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2025-09-09 09:43:13,462 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Milvus'} at: file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar
2025-09-09 09:43:13,469 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Milvus'} at: file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar
2025-09-09 09:43:13,473 INFO [p.MultipleTableJobConfigParser] [main] - start generating all sources.
2025-09-09 09:43:13,833 WARN [i.m.v.u.ClientUtils ] [main] - Failed to get host name, error:NJS702IO-NEBULA-Monitor01: NJS702IO-NEBULA-Monitor01: Temporary failure in name resolution
2025-09-09 09:43:14,285 INFO [s.u.MilvusSourceConnectorUtils] [main] - describe collection database: zhengqi, collection: test, response: DescribeCollectionResp(collectionName=test, collectionID=457209259783653913, databaseName=zhengqi, description=, numOfPartitions=1, fieldNames=[id, embedding, text, reference, metadata], vectorFieldNames=[embedding], primaryFieldName=id, enableDynamicField=false, autoID=true, collectionSchema=CreateCollectionReq.CollectionSchema(fieldSchemaList=[CreateCollectionReq.FieldSchema(name=id, description=, dataType=Int64, maxLength=65535, dimension=null, isPrimaryKey=true, isPartitionKey=false, isClusteringKey=false, autoID=true, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=embedding, description=, dataType=FloatVector, maxLength=65535, dimension=1024, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=text, description=, dataType=VarChar, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=reference, description=, dataType=VarChar, maxLength=2048, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=metadata, description=, dataType=JSON, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null)], enableDynamicField=false, functionList=[]), createTime=457339532121997318, consistencyLevel=STRONG, shardsNum=1, properties={})
2025-09-09 09:43:14,331 INFO [s.u.MilvusSourceConnectorUtils] [main] - describe collection database: zhengqi, collection: GandB_Contact_Information, response: DescribeCollectionResp(collectionName=GandB_Contact_Information, collectionID=457209259826775264, databaseName=zhengqi, description=This document contains the contact information of managers for various industries and products of China Telecom., numOfPartitions=1, fieldNames=[id, embedding, text, reference, metadata], vectorFieldNames=[embedding], primaryFieldName=id, enableDynamicField=false, autoID=true, collectionSchema=CreateCollectionReq.CollectionSchema(fieldSchemaList=[CreateCollectionReq.FieldSchema(name=id, description=, dataType=Int64, maxLength=65535, dimension=null, isPrimaryKey=true, isPartitionKey=false, isClusteringKey=false, autoID=true, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=embedding, description=, dataType=FloatVector, maxLength=65535, dimension=1024, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=text, description=, dataType=VarChar, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=reference, description=, dataType=VarChar, maxLength=2048, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=metadata, description=, dataType=JSON, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null)], enableDynamicField=false, functionList=[]), createTime=459312980648787992, consistencyLevel=STRONG, shardsNum=1, properties={})
2025-09-09 09:43:14,347 INFO [s.u.MilvusSourceConnectorUtils] [main] - describe collection database: zhengqi, collection: GandB_products, response: DescribeCollectionResp(collectionName=GandB_products, collectionID=459742145065187890, databaseName=zhengqi, description=The document from China Telecom regarding government and enterprise products provides detailed information about each product, including the product name, contact details of the product manager, a description of the product, the industry it serves, and its target customers. It also highlights the key selling points and advantages, pricing details, successful marketing case studies, marketing pitches, and frequently asked questions to help guide potential customers., numOfPartitions=1, fieldNames=[id, embedding, text, reference, metadata], vectorFieldNames=[embedding], primaryFieldName=id, enableDynamicField=false, autoID=true, collectionSchema=CreateCollectionReq.CollectionSchema(fieldSchemaList=[CreateCollectionReq.FieldSchema(name=id, description=, dataType=Int64, maxLength=65535, dimension=null, isPrimaryKey=true, isPartitionKey=false, isClusteringKey=false, autoID=true, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=embedding, description=, dataType=FloatVector, maxLength=65535, dimension=1024, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=text, description=, dataType=VarChar, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=reference, description=, dataType=VarChar, maxLength=2048, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=metadata, description=, dataType=JSON, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null)], enableDynamicField=false, functionList=[]), createTime=460234832666427404, consistencyLevel=STRONG, shardsNum=1, properties={})
2025-09-09 09:43:14,360 INFO [s.u.MilvusSourceConnectorUtils] [main] - describe collection database: zhengqi, collection: products_solution, response: DescribeCollectionResp(collectionName=products_solution, collectionID=457209259801087600, databaseName=zhengqi, description=Solution materials of ChinaTelecom in 12 government and enterprise industries (industry, transportation, education, finance, agriculture, health, cultural propaganda, key clients, emergency response, law enforcement & public security, government affairs, urban construction) and the product whitelist of ChinaTelecom., numOfPartitions=1, fieldNames=[id, embedding, text, reference, metadata], vectorFieldNames=[embedding], primaryFieldName=id, enableDynamicField=false, autoID=true, collectionSchema=CreateCollectionReq.CollectionSchema(fieldSchemaList=[CreateCollectionReq.FieldSchema(name=id, description=, dataType=Int64, maxLength=65535, dimension=null, isPrimaryKey=true, isPartitionKey=false, isClusteringKey=false, autoID=true, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=embedding, description=, dataType=FloatVector, maxLength=65535, dimension=1024, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=text, description=, dataType=VarChar, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=reference, description=, dataType=VarChar, maxLength=2048, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=metadata, description=, dataType=JSON, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null)], enableDynamicField=false, functionList=[]), createTime=458156421846925319, consistencyLevel=STRONG, shardsNum=1, properties={})
2025-09-09 09:43:14,378 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - get the CatalogTable from source Milvus: milvus.zhengqi.GandB_Contact_Information,milvus.zhengqi.test,milvus.zhengqi.products_solution,milvus.zhengqi.GandB_products
2025-09-09 09:43:14,393 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSource Plugin from /opt/seatunnel/connectors
2025-09-09 09:43:14,399 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Milvus'} at: file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar
2025-09-09 09:43:14,402 INFO [p.MultipleTableJobConfigParser] [main] - start generating all transforms.
2025-09-09 09:43:14,402 INFO [p.MultipleTableJobConfigParser] [main] - start generating all sinks.
2025-09-09 09:43:14,409 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2025-09-09 09:43:14,410 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Milvus'} at: file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar
2025-09-09 09:43:14,421 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Milvus' with upstream input catalog-table[database: zhengqi, schema: null, table: GandB_Contact_Information]
2025-09-09 09:43:14,428 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Milvus' with upstream input catalog-table[database: zhengqi, schema: null, table: test]
2025-09-09 09:43:14,431 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Milvus' with upstream input catalog-table[database: zhengqi, schema: null, table: products_solution]
2025-09-09 09:43:14,434 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Milvus' with upstream input catalog-table[database: zhengqi, schema: null, table: GandB_products]
2025-09-09 09:43:14,521 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Start submit job, job id: 1017611102499897345, with plugin jar [file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar]
2025-09-09 09:43:15,321 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Submit job finished, job id: 1017611102499897345, job name: SeaTunnel_Job
2025-09-09 09:43:15,340 WARN [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-1017611102499897345] - Failed to get job metrics summary, it maybe first-run
Using milvus vts to migrate from milvus to milvus.
source milvus : k8s distributed v2.5.11
target milvus : k8s distributed v2.5.11
### migration.conf:
env {
parallelism = 1
job.mode = "BATCH"
}
source {
Source configuration (e.g., Milvus, Elasticsearch, etc.)
Milvus {
url = "http://******:19530"
token = "root:Milvus"
database = "default"
# collections = [""]
batch_size = 1000
}
}
sink {
Target configuration
Milvus {
url = "http://***:30735"
token = "root:Milvus"
database = "default"
batch_size = 1000
}
}
### Here is the log:
root@NJS702IO-NEBULA-Monitor01:/opt/seatunnel/bin# ./seatunnel.sh --config ./migration.conf
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2025-09-09 09:43:11,833 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2025-09-09 09:43:11,838 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2025-09-09 09:43:11,841 INFO [o.a.s.e.c.c.SeaTunnelConfig ] [main] - seatunnel.home is /opt/seatunnel/bin
2025-09-09 09:43:11,943 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2025-09-09 09:43:11,943 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2025-09-09 09:43:12,311 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
2025-09-09 09:43:12,311 INFO [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast-client.yaml
2025-09-09 09:43:12,608 INFO [.c.i.s.ClientInvocationService] [main] - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2025-09-09 09:43:12,680 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2025-09-09 09:43:12,681 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2025-09-09 09:43:12,754 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2025-09-09 09:43:12,758 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:5801
2025-09-09 09:43:12,809 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2025-09-09 09:43:12,809 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:26c68d92-c16f-436c-99e1-138cf63084b9, server version: 5.1, local address: /127.0.0.1:2419
2025-09-09 09:43:12,811 INFO [c.h.i.d.Diagnostics ] [main] - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2025-09-09 09:43:12,833 INFO [c.h.c.i.s.ClientClusterService] [hz.client_1.event-2] - hz.client_1 [seatunnel] [5.1]
Members [1] {
Member [localhost]:5801 - 26c68d92-c16f-436c-99e1-138cf63084b9
}
2025-09-09 09:43:12,874 INFO [.c.i.s.ClientStatisticsService] [main] - Client statistics is enabled with period 5 seconds.
2025-09-09 09:43:13,146 INFO [o.a.s.c.s.u.ConfigBuilder ] [main] - Loading config file from path: ./migration.conf
2025-09-09 09:43:13,363 INFO [o.a.s.c.s.u.ConfigShadeUtils ] [main] - Load config shade spi: [base64]
2025-09-09 09:43:13,429 INFO [o.a.s.c.s.u.ConfigBuilder ] [main] - Parsed config file:
{
"sink" : [
{
"database" : "default",
"batch_size" : 10,
"plugin_name" : "Milvus",
"url" : "http://:30735",
"token" : ""
}
],
"source" : [
{
"database" : "",
"batch_size" : 100,
"plugin_name" : "Milvus",
"url" : "http://:30282",
"token" : "root:***"
}
],
"env" : {
"job.mode" : "BATCH",
"parallelism" : 1
}
}
2025-09-09 09:43:13,439 INFO [p.MultipleTableJobConfigParser] [main] - add common jar in plugins :[]
2025-09-09 09:43:13,456 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2025-09-09 09:43:13,462 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Milvus'} at: file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar
2025-09-09 09:43:13,469 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Milvus'} at: file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar
2025-09-09 09:43:13,473 INFO [p.MultipleTableJobConfigParser] [main] - start generating all sources.
2025-09-09 09:43:13,833 WARN [i.m.v.u.ClientUtils ] [main] - Failed to get host name, error:NJS702IO-NEBULA-Monitor01: NJS702IO-NEBULA-Monitor01: Temporary failure in name resolution
2025-09-09 09:43:14,285 INFO [s.u.MilvusSourceConnectorUtils] [main] - describe collection database: zhengqi, collection: test, response: DescribeCollectionResp(collectionName=test, collectionID=457209259783653913, databaseName=zhengqi, description=, numOfPartitions=1, fieldNames=[id, embedding, text, reference, metadata], vectorFieldNames=[embedding], primaryFieldName=id, enableDynamicField=false, autoID=true, collectionSchema=CreateCollectionReq.CollectionSchema(fieldSchemaList=[CreateCollectionReq.FieldSchema(name=id, description=, dataType=Int64, maxLength=65535, dimension=null, isPrimaryKey=true, isPartitionKey=false, isClusteringKey=false, autoID=true, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=embedding, description=, dataType=FloatVector, maxLength=65535, dimension=1024, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=text, description=, dataType=VarChar, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=reference, description=, dataType=VarChar, maxLength=2048, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=metadata, description=, dataType=JSON, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null)], enableDynamicField=false, functionList=[]), createTime=457339532121997318, consistencyLevel=STRONG, shardsNum=1, properties={})
2025-09-09 09:43:14,331 INFO [s.u.MilvusSourceConnectorUtils] [main] - describe collection database: zhengqi, collection: GandB_Contact_Information, response: DescribeCollectionResp(collectionName=GandB_Contact_Information, collectionID=457209259826775264, databaseName=zhengqi, description=This document contains the contact information of managers for various industries and products of China Telecom., numOfPartitions=1, fieldNames=[id, embedding, text, reference, metadata], vectorFieldNames=[embedding], primaryFieldName=id, enableDynamicField=false, autoID=true, collectionSchema=CreateCollectionReq.CollectionSchema(fieldSchemaList=[CreateCollectionReq.FieldSchema(name=id, description=, dataType=Int64, maxLength=65535, dimension=null, isPrimaryKey=true, isPartitionKey=false, isClusteringKey=false, autoID=true, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=embedding, description=, dataType=FloatVector, maxLength=65535, dimension=1024, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=text, description=, dataType=VarChar, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=reference, description=, dataType=VarChar, maxLength=2048, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=metadata, description=, dataType=JSON, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null)], enableDynamicField=false, functionList=[]), createTime=459312980648787992, consistencyLevel=STRONG, shardsNum=1, properties={})
2025-09-09 09:43:14,347 INFO [s.u.MilvusSourceConnectorUtils] [main] - describe collection database: zhengqi, collection: GandB_products, response: DescribeCollectionResp(collectionName=GandB_products, collectionID=459742145065187890, databaseName=zhengqi, description=The document from China Telecom regarding government and enterprise products provides detailed information about each product, including the product name, contact details of the product manager, a description of the product, the industry it serves, and its target customers. It also highlights the key selling points and advantages, pricing details, successful marketing case studies, marketing pitches, and frequently asked questions to help guide potential customers., numOfPartitions=1, fieldNames=[id, embedding, text, reference, metadata], vectorFieldNames=[embedding], primaryFieldName=id, enableDynamicField=false, autoID=true, collectionSchema=CreateCollectionReq.CollectionSchema(fieldSchemaList=[CreateCollectionReq.FieldSchema(name=id, description=, dataType=Int64, maxLength=65535, dimension=null, isPrimaryKey=true, isPartitionKey=false, isClusteringKey=false, autoID=true, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=embedding, description=, dataType=FloatVector, maxLength=65535, dimension=1024, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=text, description=, dataType=VarChar, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=reference, description=, dataType=VarChar, maxLength=2048, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=metadata, description=, dataType=JSON, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null)], enableDynamicField=false, functionList=[]), createTime=460234832666427404, consistencyLevel=STRONG, shardsNum=1, properties={})
2025-09-09 09:43:14,360 INFO [s.u.MilvusSourceConnectorUtils] [main] - describe collection database: zhengqi, collection: products_solution, response: DescribeCollectionResp(collectionName=products_solution, collectionID=457209259801087600, databaseName=zhengqi, description=Solution materials of ChinaTelecom in 12 government and enterprise industries (industry, transportation, education, finance, agriculture, health, cultural propaganda, key clients, emergency response, law enforcement & public security, government affairs, urban construction) and the product whitelist of ChinaTelecom., numOfPartitions=1, fieldNames=[id, embedding, text, reference, metadata], vectorFieldNames=[embedding], primaryFieldName=id, enableDynamicField=false, autoID=true, collectionSchema=CreateCollectionReq.CollectionSchema(fieldSchemaList=[CreateCollectionReq.FieldSchema(name=id, description=, dataType=Int64, maxLength=65535, dimension=null, isPrimaryKey=true, isPartitionKey=false, isClusteringKey=false, autoID=true, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=embedding, description=, dataType=FloatVector, maxLength=65535, dimension=1024, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=text, description=, dataType=VarChar, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=reference, description=, dataType=VarChar, maxLength=2048, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null), CreateCollectionReq.FieldSchema(name=metadata, description=, dataType=JSON, maxLength=65535, dimension=null, isPrimaryKey=false, isPartitionKey=false, isClusteringKey=false, autoID=false, elementType=None, maxCapacity=null, isNullable=false, defaultValue=null, enableAnalyzer=null, analyzerParams=null, enableMatch=null, typeParams=null)], enableDynamicField=false, functionList=[]), createTime=458156421846925319, consistencyLevel=STRONG, shardsNum=1, properties={})
2025-09-09 09:43:14,378 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - get the CatalogTable from source Milvus: milvus.zhengqi.GandB_Contact_Information,milvus.zhengqi.test,milvus.zhengqi.products_solution,milvus.zhengqi.GandB_products
2025-09-09 09:43:14,393 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSource Plugin from /opt/seatunnel/connectors
2025-09-09 09:43:14,399 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Milvus'} at: file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar
2025-09-09 09:43:14,402 INFO [p.MultipleTableJobConfigParser] [main] - start generating all transforms.
2025-09-09 09:43:14,402 INFO [p.MultipleTableJobConfigParser] [main] - start generating all sinks.
2025-09-09 09:43:14,409 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2025-09-09 09:43:14,410 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Milvus'} at: file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar
2025-09-09 09:43:14,421 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Milvus' with upstream input catalog-table[database: zhengqi, schema: null, table: GandB_Contact_Information]
2025-09-09 09:43:14,428 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Milvus' with upstream input catalog-table[database: zhengqi, schema: null, table: test]
2025-09-09 09:43:14,431 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Milvus' with upstream input catalog-table[database: zhengqi, schema: null, table: products_solution]
2025-09-09 09:43:14,434 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'Milvus' with upstream input catalog-table[database: zhengqi, schema: null, table: GandB_products]
2025-09-09 09:43:14,521 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Start submit job, job id: 1017611102499897345, with plugin jar [file:/opt/seatunnel/connectors/connector-milvus-2.3.8-SNAPSHOT.jar]
2025-09-09 09:43:15,321 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Submit job finished, job id: 1017611102499897345, job name: SeaTunnel_Job
2025-09-09 09:43:15,340 WARN [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-1017611102499897345] - Failed to get job metrics summary, it maybe first-run