How can I config node to get good performance of mir

[609127400]

I runned benchmark in 3 physical machines, every machine run 3 mir node, etc. 9 mir nodes totally.
the config of node like as follow:

{
  "Trantor": {
    "Mempool": {
      "MaxTransactionsInBatch": "1024",
      "MaxPayloadInBatch": "1048576",
      "BatchTimeout": "100000000"
    },
    "Iss": {
      "InitialMembership": {
        "Nodes": {
          "0": {
            "Id": "0",
            "Addr": "/ip4/192.168.253.128/tcp/10000",
            "Key": null,
            "Weight": "1"
          },
          "1": {
            "Id": "1",
            "Addr": "/ip4/192.168.253.128/tcp/10001",
            "Key": null,
            "Weight": "1"
          },
          "2": {
            "Id": "2",
            "Addr": "/ip4/192.168.253.128/tcp/10002",
            "Key": null,
            "Weight": "1"
          },
          "3": {
            "Id": "3",
            "Addr": "/ip4/192.168.253.129/tcp/10000",
            "Key": null,
            "Weight": "1"
          },
          "4": {
            "Id": "4",
            "Addr": "/ip4/192.168.253.129/tcp/10001",
            "Key": null,
            "Weight": "1"
          },
          "5": {
            "Id": "5",
            "Addr": "/ip4/192.168.253.129/tcp/10002",
            "Key": null,
            "Weight": "1"
          },
          "6": {
            "Id": "6",
            "Addr": "/ip4/192.168.253.130/tcp/10000",
            "Key": null,
            "Weight": "1"
          },
          "7": {
            "Id": "7",
            "Addr": "/ip4/192.168.253.130/tcp/10001",
            "Key": null,
            "Weight": "1"
          },
          "8": {
            "Id": "8",
            "Addr": "/ip4/192.168.253.130/tcp/10002",
            "Key": null,
            "Weight": "1"
          }
        }
      },
      "ConfigOffset": "2",
      "SegmentLength": "4",
      "EpochLength": "0",
      "MaxProposeDelay": "0",
      "MsgBufCapacity": "33554432",
      "RetainedEpochs": "1",
      "CatchUpTimerPeriod": "1000000000",
      "CheckpointResendPeriod": "1000000000",
      "LeaderSelectionPolicy": "simple",
      "PBFTDoneResendPeriod": "1000000000",
      "PBFTCatchUpDelay": "1000000000",
      "PBFTViewChangeSNTimeout": "4000000000",
      "PBFTViewChangeSegmentTimeout": "8000000000",
      "PBFTViewChangeResendPeriod": "1000000000"
    },
    "Net": {
      "ConnectionBufferSize": "512",
      "ReconnectionPeriod": "3000000000",
      "MinComplainPeriod": "1000000000"
    },
    "Availability": {
      "Limit": "1",
      "MaxRequests": "9223372036854775807"
    }
  },
  "TxGen": {
    "ClientID": "2",
    "PayloadSize": "512",
    "NumClients": "100"
  },
  "Duration": "600000000000"
}

finally, I got result of mir network as follow:

<style> </style>

experiment-id	batch-timeout	clients-node	clients-total	duration-target	max-batch-bytes	max-batch-tx	num-nodes	segment-length	duration-actual	latency-95p	latency-avg	latency-max	latency-median	net-down-avg	net-loss-avg	net-up-avg	throughput
0	0.1	100	900	600	1048576	1024	9	4	602	20.161	5.3315	37.785	4.344	0.0922	0	0.1205	18.5738
1	0.1	100	900	600	1048576	1024	9	4	601	20.98	6.3899	37.621	4.359	0.0975	0	0.1137	15.2841
2	0.1	100	900	600	1048576	1024	9	4	602	17.142	6.2795	36.752	4.41	0.0943	0	0.1073	15.7645
3	0.1	100	900	600	1048576	1024	9	4	602	20.844	7.0576	33.588	4.402	0.0938	0	0.0973	14.0332
4	0.1	100	900	600	1048576	1024	9	4	601	24.423	7.4515	29.417	4.412	0.1021	0	0.0926	13.289
5	0.1	100	900	600	1048576	1024	9	4	600	24.694	8.6646	41.267	4.966	0.0998	0	0.0899	11.4825
6	0.1	100	900	600	1048576	1024	9	4	602	21.712	8.6545	24.569	8.314	0.0984	0	0.0909	11.4428
7	0.1	100	900	600	1048576	1024	9	4	601	24.901	9.0406	49.875	4.958	0.1045	0	0.0835	10.7973
8	0.1	100	900	600	1048576	1024	9	4	602	24.498	8.7784	32.59	8.413	0.098	0	0.0852	11.2769

I wanna known how can I modify the config to improve the performance of mir network, I readly need it!
of course, I wanna run mir network with different node number like 16/32/64, but I need known what's the best config for node's performance firstly.

Thanks!

[matejpavlovic]

Hi, looking at the table, it looks like the nodes might have been overloaded. The latency is on the order of seconds, the tail latency even tens of seconds. What were the hardware specs of the machines? Running 3 nodes on a single machine is generally a lot, unless the machine is huge. The experiments I ran for the Trantor evaluation used a separate machine for every node:

For each replica, we use an Amazon EC2 virtual machine of type t4g.xlarge with 4 virtual Arm-based AWS Graviton2 CPUs and 16GB of memory (even though most of the memory is not used)

[609127400]

Hi, looking at the table, it looks like the nodes might have been overloaded. The latency is on the order of seconds, the tail latency even tens of seconds. What were the hardware specs of the machines? Running 3 nodes on a single machine is generally a lot, unless the machine is huge. The experiments I ran for the Trantor evaluation used a separate machine for every node:

For each replica, we use an Amazon EC2 virtual machine of type t4g.xlarge with 4 virtual Arm-based AWS Graviton2 CPUs and 16GB of memory (even though most of the memory is not used)

Hello,
I reached out to you on GitHub. I am currently writing my thesis and need to run Mir to obtain good performance data for comparison, but I still haven't been able to achieve this. I used https://github.com/consensus-shipyard/mir, compiled the benchmark program by executing make bin/bench, and then successfully manually launched a network.
I ran eight Mir nodes using four machines (each with 8 cores and 32GB of memory), with two Mir nodes on each machine. the configuration files for the Mir nodes and the final results as follow:

{
  "Trantor": {
    "Mempool": {
      "MaxTransactionsInBatch": "2048",
      "MaxPayloadInBatch": "16777216",
      "BatchTimeout": "500000000"
    },
    "Iss": {
      "InitialMembership": {
        "Nodes": {
          "0": {
            "Id": "0",
            "Addr": "/ip4/172.16.0.6/tcp/10000",
            "Key": null,
            "Weight": "1"
          },
          "1": {
            "Id": "1",
            "Addr": "/ip4/172.16.0.6/tcp/10001",
            "Key": null,
            "Weight": "1"
          },
          "2": {
            "Id": "2",
            "Addr": "/ip4/172.16.0.17/tcp/10000",
            "Key": null,
            "Weight": "1"
          },
          "3": {
            "Id": "3",
            "Addr": "/ip4/172.16.0.17/tcp/10001",
            "Key": null,
            "Weight": "1"
          },
          "4": {
            "Id": "4",
            "Addr": "/ip4/172.16.0.26/tcp/10000",
            "Key": null,
            "Weight": "1"
          },
          "5": {
            "Id": "5",
            "Addr": "/ip4/172.16.0.26/tcp/10001",
            "Key": null,
            "Weight": "1"
          },
          "6": {
            "Id": "6",
            "Addr": "/ip4/172.16.0.5/tcp/10000",
            "Key": null,
            "Weight": "1"
          },
          "7": {
            "Id": "7",
            "Addr": "/ip4/172.16.0.5/tcp/10001",
            "Key": null,
            "Weight": "1"
          }
        }
      },
      "ConfigOffset": "2",
      "SegmentLength": "4",
      "EpochLength": "0",
      "MaxProposeDelay": "0",
      "MsgBufCapacity": "33554432",
      "RetainedEpochs": "1",
      "CatchUpTimerPeriod": "1000000000",
      "CheckpointResendPeriod": "1000000000",
      "LeaderSelectionPolicy": "simple",
      "PBFTDoneResendPeriod": "1000000000",
      "PBFTCatchUpDelay": "1000000000",
      "PBFTViewChangeSNTimeout": "4000000000",
      "PBFTViewChangeSegmentTimeout": "8000000000",
      "PBFTViewChangeResendPeriod": "1000000000"
    },
    "Net": {
      "ConnectionBufferSize": "512",
      "ReconnectionPeriod": "3000000000",
      "MinComplainPeriod": "1000000000"
    },
    "Availability": {
      "Limit": "1",
      "MaxRequests": "9223372036854775807"
    }
  },
  "TxGen": {
    "ClientID": "0",
    "PayloadSize": "512",
    "NumClients": "1000"
  },
  "Duration": "300000000000"
}

<style> </style>

experiment-id	batch-timeout	clients-node	clients-total	duration-target	max-batch-bytes	max-batch-tx	num-nodes	segment-length	duration-actual	latency-95p	latency-avg	latency-max	latency-median	net-down-avg	net-loss-avg	net-up-avg	throughput
1	0.2	0	0	300	16777216	2048	8	4	300	0	0	0	0	0.8854	0	0.1985	0
2	0.1	1000	8000	300	16777216	2048	8	4	301	15.944	2.4187	16.104	0.526	0.6333	0	1.6533	394.0397
3	0.4	0	0	300	16777216	2048	8	4	302	0	0	0	0	0.8911	0	0.1557	0
4	0.1	1000	8000	300	16777216	2048	8	4	301	15.953	2.3981	16.097	0.527	0.6469	0	1.5798	397.351
5	0.5	0	0	300	16777216	2048	8	4	302	0	0	0	0	0.8112	0	0.1574	0
6	0.2	1000	8000	300	16777216	2048	8	4	301	16.075	7.0225	39.631	4.852	0.7859	0	0.632	135.7616
7	0.2	0	0	300	16777216	2048	8	4	302	0	0	0	0	0.8558	0	0.1786	0

I think there are still issues with the above configuration, because the results I obtained are far from satisfactory.
I read the paper you mentioned (https://github.com/consensus-shipyard/trantor-doc/blob/main/main.pdf), but I only found a few configuration parameters in Section 7.2, such as MaxPayloadInBlock = 16MB and MaxTransactionsInBlock = 2048. I used these configurations, but I still couldn't get good results.
So I still need your help. Do I need to change the following configuration items? To be honest, I don't quite understand how to modify these configuration items to achieve better results (low latency + high throughput).

  "ConfigOffset": "2",
  "SegmentLength": "4",
  "EpochLength": "0",
  "MaxProposeDelay": "0",
  "MsgBufCapacity": "33554432",
  "RetainedEpochs": "1",
  "CatchUpTimerPeriod": "1000000000",
  "CheckpointResendPeriod": "1000000000",
  "LeaderSelectionPolicy": "simple",
  "PBFTDoneResendPeriod": "1000000000",
  "PBFTCatchUpDelay": "1000000000",
  "PBFTViewChangeSNTimeout": "4000000000",
  "PBFTViewChangeSegmentTimeout": "8000000000",
  "PBFTViewChangeResendPeriod": "1000000000"

I would really appreciate it if I could get your response.