RADE benchmarking

[tmiw]

This is mainly to collect data as to RADEV1's current performance, especially on lower-end hardware / hardware that is representative of what could be found inside modern radios (i.e. Flex 8000 series).

Test Setup

• SBC: Raspberry Pi CM4 on Raspberry Pi official carrier board
• RAM: 1 GB (Note: Flex 8000 series has significantly more than that available.)
• eMMC: 16 GB
• OS: Raspberry Pi OS 'bookworm'
  • uname -a: Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
• Git branch: main
• Using RADE C encoder/decoder (full Python encoder/decoder would not run on the above setup).

TX performance

Input file:

(rade-venv) mooneer@raspberrypi:~/radae $ soxi wav/all.wav

Input File     : 'wav/all.wav'
Channels       : 1
Sample Rate    : 16000
Precision      : 16-bit
Duration       : 00:00:49.76 = 796227 samples ~ 3732.31 CDDA sectors
File Size      : 1.59M
Bit Rate       : 256k
Sample Encoding: 16-bit Signed Integer PCM

(rade-venv) mooneer@raspberrypi:~/radae $

Commands executed:

(rade-venv) mooneer@raspberrypi:~/radae $ time build/src/lpcnet_demo -features wav/all.wav - > tx.tmp

real	0m0.442s
user	0m0.435s
sys	0m0.006s

(rade-venv) mooneer@raspberrypi:~/radae $ cat tx.tmp | PYTHONPATH=$(pwd) time build/src/radae_tx > tx.raw
model file: dummy
import_array returned: 0
model_name: model19_check3/checkpoints/checkpoint_epoch_100.pth bypass_enc: 1
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
n_features_in: 432 n_floats_in: 240 Nmf: 960 Neoo: 1152 n_eoo_bits: 180
loading: radae_rxe
bypass_dec: 0 foff_err: 0.000000
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
Input BPF bandwidth: 1740.000244 centre: 1475.000000
n_features_out: 432 n_eoo_bits: 180 n_floats_out: 432 nin_max: 1120 nin: 960
9.78user 1.00system 0:10.63elapsed 101%CPU (0avgtext+0avgdata 300164maxresident)k
71336inputs+6248outputs (0major+56011minor)pagefaults 0swaps
(rade-venv) mooneer@raspberrypi:~/radae $ 

• FARGAN feature generation: 0.442s
• RADE TX: 10.63s
• Total: 11.07s (~4.5x faster than real-time)

RX performance

Commands executed:

(rade-venv) mooneer@raspberrypi:~/radae $ cat tx.raw | OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 PYTHONPATH=$(pwd) time build/src/radae_rx > rx.tmp
model file: dummy
import_array returned: 0
model_name: model19_check3/checkpoints/checkpoint_epoch_100.pth bypass_enc: 0
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
n_features_in: 432 n_floats_in: 432 Nmf: 960 Neoo: 1152 n_eoo_bits: 180
loading: radae_rxe
bypass_dec: 2 foff_err: 0.000000
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
Input BPF bandwidth: 1740.000244 centre: 1475.000000
n_features_out: 432 n_eoo_bits: 180 n_floats_out: 240 nin_max: 1120 nin: 960
SNR3k (dB): 0
SNR3k (dB): 0
SNR3k (dB): 0
SNR3k (dB): 0
SNR3k (dB): 3
...
SNR3k (dB): 33
SNR3k (dB): 33
SNR3k (dB): 33
SNR3k (dB): 33
37.93user 0.98system 0:39.81elapsed 97%CPU (0avgtext+0avgdata 289712maxresident)k
93368inputs+1400outputs (9major+52846minor)pagefaults 0swaps

(rade-venv) mooneer@raspberrypi:~/radae $ cat tx.tmp | time build/src/lpcnet_demo -fargan-synthesis - rx.raw
4.27user 0.03system 0:04.31elapsed 99%CPU (0avgtext+0avgdata 2508maxresident)k
0inputs+3112outputs (0major+108minor)pagefaults 0swaps

(rade-venv) mooneer@raspberrypi:~/radae $ 

• RADE RX: 39.81s
• FARGAN synthesis: 4.31s
• Total: 44.12s (~1.1x faster than real-time)

[tmiw]

Another attempt after building with cmake -DCMAKE_BUILD_TYPE=Release:

(rade-venv) mooneer@raspberrypi:~/radae $ cat tx.raw | OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 PYTHONPATH=$(pwd) time build/src/radae_rx > rx.tmp
model file: dummy
import_array returned: 0
model_name: model19_check3/checkpoints/checkpoint_epoch_100.pth bypass_enc: 0
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
n_features_in: 432 n_floats_in: 432 Nmf: 960 Neoo: 1152 n_eoo_bits: 180
loading: radae_rxe
bypass_dec: 2 foff_err: 0.000000
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
Input BPF bandwidth: 1740.000244 centre: 1475.000000
n_features_out: 432 n_eoo_bits: 180 n_floats_out: 240 nin_max: 1120 nin: 960
...
38.56user 0.79system 0:39.44elapsed 99%CPU (0avgtext+0avgdata 289500maxresident)k
0inputs+1400outputs (0major+52421minor)pagefaults 0swaps

So basically no difference (but IIRC we're forcing at least -O2 in the CMakeLists.txt regardless of CMAKE_BUILD_TYPE).

Also, Python loading does seem to take a significant amount of time. We probably need to figure out a way to benchmark after Python's fully up to get a good picture.

[tmiw]

Benchmarking just the audio processing

Code changes:

diff --git a/src/radae_rx.c b/src/radae_rx.c
index 4ca62ec..eb5e9d2 100644
--- a/src/radae_rx.c
+++ b/src/radae_rx.c
@@ -7,6 +7,8 @@
 #include <fcntl.h>
 #endif // _WIN32
 
+#include <time.h>
+
 #include "rade_api.h"
 
 int main(int argc, char *argv[])
@@ -39,6 +41,8 @@ int main(int argc, char *argv[])
     _setmode(_fileno(stdout), O_BINARY);
 #endif // _WIN32
 
+    struct timespec start, end;
+    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &start);
     while((size_t)nin == fread(rx_in, sizeof(RADE_COMP), nin, stdin)) {
         int n_out = rade_rx(r,features_out,&has_eoo_out,eoo_out,rx_in);
         if (n_out) {
@@ -51,7 +55,9 @@ int main(int argc, char *argv[])
         nin = rade_nin(r);
         fprintf(stderr, "SNR3k (dB): %d\n", rade_snrdB_3k_est(r));
     }
-
+    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &end);
+    double totalTime = (end.tv_sec - start.tv_sec) * 1e9 + (end.tv_nsec - start.tv_nsec);
+    fprintf(stderr, "Took %f ns to finish processing input\n", totalTime);
     rade_close(r);
     rade_finalize();
     fclose(feoo);
diff --git a/src/radae_tx.c b/src/radae_tx.c
index 55b8b4f..1dc379a 100644
--- a/src/radae_tx.c
+++ b/src/radae_tx.c
@@ -7,6 +7,8 @@
 #include <fcntl.h>
 #endif // _WIN32
 
+#include <time.h>
+
 #include "rade_api.h"
 
 int main(void)
@@ -39,6 +41,9 @@ int main(void)
     _setmode(_fileno(stdout), O_BINARY);
 #endif // _WIN32
 
+    struct timespec start, end;
+    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &start);
+
     while((size_t)n_features_in == fread(features_in, sizeof(float), n_features_in, stdin)) {
         rade_tx(r,tx_out,features_in);
         fwrite(tx_out, sizeof(RADE_COMP), n_tx_out, stdout);
@@ -50,7 +55,11 @@ int main(void)
     // extra silence buf to let Rx finish processing EOO
     memset(tx_eoo_out,0,sizeof(tx_eoo_out));
     fwrite(tx_eoo_out, sizeof(RADE_COMP), n_tx_eoo_out, stdout);
-    
+   
+    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &end);
+    double totalTime = (end.tv_sec - start.tv_sec) * 1e9 + (end.tv_nsec - start.tv_nsec);
+    fprintf(stderr, "Took %f ns to finish processing input\n", totalTime);
+ 
     rade_close(r);
     rade_finalize();

TX time

(rade-venv) mooneer@raspberrypi:~/radae $ cat tx.tmp | PYTHONPATH=$(pwd) build/src/radae_tx > tx.raw
model file: dummy
import_array returned: 0
model_name: model19_check3/checkpoints/checkpoint_epoch_100.pth bypass_enc: 1
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
n_features_in: 432 n_floats_in: 240 Nmf: 960 Neoo: 1152 n_eoo_bits: 180
loading: radae_rxe
bypass_dec: 0 foff_err: 0.000000
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
Input BPF bandwidth: 1740.000244 centre: 1475.000000
n_features_out: 432 n_eoo_bits: 180 n_floats_out: 432 nin_max: 1120 nin: 960
Took 1257253322.000000 ns to finish processing input
(rade-venv) mooneer@raspberrypi:~/radae $

1257253322.000000 / 1e9 = 1.26 seconds (39x faster than real-time)

RX time

(rade-venv) mooneer@raspberrypi:~/radae $ cat tx.raw | OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 PYTHONPATH=$(pwd) build/src/radae_rx > rx.tmp
model file: dummy
import_array returned: 0
model_name: model19_check3/checkpoints/checkpoint_epoch_100.pth bypass_enc: 0
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
n_features_in: 432 n_floats_in: 432 Nmf: 960 Neoo: 1152 n_eoo_bits: 180
loading: radae_rxe
bypass_dec: 2 foff_err: 0.000000
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
Input BPF bandwidth: 1740.000244 centre: 1475.000000
n_features_out: 432 n_eoo_bits: 180 n_floats_out: 240 nin_max: 1120 nin: 960
SNR3k (dB): 0
SNR3k (dB): 0
SNR3k (dB): 0
SNR3k (dB): 0
SNR3k (dB): 3
SNR3k (dB): 6
SNR3k (dB): 9
SNR3k (dB): 11
SNR3k (dB): 13
SNR3k (dB): 15
SNR3k (dB): 17
SNR3k (dB): 19
SNR3k (dB): 20
SNR3k (dB): 22
...
SNR3k (dB): 33
SNR3k (dB): 33
SNR3k (dB): 33
SNR3k (dB): 33
SNR3k (dB): 33
Took 30164841433.000000 ns to finish processing input
(rade-venv) mooneer@raspberrypi:~/radae $

30164841433.000000 / 1e9 = 30.2 seconds (1.65x faster than real-time)

[drowe67]

Deprecated by C port