[BUG] Transferring large IQ data sets (RFSA, RFSG, RFmx, etc.) is incredibly slow

[niTsileo]

Bug Report

Because many proto files utlize the repeated NIComplexF32 custom data type there is significant slow-down when transferring large data sets (e.g. 5 GSps, few ms of data). For newer, high sample rate devices it is not always possible to avoid these transfers. As an example, when transfering 10 ms of 5GSps IQ data using FetchIQSingleRecordComplexF32Request (nirfsa.proto) this takes ~5 seconds to fetch from the HW and an additional ~10 seconds (or more) to convert to a 1D array of Complex Numbers (to make the data type compatible with numpy). By comparison, the process to return a 1D array of CSGL numbers is only about 500 ms in the LabVIEW RFSA fetch shipping example, not 15-20 seconds!

Note

grpc-device provides .proto files and a server to support calling NI driver APIs over gRPC. Can you reproduce your problem by calling the driver API directly?

Repo or Code Sample

[NI] https://dev.azure.com/ni/DevCentral/_git/mst-adg-rf?path=/General/Examples/gRPC%20RF%20Demos/src/Python/nirfsa-acq-to-file-threaded.py&version=GBmain&_a=contents

RFSA Acquire IQ Data in Blocks.vi shipping example VI as comparison.

Expected Behavior

Current Behavior

Possible Solution

Change the data types to two repeated Floats (one for I and one for Q data). That should help a little bit. Otherwise, investigate sideband IP and/or streams incorporated into the grpc-device api's that may have large data transfers.

Context

Your Environment

• Operating system and version: [e.g. Windows 11 24H2, Ubuntu Linux 24.04, NI Linux RT 2024 Q4]
• All NI driver versions: [e.g. NI-DAQmx 2024 Q4, NI-FGEN 2024 Q4]
• grpc-device version: [e.g. 2.11.0]

AB#3179219

[bkeryan]

Change the data types to two repeated Floats (one for I and one for Q data). That should help a little bit.

Using a single array of interleaved I/Q data is likely to be more performant than deinterleaving/re-interleaving the data. In C/C++, LabVIEW, and Python w/NumPy an array of complex numbers should have the same memory layout as an array of interleaved real/imag values and you should be able to convert between the two representations without copying.

[npowelso]

I'm implemented and tested a fix using the interleaved data, and can confirm performance improvement. code can be found here: https://dev.azure.com/ni/Users/_git/npowelso?path=/Custom%20gRPC

Performance is improved on the server since serializing messages has less overhead with the repeated float, as well as deserialization in python.

Results:
Fetching a single 10M record with the existing message structure, and then with the interleaved data:

Format	Fetch Time (s)	Convert to Complex Array (s)	File IO Time (s)	Total Time (s)
Original	13.24	5.87	0.1936	19.312
Flat (Interleaved)	1.21	1.86	0.18	3.26

EDIT
I want to add that the fetch time and array conversion time for the existing implementation is so long because of the additional memory copies and data handling that is needed. Fetch takes so much longer when it's an array of messages because each message has to be serialized and deserialized individually, whereas with a repeated float that isn't the case. So the fetch time is longer because of the gRPC overhead, which simply by changing to a repeated float reduces fetch time by 10X.

[bkeryan]

@npowelso Converting interleaved to complex should take much less time than the fetch. Use a view to convert a NumPy array of interleaved real/imag to a NumPy array of complex without copying.

>>> import numpy as np 
>>> x = np.array([1.0, 2.0, 3.0, 4.0], np.float32)
>>> y = x.view(np.complex64)
>>> y
array([1.+2.j, 3.+4.j], dtype=complex64)
>>> y.base
array([1., 2., 3., 4.], dtype=float32) 
>>> np.shares_memory(x,y)
True

This assumes that you can also convert the gRPC repeated float to a NumPy array quickly.

[npowelso]

@npowelso Converting interleaved to complex should take much less time than the fetch. Use a view to convert a NumPy array of interleaved real/imag to a NumPy array of complex without copying.
This assumes that you can also convert the gRPC repeated float to a NumPy array quickly.

@bkeryan You are absolutely correct, and that is exactly what I did in my benchmarking python client:

flat = np.array(read_response.data, dtype=np.float32)
complex_array = flat.view(np.complex64)

As you said, there is still a copy being done during the memory allocation and deserialization from the repeated float, which is significantly faster than deserializing an array of messages as is required with the shipping server.