New gain controls

[fventuri]

Given the diversity of opinions on how the gain controls should work (see discussion here: #27), I spent some time over the weekend thinking about a possible way of dealing with all the different tastes and preferences, and I just pushed my changes to the new-gain-controls branch.

Basically I created four different 'models' of gain controls, selectable via a line in the cmake config file CMakeLists.txt (https://github.com/pothosware/SoapySDRPlay3/blob/new-gain-controls/CMakeLists.txt#L43).

The four models are:

• Legacy: this one behaves exactly like the code currently in master (IFGR and RFGR, no "generic" setGain(), etc); if these changes land in master, the default could be initially set to 'Legacy' for compatibility with the existing clients; this way the distributions would use the 'Legacy' mode by default, and there would be little to no disruption to the end users.
• SDRplay: this one is exactly the new gain controls code that @SDRplay wrote almost a year ago (see here: Customising gain sliders, etc. cjcliffe/CubicSDR#825), and it could be used to compare the other gain control models with his proposal.
• DB: this is a new one I added based on our discussion last week; the gain controls are just RF and IF, and the values are expressed in dB (higher values mean more gain, i.e. stronger signal). Since the two gain control values use the same units, I also added a generic setGain() (and getGain()), where the requested gain is split between RF and IF in parts proportional to the range of each gain control (see the source code for details). The dB range of these "gains" is arbitrary: for IF I chose the range [20,59], while for RF I chose the range [X,100] (where X depends on the RSP model, frequency band, etc). For those who prefer negative values like @dlaw, commenting out lines 49-51 in CMakeLists.txt will do just that.
• LNA: finally this is another new gain control model I added based on @vsonnier's wish to be able to see and use the raw LNA state: the gain controls in this case are LNA and IF(higher values mean more gain, i.e. stronger signal). The default range for IF is the same as before ([20,59]), while the default range for LNA is [0,max(LNAstate)]. Since these values use incompatible units, I did not define a generic setGain() (or getGain()) for this case. For those who prefer negative values, again commenting out lines 46-48 in CMakeLists.txt will do the trick.

To make it easier to work with all the different gain control models (and perhaps to add more if we like to explore other options), I pulled out of Settings.cpp all the code for the gain API, and reorganized it in four different source files, one for each of these gain control models (plus one for some common code related to the gain reduction tables from the SDRplay API specification document).

One final note about the gains in dB: I spent some time yesterday afternoon playing around with SDRuno, switching the RF gain display mode back and forth between 'Gain' and 'Attenuation', and I noticed that while the dB values for RF attenuations in SDRuno match exactly the values in the gain reduction tables in the SDRplay API specification, the same is not true when the RF gain display mode is set to 'Gain': the dB values in this case have decimals, their spacing is different than shown in the tables (for instance the gain reduction tables might have 7dB difference between two successive LNA state values, while the SDRuno RF gain changes 6.5dB between the same LNA states); the minimum and maximum values depend on the frequency band, the RSP model, and possibly other factors.
In other words I have the impression that SDRplay have done a good amount of work in figuring out the real gains in dB (and those are displayed in SDRuno when the RF gain control is set to 'Gain'), while the figures from the gain reduction tables are just rough approximations, and it is not obvious to me how the two are related, i.e. how to compute the real RF gain from the LNA state (I searched around for a bit, but I couldn't find much information about these RF gain values).
As a consequence the RF "gains" in dB of the third gain control model (the one I called 'DB') are not the real gains, and shouldn't be treated as such; they are just some arbitrary values in dB that "approximate" the behavior of the real gain, in the sense that increasing the RF gain control by 10dB increases the signal by about 10dB, but the RF gain value shown is purely arbitrary (I hope this rambling makes sense).

One last caveat when using CubicSDR with the new gain controls: CubicSDR stores the last used config under $HOME/.CubicSDR/config.xml (on Linux; there's probably a similarly named file on Windows and Mac); in my case this file had the last used gains controls as IFGR and RFGR, and threw a few error messages about invalid gain range when starting the RSP, because those names are different in some of the gain control models; if you see similar error messages, you may want to recreate the CubicSDR config.xml file from scratch (or edit it to change the names and values of the gain controls, if you don't want to lose the rest of the CubicSDR configuration).

@SDRplay, @vsonnier, @dlaw, @nmaster2042, please give it a try and let me know what you think (I just ran a few quick tests here streaming a local FM station, and they worked, but there are probably still bugs in the code).

Franco

[dlaw]

Wow! That is quite an amazing piece of work! Thank you @fventuri !

I will give it a test run sometime over this next week, but in the meantime, a comment about the "generic" setGain and getGain functions:

SoapySDR provides a default implementation of the "generic" setGain and getGain functions which SoapySDRPlay inherits:
https://github.com/pothosware/SoapySDR/blob/master/lib/Device.cpp#L292

In the case where all gains are in dB, the default implementation is already correct and we do not need to implement these functions ourselves. (Well, technically we may wish to tweak setGain a bit to change how the gain is split between IF and RF, but the default is certainly good enough.)

It is in the other three cases where we do need to implement our own generic setGain and getGain in order to prevent the inherited setGain from always setting the LNA state to 9 in a misguided attempt to distribute the gain.

[dlaw]

I think the IF gain range from 20 dB to 59 dB is a bit of an anti-pattern, and if you must avoid the negative numbers, a range from 0 dB to 39 dB would be better. Here's why:

An IF gain range of -59 dB to -20 dB will be confusing to a user who does not expect negative numbers, but at least the correspondence with IFGR value will be obvious for any user who uses SDRuno or is looking at SDRPlay documentation: IF -20 is IFGR 20, IF -30 is IFGR 30, etc.

An IF gain range of 20 dB to 59 dB eliminates the confusion with negative numbers, but a user could easily look at the range from 20 to 59 and conclude that this IF gain is exactly the same as the IFGR value from the datasheet! Whereas I think few users will readily deduce that e.g. an IF gain of 30 turns into an IFGR of 49.

So, for the case where you don't want to use negative numbers, I think it is better for IF gain to go from 0 to 39. This makes it obvious that the IF gain is something different than the IFGR and removes the possibility for confusion.

I still think the negative gains are cleanest of all, but 🤷

I will test out the options shortly and report back to confirm functionality and share any notes on usability.

Cheers!
David

[fventuri]

@dlaw - many thanks for your comments and for giving it a try so quickly.

A few comments:

• the offset for the IF gain scale can be changed in CMakeLists.txt (line 50): if you set -DGAIN_MODE_IF_OFFSET_DB=59 there, then the IF gain range will be [0,39] - the combinations of these different choices are almost endless, feel free to play around with these parameters, and we can evaluate together the various options in the next few days
• I forgot to mention earlier in my long post that I also looked at how SoapySDR defines the default generic setGain() (https://github.com/pothosware/SoapySDR/blob/master/lib/Device.cpp#L292-L316) and getGain() (https://github.com/pothosware/SoapySDR/blob/master/lib/Device.cpp#L323-L333), and I agree with you that those two functions do not yield the expected result when the gains (or gain reductions) use different units
• on the other hand I am strongly against providing such generic functions in the case of non-homogeneous variables, therefore I am seriously considering to implement them so that they just throw a runtime error in cases where they do not make sense. As discussed in the past (see here: Additional optional sample rates in SoapySDRPlay driver #16 (comment)), exceptions are the way a SoapySDR module can inform the client application about a problem (for instance invalid input, or unsupported request)
• lastly, thinking about your comments, and reading the source code for the default implementation of getGain(), gave me the idea for yet another gain control model, that I called 'Normalized', with the usual two gain controls IF and RF, but this time their range goes always from 0 to 100, where 0 means minimum gain (maximum attenuation), and 100 maximum gain (minimum attenuation). This approach has the merit, in my opinion, that it gets rid of all the dubious aspects of the gain in dB (what it actually means); it is also very easy to understand for the casual user (think volume knob on a radio or TV), and since the normalized values are by definition homogeneous, having a generic setGain() makes sense and it is very easy to implement (literally two lines of code), so I went ahead and implemented this option too; I pushed the code to the new-gain-controls branch a few minutes ago, so if you refresh it, you should be able to compare this approach too.

Franco

[SDRplay]

I've been reading the gain thread with great interest. Fundamentally the RSPs have 2 gain systems with a single hardware AGC system. How this is all implemented in SoapySDRPlay isn't something that you need my input on. All I would do is point out some things that we have found in the development of SDRuno over the last few years...

1. The way that 90+% people use the system is to leave the IFAGC enabled and just use the RF gain slider. The new AGC scheme in the API 3 is way better than the one used in API 2 and doesn't "bounce" like it could do before.
2. When the IFAGC is enabled, setting the IFGR has no effect.
3. Given 1 and 2, setting a single gain value for the entire system doesn't work when IFAGC is enabled.
4. SDRuno will always display the gain sliders going from min to max for the gain type and then deal with any translation internally. For example IFGR is shown from 20 to 59 and IF gain is also shown as 20 to 59 - just that internally the IF gain value is inversed to convert to IFGR before sending to the API
5. The gain value displayed in SDRuno comes back from the API - unlike the gain reduction value which can be calculated.
6. Specifying RF gain as a dB value doesn't really work - for different RSPs and LO frequencies there are different gain steps and different gain ranges. 0 - 9 might work for the RSP1A, but the RSPdx supports 0-27 in some circumstances, so 0 - 9 would only allow about 50% of gain control range. The RSP1 only supports 0 - 3 so the API will error if 4 - 9 is used.

One thing that I looked at doing once for the TCP server was to map each RSP/LO frequency combination to an appropriate IFGR + RGFR value for a given single gain value. Of course this only works for when the IFAGC is off, but if the IFAGC is enabled, you could disable it, set the IFGR+RFGR and then enable the IFAGC - this would have the effect of putting the user close to where they wanted to be. If they want total gain control then the IFAGC would need to be disabled. There were some other issues that occurred in that piece of work that meant the single gain value work was not completed or implemented.

If you wanted to mimic how SDRuno works then you would need to have 2 independent gain control systems that could be individually set to have AGC or not. Then some code that internally could report back to the user on what GR or gain has actually been set.

Also, I think the key thing is to remember what the target audience is for this library. It's developers and not end users. I personally would look at popular end user software, see how they have been developed to work in gain and then work out the best way to interface to that with minimal impact.

Just my two penneth.

Best regards,

Andy

[fventuri]

@SDRplay - thanks for the interesting and useful thoughts, Andy

I have a few personal considerations about your last comment, where you say that the target audience is developers and not end users.

I suspect that a SoapySDR module like SoapySDRPlay3 is mostly used within client applications like CubicSDR or gqrx; these SDR receiver programs cater to a wider audience than just developers (in my opinion), as opposed to a software like GNU Radio, where the average user is probably someone more technical and more familiar with the concepts of SDRs.
In addition both CubicSDR and gqrx (and most of the applications that build on the SoapySDR framework) are by their own nature pretty 'generic' (i.e. not developed with a specific set of devices in mind, like SDRuno), and therefore they have to try to accomodate the many different controls (and quirks) of a disparate set of devices, from the $20 USB dongles to say a USRP, that can cost several thousand dollars, and everything in between.
Because of this variety of SDRs, many of these applications choose to expose to the end user the same gain controls that the SoapySDR (and therefore SoapySDRPlay3) shows them.

This whole preamble is to say that I think we have to put ourselves in the shoes of the end user when making decisions on these controls: which controls could be more useful to them, easier for them to understand, or just "make sense".

On the other hand, as a developer of a module like SoapySDRPlay3, I am bound to the contract that is the SoapySDR API; namely in this case to the description of the arguments of setGain() and return value of getGain() in Device.hpp (https://github.com/pothosware/SoapySDR/blob/master/include/SoapySDR/Device.hpp):

\param value the new amplification value in dB

\return the value of the gain in dB

In closing, what I am trying to say is that whatever we end up choosing will be some sort of trade-off, and there will always be someone who won't be 100% happy with it.
I personally like to experiment and play around with the different options, and I am really looking forward to your and the others feedback.

Franco

[SDRplay]

Hi Franco,

What I meant by that is that of course the end user will need to be able to set frequency, sample rate, gain, etc. however they will not care how that is handled by the SoapySDRPlay library. It is the job of the developer to understand these concepts and present them to the end user.

For example when we point people to CubicSDR and the SoapySDR framework, almost all of them won't know or care what the SoapySDRPlay library does, just that they can control the hardware through the CubicSDR interface.

For the large part, and I'm not saying this is you, what I've seen here is "how do we get the SDRplay gain control to operate like a RTL device does" - if that's the end goal then that's one thing, but that's not to understand what benefits the RSP gain control system has.

We have many thousands of customers that use various software and I would say a lot less than 1% of them want, or have the coding knowledge, to understand or use the SoapySDR framework in anything other than an end application.

From our experience you guys are the exception rather than the rule. If you see what I mean.

Best regards,

Andy

[fventuri]

@SDRplay - Andy, thanks for your always welcome advice and for keeping me (and this group) grounded down to earth, i.e. to the reality of the actual end users.

I followed your suggestion of taking a look at what the other popular SDR applications are doing regarding the gain controls for the SDRplay RSPs; since I think that 90% of the SDR end users run Windows, I looked at what is the approach for gains in the most popular SDR applications for Windows:

• SDRuno: we are all very familiar with this one. The RF gain control can display either attenuation (0-varies) or gain (from the API); the RF gain control has little 'ticks' that match the LNA state. The IF gain control is not in the main panel (AGC is enabled by default); one has to go to the 'Settings' menu and the control (from 20 to 59) is only displayed when the AGC is disabled
• SDR console: the RF gain goes from 0 to the max LNA state, depending on the RSP device only (i.e. for the RSPdx it always goes from 0 to 27, regardless of the frequency band); 0 is the lowest attenuation (maximum gain). The IF gain control goes from -20 (minimum attenuation; maximum gain) to -59 (maximum attenuation; minimum gain). A note informs the user that the IF gain does not have any affect when the AGC is enabled
• HDSDR: HDSDR (and other SDR applications) use the ExtIO plugin approach, which is a concept similar to the SoapySDR modules. The ExtIO plugin for the RSPdx (developed and distributed by SDRplay) has a control called 'LNA' that shows the LNA state directly (and the range of that control does depend on the frequency band), and another control called 'Gain Control' whose range is 20-59, and it is grayed out when the AGC is enabled

I also googled terms like 'sdrconsole gain controls', and there seems to be some confusion among the end users on how these controls work (see for instance here: https://sdrplay.com/community/viewtopic.php?t=3524 and here: https://sdr-radio.groups.io/g/main/topic/16932930). I found similar results when googling 'hdsdr sdrplay gain controls' (for example: https://www.sdrplay.com/community/viewtopic.php?t=356).

Finally I implemented yet another gain control model, that I called 'NO_IF_GAIN_CONTROL_ON_AGC' for lack of a better term; it is essentially the same as the 'DB' gain control model, except that I changed listGains() not to return 'IF' whenever the AGC is enabled, and the other relevant functions (getGainRange(), getGain(), setGain()) to return an error message back to the application when invoked with the 'IF' gain control if the AGC is enabled.
I ran a few quick tests with CubicSDR, but it seems to get confused when the list of available gain changes (i.e. when I start it , it displays correctly both 'IF' and 'RF'; then, if I turn AGC on, they both disappear; after that, if I turn AGC back off again, I only see the 'RF' gain control, but it does not seem to act properly when I try to change RF gain).
I pushed this latest gain control model too to the new-gain-controls branch in the repo, so you can try it out if you like.

Franco

[vsonnier]

Thanks @fventuri for your dedication. I gave some try on your branch, and the most usable was the LNA one from me (Ha!) Here are my random remarks, in no particular order:

• I don't see the the point of 2 variantsdB or LNA for me, is RF control not the same as a LNA state, minus the reverse scale maybe ?
• The setting RF control is not consistent with either LNA or RF gains, they must reflect on each other : Whatever RF or LNA gains are, corresponding settings (readSetting / writeSetting) must be maintained and doing identical things as the gains do.
• In the LNA variant, the RF control should be renamed LNA state and doing the same thing as the LNA gain slider.
• All other variants (except Legacy) have funky or non-functional gains, probably because of the read-write properties of the gains:
  a) Idle control : the gains are re-read and displayed by requesting the corresponding getGain. This happens on the GUI callback OnIdle(wxIdleEvent &event) so roughly at every screen refresh : a 'realtime' update of sorts.
  b) User changing the gain with mouseclick or using mousewheel : The requested gain is written by setGain, then for 150 ms getGain is purposefully not used to refresh the control, in order to let the gain settle then we return to a).
• RfGainArg.type = SoapySDR::ArgInfo::INT; while it was STRING before, which leads to Cubic opening an additionnal dialog to set the value, which is slightly less efficient. Using STRING make Cubic consider it is an enumeration, which only leads to a submenu with a radio-control list. A minor point rally, and maybe more practical for the other SoapySDR clients.

About your last message:

I ran a few quick tests with CubicSDR, but it seems to get confused when the list of available gain changes

Good heavens, CubicSDR certainly didn't expect the gain list to change dynamically. I'll have a look If it can be changed when I have some time.

if I turn AGC on, they both disappear

Standard procedure for Cubic, yes. It has been discussed in the past if we can add configurable behaviour for the Gain controls, i.e choosing which one you want to hide if AGC is on. It is certainly possible to do so, but I'm lacking the motivation to do it myself. Is it not a niche feature anyway ?
The problem also originates from SoapySDR which has no concept of AGC on per-gain, only globally. That would solve the issue to have per-gain AGC and it could be implemented generically in Cubic but 🤷

I only see the 'RF' gain control, but it does not seem to act properly when I try to change RF gain).

See the points above :)

[fventuri]

@vsonnier - many thanks for your feedback and useful suggestions!

I finally found some time this weekend to fix the problem with the settings selection for the RF Gain (or RFGR, or LNA, depending on the gain control model); it is now back to a string type (instead of an integer), so it should be easier to use, and the avalable selections now match the correct options for each of the gain control models.
I just pushed the code to the new-gain-controls branch; please give it a try and let me know what I missed (or what doesn't work as expected).

On the AGC front, I knew it was a very tall order for CubicSDR (and probably for any other SDR application out there) to be able to deal with a list of gains that changes dynamically, but I was just too curious to see what would happen - so don't worry about try to handle a gain list changing dynamically, since I doubt any other SDR client would be happy about it anyway (I removed that gain control model with this latest commit).
Another option that I am exploring now is to leave the IF gain control even when the AGC is enabled, but have it return a gain range (in getGainRange()) of just one value (the current value of gRdB) when the AGC is enabled. To play around with this idea, tonight I created a new gain control model called LNA_AGC, which is just a copy of LNA with this one change for the IF gain range.
I ran a couple of very quick tests with CubicSDR, enabling and then disabling AGC, and it seems that after the AGC is back to disabled (i.e. the IF gain control is available), the IF gain slider only displays one value (probably the one that was fixed when the AGC was on), but it seems to be still changing the IF gain value.

Franco

[fventuri]

Working on the RF control via a SoapySDR setting yesterday, gave me the idea of a possibly better approach for handling IF AGC; instead of using SoapySDR hasGainMode(), setGainMode(), and getGainMode() functions, I decided to try to implement enabling/disabling IF AGC as a boolean SoapySDR setting (ifagc_ctrl), and leave getGainRange() unchanged from the original.

With this approach, and a small change to the CubicSDR source code (see below), I was able IMHO to have a much better user experience using the IF AGC with CubicSDR.

This new approach can be enabled by setting the cmake variable GAIN_MODE_IF_AGC_AS_SETTING to True in line 45 of CMakeLists.txt, and it is available only for the new 'LNA', 'DB', and 'Normalized' gain control models. The new code is in the usual new-gain-controls branch.

As per the CubicSDR source code change mentioned above, I had to change lines 1028-1034 of AppFrame.cpp (https://github.com/cjcliffe/CubicSDR/blob/master/src/AppFrame.cpp#L1028-L1034) from this:

    agcMenuItem = nullptr;
    if (soapyDev->listGains(SOAPY_SDR_RX, 0).size()) {
        agcMenuItem = newSettingsMenu->AppendCheckItem(wxID_AGC_CONTROL, "Automatic Gain");
        agcMenuItem->Check(wxGetApp().getAGCMode());
    } else if (!wxGetApp().getAGCMode()) {
        wxGetApp().setAGCMode(true);
    }

to something like this:

    agcMenuItem = nullptr;
    if (soapyDev->hasGainMode(SOAPY_SDR_RX, 0)) {
        agcMenuItem = newSettingsMenu->AppendCheckItem(wxID_AGC_CONTROL, "Automatic Gain");
        agcMenuItem->Check(wxGetApp().getAGCMode());
    } else if (!soapyDev->listGains(SOAPY_SDR_RX, 0).size()) {
        wxGetApp().setAGCMode(true);
    }

Before trying the new GAIN_MODE_IF_AGC_AS_SETTING with CubicSDR, you may have to either recreate your CubicSDR config file ($HOME/.CubicSDR/config.xml), or edit it to make sure that agc_mode is set to 0 (instead of 1).

Franco

[fventuri]

For the cases where the client application can only use the generic setGain() and getGain() functions (i.e. the ones without a gain control name), as for instance trunk-recorder mentionedby @dlaw, I thought today of a possible approach to convey both the desired RFGR value and the desired IFGR value in the so called 'Legacy' gain control model, similar to the way we specify (military) time: for instance when we say 1259, we mean 12 hours and 59 minutes (after midnight).
Using this convention, a generic gain value of 1259 would mean: set the RFGR (LNA state) to 12 and the IFGR to 59.
Also with this approach an IFGR of say 0 could mean to enable the AGC; therefore a generic gain value of 1200 would mean RFGR=12 and IF AGC on
Setting/getting an RFGR/LNA state of 0 might be a little confusing though, because it would just be '0059', i.e. 59 (for the case of an IFGR=59).
Also since the gain value in setGain() and getGain() is defined as a double, another possible option could be to pass RFGR and IFGR as 'RFGR.IGFR', for instance 12.59 in the case above; with this option the case where the RFGR/LNAstate=0 would be more explicit, because it would look like 0.59.

Tonight I went ahead and implemented both options for the 'Legacy' gain control model; you can try them by editing the cmake config file CMakeLists.txt, and changing line 46 where it says 'GAIN_MODE_LEGACY_GENERIC_GAIN'; the values can be:

• None (same behavior as currently in master where the SoapySDRPlay3 module does not implement the generic setGain() and getGain())
• RFGRIFGR - which is the '1259' option presented above
• RFGR.IGFR - which is the '12.59' option presented above

The code with these options is in the usual new-gain-controls branch in the repository; play around with it, and let me know what you think.

Franco

[dlaw]

Thanks again @fventuri for implementing all of these different options. And thanks to @SDRplay for the insight about how users tend to adjust the gains. I played around with all the different options and here are some thoughts:

1. GAIN_MODE_IF_AGC_AS_SETTING is a big improvement in user experience. It makes perfect sense to see IF gain still present, but automatically changing when IF AGC is enabled. 👍
2. With all the different issues about setGain and getGain, my personal preference is for LNA state to be a menu item only and for IF gain to be the only gain returned by listGains. Nice and simple. Users should already be familiar with the concept of visiting the menu for front-end settings, because that is also where we find bias-T control and the various notch filters.
3. But some users will wish for the convenience of setting the RF gain using a RF gain slider... in this case, I find "DB" mode to be the most preferable. I think there are just too many issues to ever permit a gain slider that is not in decibels.

It seems that @fventuri has the code well in hand, but once it is decided which of these options to merge, I volunteer to write some documentation about gains for the SoapySDRPlay wiki ✍️

Cheers!

[fventuri]

Thanks for your feedback @dlaw

This weekend I thought that with so many choices on the gain controls it is almost impossible to make everyone happy.

Perhaps a better approach would be to select the more sensible and useful models among them, and create a a new 'meta' setting called say 'gain_control_model' (or something like that), whose values could be something like 'Legacy', 'DB', 'LNA', etc. This meta setting would not be listed by getSettingInfo() to avoid applications like CubicSDR to expose it in their UI, but it would be available via the readSetting() and writeSetting() methods.

This way a more advanced user could set up the gain control as they prefer - they would have to just add an entry for that setting in the CubicSDR config.xml file with their preferred choice.
Since the code in the new-gain-controls branch is already organized with one source code file for each gain control model, it wouldn't be too much work to implement this approach (and a user with enough source code experience could probably add their own gain control model very easily).

That's just an idea that popped up in my head; if there's a general consensus that it is worth trying, I could work on it this coming weekend.

Franco

[nmaster2042]

@fventuri

Sorry, for late reply.

I see a lot of development on gain modes, how can I select one ?
Is it at compile time ?