How are we doing?

[katieannemills]

Since Hamburg, the number of qctests implemented has more than doubled (thanks @s-good and @castelao!) - it might be interesting to run the complete stack on quota again, to see how much of an impact we've made on our 55% true positive rate since then. I'm happy to do this, but it also might be good for someone other than me to go through the procedure of deploying and running on AWS (or wherever), so more than one person is familiar with how to do it, and we can double check the instructions are complete and clear.

[s-good]

Good idea! I'm pretty busy here so don't know if I'll get chance in the near future, so it might be best for you to do it if you are happy with that.

[katieannemills]

No problem! I want to finish the track check first (getting there finally), then this.

[s-good]

Hi @BillMills, I was thinking we should put together a list of things we want to achieve before May. I thought of:

1. Add checks using real examples for the EN std level check.
2. Extract/convert the datasets we are going to use for the testing.
3. Implement the CSIRO tests.
4. At a stretch, add the objective analysis test from CORA dataset.

I'll need to do the first one of course and the second should be in hand. I think the fourth one would be difficult to implement but could be possible with help from the CORA team. We should probably do everything else and then see if we have time to do it. How are the CSIRO tests going? Is there anything I can do to help?

[katieannemills]

Hi @s-good - sounds like a plan to me! Also on the topic of running the full stack, I've got a test run going on AWS as we speak, but there are a couple issues:

• the ICDC tests break for me on AWS; before I spend a bunch of time trying to debug, were there any new dependencies (code or data) introduced by these? This is almost certainly the cause.
• Currently test-running on 1/30th of quota in a single thread on the free tier, we've just passed the 20-hour mark (and that's without the ICDC tests). This is fine for an unparallelized dry run, but if this speed estimate is correct (not sure if it is, could just be hanging...), that implies we're getting up in the $30-$40 range per run, which is not a cost I can eat. Might be time to look into institutional infrastructure; I might be able to pull in the earth & ocean science folks at UBC who could then get us access to Compute Canada resources - let me know if there is a preferred option.
• CSIRO: sure, I'll have another go there. There's currently an open PR with a looooong list of clarifying questions after my first attempt to port the FORTRAN implementations. Let me take another look there to see if fresh eyes will solve anything, after which I'll try and condense the list of questions to something a little less sprawling (likely have time for this before next week).

[s-good]

One of the ICDC tests requires a new data file which is in the docker image but the rest should run as they are. Do you have the error message?

If it is going to cost that much on AWS I agree we should look into other options. I can try to run it at my work - it may take some time to get it all run but I think it should be possible.

[s-good]

Also, I think if it is running very slow we should not attempt the CORA test as I think it would be very resource heavy to run. We could consider, as a one off, asking them to run the profiles through their system and return the results to us. If it turns out to be very good we could implement it at a later date.

[katieannemills]

A few updates:

• things are actually not nearly as bad as I feared on AWS; turns out the free tier which I was using to do a 'dress rehearsal' actually throttles CPU usage in addition to only allocating one CPU, so that's why things ground to a halt.
• On a proper compute instance, we're back to a couple hours running for the cost of a cup of coffee (sorry for the false alarm).
• EN_increasing_depth check, however, sometimes hangs for a very long time (minutes to possibly indefinitely) - this is next on my list to investigate.

[katieannemills]

In 2-3 hours on AWS c3large:

Number of profiles tested was 12787

                       NAME OF TEST   FAILS     TPR     FPR     TNR     FNR
            Argo_global_range_check     102   16.0%    0.0%  100.0%   84.0%
                 Argo_gradient_test     186   27.4%    0.1%   99.9%   72.6%
          Argo_impossible_date_test       0    0.0%    0.0%  100.0%  100.0%
      Argo_impossible_location_test       0    0.0%    0.0%  100.0%  100.0%
      Argo_pressure_increasing_test    1696   11.6%   13.4%   86.6%   88.4%
           Argo_regional_range_test       5    0.6%    0.0%  100.0%   99.4%
                    Argo_spike_test      19    3.0%    0.0%  100.0%   97.0%
                        CSIRO_depth    9473   83.7%   73.6%   26.4%   16.3%
                     CSIRO_gradient    2036   43.8%   14.5%   85.5%   56.2%
                   CSIRO_wire_break     161   25.2%    0.0%  100.0%   74.8%
      CoTeDe_Argo_density_inversion    6028   34.9%   47.8%   52.2%   65.1%
          CoTeDe_GTSPP_WOA_normbias    2953   85.0%   19.8%   80.2%   15.0%
          CoTeDe_GTSPP_global_range     103   16.1%    0.0%  100.0%   83.9%
              CoTeDe_GTSPP_gradient     123   17.5%    0.1%   99.9%   82.5%
           CoTeDe_GTSPP_spike_check      24    3.3%    0.0%  100.0%   96.7%
                         CoTeDe_RoC     174   24.9%    0.1%   99.9%   75.1%
                CoTeDe_WOA_normbias    1328   61.2%    7.7%   92.3%   38.8%
           CoTeDe_anomaly_detection    2180   51.3%   15.2%   84.8%   48.7%
             CoTeDe_digit_roll_over     294   28.6%    0.9%   99.1%   71.4%
                 CoTeDe_fuzzy_logic    1826   74.5%   11.1%   88.9%   25.5%
                    CoTeDe_gradient     131   18.8%    0.1%   99.9%   81.2%
        CoTeDe_location_at_sea_test       6    0.5%    0.0%  100.0%   99.5%
             CoTeDe_profile_envelop     185   29.0%    0.0%  100.0%   71.0%
                       CoTeDe_spike      14    2.2%    0.0%  100.0%   97.8%
               CoTeDe_tukey53H_norm      68    6.6%    0.2%   99.8%   93.4%
      EN_background_available_check      15    1.9%    0.0%  100.0%   98.1%
                EN_background_check     931   43.8%    5.4%   94.6%   56.2%
            EN_constant_value_check       1    0.2%    0.0%  100.0%   99.8%
                     EN_range_check     102   16.0%    0.0%  100.0%   84.0%
            EN_spike_and_step_check      17    2.5%    0.0%  100.0%   97.5%
          EN_spike_and_step_suspect     210   29.1%    0.2%   99.8%   70.9%
                 EN_stability_check      28    1.3%    0.2%   99.8%   98.7%
                 WOD_gradient_check     266   30.0%    0.6%   99.4%   70.0%
                    WOD_range_check     192   29.6%    0.0%  100.0%   70.4%
              loose_location_at_sea       2    0.3%    0.0%  100.0%   99.7%
               RESULT OF OR OF ALL:   11395   98.4%   88.6%   11.4%    1.6%

Taking all those that had TPR > FNR, minus CSIRO_depth for its large FPR, and in about 30 minutes:

Number of profiles tested was 12787

                       NAME OF TEST   FAILS     TPR     FPR     TNR     FNR
          CoTeDe_GTSPP_WOA_normbias    2953   85.0%   19.8%   80.2%   15.0%
                CoTeDe_WOA_normbias    1328   61.2%    7.7%   92.3%   38.8%
           CoTeDe_anomaly_detection    2180   51.3%   15.2%   84.8%   48.7%
                 CoTeDe_fuzzy_logic    1826   74.5%   11.1%   88.9%   25.5%
               RESULT OF OR OF ALL:    4363   90.5%   31.2%   68.8%    9.5%

Or instead, taking all those with FPR < 1% and in <1h:

Number of profiles tested was 12787

                       NAME OF TEST   FAILS     TPR     FPR     TNR     FNR
            Argo_global_range_check     102   16.0%    0.0%  100.0%   84.0%
                 Argo_gradient_test     186   27.4%    0.1%   99.9%   72.6%
           Argo_regional_range_test       5    0.6%    0.0%  100.0%   99.4%
                    Argo_spike_test      19    3.0%    0.0%  100.0%   97.0%
                   CSIRO_wire_break     161   25.2%    0.0%  100.0%   74.8%
          CoTeDe_GTSPP_global_range     103   16.1%    0.0%  100.0%   83.9%
              CoTeDe_GTSPP_gradient     123   17.5%    0.1%   99.9%   82.5%
           CoTeDe_GTSPP_spike_check      24    3.3%    0.0%  100.0%   96.7%
                         CoTeDe_RoC     174   24.9%    0.1%   99.9%   75.1%
             CoTeDe_digit_roll_over     294   28.6%    0.9%   99.1%   71.4%
                    CoTeDe_gradient     131   18.8%    0.1%   99.9%   81.2%
        CoTeDe_location_at_sea_test       6    0.5%    0.0%  100.0%   99.5%
             CoTeDe_profile_envelop     185   29.0%    0.0%  100.0%   71.0%
                       CoTeDe_spike      14    2.2%    0.0%  100.0%   97.8%
               CoTeDe_tukey53H_norm      68    6.6%    0.2%   99.8%   93.4%
            EN_constant_value_check       1    0.2%    0.0%  100.0%   99.8%
                     EN_range_check     102   16.0%    0.0%  100.0%   84.0%
            EN_spike_and_step_check      17    2.5%    0.0%  100.0%   97.5%
          EN_spike_and_step_suspect     210   29.1%    0.2%   99.8%   70.9%
                 EN_stability_check      28    1.3%    0.2%   99.8%   98.7%
                 WOD_gradient_check     266   30.0%    0.6%   99.4%   70.0%
                    WOD_range_check     192   29.6%    0.0%  100.0%   70.4%
              loose_location_at_sea       2    0.3%    0.0%  100.0%   99.7%
               RESULT OF OR OF ALL:     474   36.9%    2.0%   98.0%   63.1%

As I think you can tell, this is me wandering through combinations fairly ad hoc - but this is looking quite strong compared to Hamburg, thanks to the CoTeDe work. More soon.

[s-good]

This is fantastic! Thanks for doing this.

I will also have a go at running it at work - I think if I break it up into small enough parallel chunks and save the output as I go rather than at the end it may work and it saves the money, even if it is only a small amount each time. I'll let you know how I get on.

[katieannemills]

Now that we're mostly just crossing t's on #130, one thing that strikes me as pressing for May is making a final decision about what set of tests to use. @s-good, let me know if you have something in mind; otherwise, I will turn my attention to this next.

We should set some performance goals, which will likely look something like "minimum false positives while demanding < n% false negatives;" of course we want n = 0, but that's more of an asymptotic goal. The experts will need to set n for IQuOD 1.0, but if forced to do it myself, I would say something like n<5% this year, and shoot for n<2% next (if we plan on updating things in future, which is what I understood from Hamburg). But, not sure what numbers will be compelling for the community - let me know and we'll make it work.

[katieannemills]

#154 contains a quick application of a poll of a couple of machine learning strategies on a recent run over ~12000 quota profiles; current best is roughly:

• 90% of bad profiles correctly identified, at the cost of
• 1/3 of good profiles incorrectly flagged.

[katieannemills]

While hammering on that last 10% of bad profiles not getting flagged (absolutely baroque details here), I noticed that a whole lot of profiles in quota which report 0 for the depth of every level are not considered truly 'flagged' by AutoQC's referenceResults, which only considers the temperature qc t_level_qc. @s-good, is this what we want? Missing depth information seems (from my naive perspective) like a pretty serious problem that perhaps should affect how we judge whether a profile should have been flagged or not.

[s-good]

It sounds like we might need to be checking the depth QC flags as well as the temperature flags. Does it help to change line 81 in main.py from:
refAssessment = profile.t_level_qc(originator=True) >= 3
to:
refAssessment = profile.t_level_qc(originator=True) >= 3 | profile.z_level_qc(originator=True) >= 3

To answer your previous question, it would be good to try to find the best combination for a few cases e.g. lowest FPR if TPR > 90%, 95% and 98%. It might be interesting to also find highest TPR if FPR < 5% but I think that is a much lower priority.

It seems like we are pretty close to where we want to be with getting the QC checks done before running AutoQC on the different test datasets we have identified? If so I will ask about getting the test datasets extracted/converted into the WOD ASCII format.

[katieannemills]

• Sure, we can update the refAssessment definition as you describe; as long as it's understood that this definition will have a very large impact on all qc performance measures and optimization. Before I go ahead and start that, can we foresee any other updates needed to the definition of what should be considered 'bad'?
• Cases: sounds good - the way the optimization is coming together in the notebook linked above, a key parameter will be the class_weight used in the SVM learning algorithm; I am currently weighting true flags very heavily since they are our priority, but it sounds like it might be interesting to plot TPR etc as a function of this weight.
• Yes, I think we're very close to establishing a shippable product, having the test data prepped and ready to go would be good. The ideal way to share this might be in an S3 bucket, if we run on AWS.