Running TOGA with an assertless dataset

[bentodaniel]

When running TOGA with the provided datasets, for example the evosuite_reaching_tests, TOGA generates two pairs of files:

• except_model_inputs.csv and exception_preds.csv;
• assert_model_inputs.csv and assertion_preds.csv;

For my project, I am trying to use TOGA on a dataset similar to ATLAS' dataset, where all of the assertions have been replaced by a placeholder, for example, something like this:

public void testIssue705() throws Exception {
    Issue705Bean input = new Issue705Bean(""key"", ""value"");
    String json = MAPPER.writeValueAsString(input);
    // TEST ORACLE
}

where // TEST ORACLE would be our placeholder (this can be replaced by something else too).

 

Problem is, when i run TOGA with a dataset like this, it fails to write something in the assert_model_inputs.csv (this file will only contain the header), which leads to not being able to generate the assertion_preds.csv and, consequently, fail because it can not find this same file.
This leads to the following stacktrace:

preparing assertion model inputs

// Defects4J: flaky method
// public void testBug2849731() {
// TestIntervalCategoryDataset d = new TestIntervalCategoryDataset();
// d.addItem(2.5, 2.0, 3.0, "R1", "C1");
// d.addItem(4.0, 0.0, 0.0, "R2", "C1");
// ,
// DatasetUtilities.iterateRangeBounds(d));
// }
/**
 * Another test for bug 2849731.
 */
public void testBug2849731_2() {
    XYIntervalSeriesCollection d = new XYIntervalSeriesCollection();
    XYIntervalSeries s = new XYIntervalSeries("S1");
    s.add(1.0, Double.NaN, Double.NaN, Double.NaN, 1.5, Double.NaN);
    d.addSeries(s);
    Range r = DatasetUtilities.iterateDomainBounds(d);
    // TEST ORACLE
}

[]
05/20/2022 21:39:40 - WARNING - __main__ -   Process rank: -1, device: cpu, n_gpu: 0, distributed training: False, 16-bits training: False
Some weights of the model checkpoint at microsoft/codebert-base were not used when initializing RobertaForSequenceClassification: ['pooler.dense.weight', 'pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at microsoft/codebert-base and are newly initialized: ['classifier.out_proj.bias', 'classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
05/20/2022 21:39:48 - INFO - __main__ -   Training/evaluation parameters Namespace(data_dir='./', model_type='roberta', model_name_or_path='microsoft/codebert-base', task_name='assertion_classifier', output_dir='./models/assertion_classifier', config_name='', tokenizer_name='', cache_dir='', max_seq_length=200, do_train=False, do_eval=False, do_predict=True, evaluate_during_training=False, do_lower_case=False, per_gpu_train_batch_size=32, per_gpu_eval_batch_size=256, gradient_accumulation_steps=1, learning_rate=1e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=8.0, max_steps=-1, warmup_steps=0, logging_steps=50, save_steps=50, eval_all_checkpoints=True, no_cuda=False, overwrite_output_dir=True, overwrite_cache=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, server_ip='', server_port='', train_file='train.txt', dev_file='valid.txt', test_file='assert_model_inputs.csv', pred_model_dir='model/assertions/pretrained/', test_result_dir='test_results.tsv', n_gpu=0, device=device(type='cpu'), output_mode='classification', start_epoch=0, start_step=0)
testing

05/20/2022 21:39:49 - INFO - __main__ -   Creating features from dataset file at ./

05/20/2022 21:39:49 - INFO - utils -   LOOKING AT ./assert_model_inputs.csv

/home/daniel/work/oracles/toga/model/assertions/utils.py:364: RuntimeWarning: Mean of empty slice.
  print(f'total_samples {len(trunc_b)} total truncations {(trunc_both > 0).sum()}, mean_trunc {trunc_both.mean()}, median {np.median(trunc_both)}, max = {trunc_both.max()}')
/home/daniel/.local/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/home/daniel/.local/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3440: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
Traceback (most recent call last):
  File "/home/daniel/work/oracles/toga/model/assertions/run_classifier.py", line 720, in <module>
    main()
  File "/home/daniel/work/oracles/toga/model/assertions/run_classifier.py", line 715, in main
    evaluate(args, model, tokenizer, checkpoint=None, prefix='', mode='test')
  File "/home/daniel/work/oracles/toga/model/assertions/run_classifier.py", line 236, in evaluate
    eval_dataset, instances = load_and_cache_examples(args, eval_task, tokenizer, ttype='test')
  File "/home/daniel/work/oracles/toga/model/assertions/run_classifier.py", line 421, in load_and_cache_examples
    features = convert_examples_to_features(examples, label_list, args.max_seq_length, tokenizer, output_mode,
  File "/home/daniel/work/oracles/toga/model/assertions/utils.py", line 364, in convert_examples_to_features
    print(f'total_samples {len(trunc_b)} total truncations {(trunc_both > 0).sum()}, mean_trunc {trunc_both.mean()}, median {np.median(trunc_both)}, max = {trunc_both.max()}')
  File "/home/daniel/.local/lib/python3.9/site-packages/numpy/core/_methods.py", line 40, in _amax
    return umr_maximum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation maximum which has no identity



Result of running bash ./model/assertions/run_eval.sh assert_model_inputs.csv
CompletedProcess(args=['bash', './model/assertions/run_eval.sh', 'assert_model_inputs.csv'], returncode=1)



Traceback (most recent call last):
  File "/home/daniel/work/oracles/toga/toga.py", line 218, in <module>
    main()
  File "/home/daniel/work/oracles/toga/toga.py", line 83, in main
    with open("assertion_preds.csv") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'assertion_preds.csv'

I have also tried to first run one of the provided datasets (evosuite_reaching_tests) to have it generate this missing file.
With this, i can run TOGA, however, as expected, the assert_model_inputs.csv is overwritten and left empty.
Also, as may be expected, as the assertion_preds.csv is not generated for the dataset being used, TOGA, despite running, can not infer any assert.

In summary:
I would like to use TOGA on a project that has absolutely no assert statements and have it generate all of the needed asserts, is this in any way possible?

Thank you,
Daniel Bento

[gryan11]

Hi Daniel,

If you have a dataset similar to ATLAS with assertion placeholders, we recommend using the TOGA implementation under eval/rq2/assertion_inference/ and usingdata/atlas_star_datagen.py to generate model inputs from your tests and eval/rq2/assertion_inference/rq2.sh to run on your inputs. You can use the same placeholder strings used by ATLAS in your inputs. Note that your test inputs will need to be in the same format as ATLAS inputs for the script to work.

The interface for TOGA implemented in toga.py is designed specifically to work with Evosuite generated tests, and replaces the Evosuite-generated assertions it sees in those tests with its own assertions. So to use it you would need to add 'dummy' assertions to your tests on variables you to wish to test so TOGA knows which variables to generate assertions on.

Best,
Gabe

[bentodaniel]

Hello Gabriel,

Thank your for your response,

About using the same format as ATLAS' inputs, that means I would have to have two files, one with the test method + focal method and another with the assertions, and both files would have to be tokenized, right? As i have no assertions for my tests, this raises two questions:

1. What if I do not have any assertions for my tests? I imagine this would not work and would actually crash the data/atlas_star_datagen.py script;
2. Could I use dummy assertions like you mention for EvoSuite?
   2.1. Also, when using dummy assertions, as you describe it, they must use the variables I want to test. This means I could not use assertions as simple as assertTrue(true), correct?

Thank you,
Daniel Bento

[gryan11]

Hi Daniel, For the ATLAS dataset an assert placeholder string is included with the tests that we replace with an assertion. So you can use the same assert placeholder in your tests and generate assertions without having to include an assertion of your own. If you wish to use our Evosuite oracle pipeline for assertion generation (which would also exception oracles), you would need to include assertions on specific variables as you describe. assertTrue(true) would not work. Best, Gabe

…

On Sun, May 22, 2022 at 6:11 AM Daniel Bento ***@***.***> wrote: Hello Gabriel, Thank your for your response, About using the same format as ATLAS' inputs, that means I would have to have two files, one with the test method + focal method and another with the assertions, and both files would have to be tokenized, right? As i have no assertions for my tests, this raises two questions: 1. What if I do not have any assertions for my tests? I imagine this would not work and would actually crash the data/atlas_star_datagen.py script; 2. Could I use dummy assertions like you mention for EvoSuite? 2.1. Also, when using dummy assertions, as you describe it, they must use the variables I want to test. This means I could not use assertions as simple as assertTrue(true), correct? Thank you, Daniel Bento — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADFQKD57EJSAGD7EDDZBDDVLIB35ANCNFSM5WRVLZAQ> . You are receiving this because you commented.Message ID: ***@***.***>

[BlackStar7713]

Hello Gabriel,

I'm currently studying computer science at the chair of professor Gordon Fraser and am trying to execute toga as part of my master thesis.
I'm trying to have it generate an assertion for a test without an assertion. Following what you said here, I formatted the test case as in the atlas input files, and used a dummy value in assertLines.txt
My input is as follows:
testMethods.txt:
getUsersWaitingNotificationNoWatchExpectEmptyList ( ) { net . jforum . repository . TopicWatchRepository dao = this . newDao ( ) ; net . jforum . entities . Topic topic = new net . jforum . entities . Topic ( ) ; topic . setId ( 13 ) ; java . util . List < net . jforum . entities . User > users = dao . getUsersWaitingNotification ( topic ) ; "<AssertPlaceHolder>" ; } getUsersWaitingNotification ( net . jforum . entities . Topic ) { java . util . List < net . jforum . entities . User > users = session . createQuery ( ( "select<sp>u<sp>from<sp>TopicWatch<sp>tw<sp>" + ( "<sp>inner<sp>join<sp>tw.user<sp>u<sp>where<sp>tw.topic<sp>=<sp>:topic<sp>" + "<sp>and<sp>(tw.read<sp>=<sp>true<sp>or<sp>u.notifyAlways<sp>=<sp>true)" ) ) ) . setEntity ( "topic" , topic ) . setComment ( "topicWatchDAO.getUsersWaitingNotification" ) . list ( ) ; if ( ( users . size ( ) ) > 0 ) { this . markAllAsUnread ( topic ) ; } return users ; }"
assertLines.txt:
org . junit . Assert . assertTrue ( true )

After then running atlas_star_datagen, I'm left with the following testcsv:
idx,label,fm,test,assertion 0,1,"getUsersWaitingNotification ( net . jforum . entities . Topic ) { java . util . List < net . jforum . entities . User > users = session . createQuery ( ( ""select<sp>u<sp>from<sp>TopicWatch<sp>tw<sp>"" + ( ""<sp>inner<sp>join<sp>tw.user<sp>u<sp>where<sp>tw.topic<sp>=<sp>:topic<sp>"" + ""<sp>and<sp>(tw.read<sp>=<sp>true<sp>or<sp>u.notifyAlways<sp>=<sp>true)"" ) ) ) . setEntity ( ""topic"" , topic ) . setComment ( ""topicWatchDAO.getUsersWaitingNotification"" ) . list ( ) ; if ( ( users . size ( ) ) > 0 ) { this . markAllAsUnread ( topic ) ; } return users ; }""","getUsersWaitingNotificationNoWatchExpectEmptyList ( ) { net . jforum . repository . TopicWatchRepository dao = this . newDao ( ) ; net . jforum . entities . Topic topic = new net . jforum . entities . Topic ( ) ; topic . setId ( 13 ) ; java . util . List < net . jforum . entities . User > users = dao . getUsersWaitingNotification ( topic ) ; ""<AssertPlaceHolder>"" ; }",assertTrue ( true ) 1,0,"getUsersWaitingNotification ( net . jforum . entities . Topic ) { java . util . List < net . jforum . entities . User > users = session . createQuery ( ( ""select<sp>u<sp>from<sp>TopicWatch<sp>tw<sp>"" + ( ""<sp>inner<sp>join<sp>tw.user<sp>u<sp>where<sp>tw.topic<sp>=<sp>:topic<sp>"" + ""<sp>and<sp>(tw.read<sp>=<sp>true<sp>or<sp>u.notifyAlways<sp>=<sp>true)"" ) ) ) . setEntity ( ""topic"" , topic ) . setComment ( ""topicWatchDAO.getUsersWaitingNotification"" ) . list ( ) ; if ( ( users . size ( ) ) > 0 ) { this . markAllAsUnread ( topic ) ; } return users ; }""","getUsersWaitingNotificationNoWatchExpectEmptyList ( ) { net . jforum . repository . TopicWatchRepository dao = this . newDao ( ) ; net . jforum . entities . Topic topic = new net . jforum . entities . Topic ( ) ; topic . setId ( 13 ) ; java . util . List < net . jforum . entities . User > users = dao . getUsersWaitingNotification ( topic ) ; ""<AssertPlaceHolder>"" ; }",assertFalse ( true )

Here we get two data points, but it only uses the assertTrue I used as a dummy to generate them. How can I execute this step without limiting myself by what I use as a dummy value? I would expect an output to somehow check the size of the returned users list but so far it would only do this if I already give it an input assertion using users.size()

I continued from here by executing rq2.sh and ended up with this test_results.csv
test_num,true_lbl,pred_lbl,logit_0,logit_1,test_name,true_assertion,test,fms 1,1,1,-4.878190517425537,4.990329265594482,test-name-placeholder,assertTrue ( true ),"getUsersWaitingNotificationNoWatchExpectEmptyList ( ) { net . jforum . repository . TopicWatchRepository dao = this . newDao ( ) ; net . jforum . entities . Topic topic = new net . jforum . entities . Topic ( ) ; topic . setId ( 13 ) ; java . util . List < net . jforum . entities . User > users = dao . getUsersWaitingNotification ( topic ) ; ""<AssertPlaceHolder>"" ; }","getUsersWaitingNotification ( net . jforum . entities . Topic ) { java . util . List < net . jforum . entities . User > users = session . createQuery ( ( ""select<sp>u<sp>from<sp>TopicWatch<sp>tw<sp>"" + ( ""<sp>inner<sp>join<sp>tw.user<sp>u<sp>where<sp>tw.topic<sp>=<sp>:topic<sp>"" + ""<sp>and<sp>(tw.read<sp>=<sp>true<sp>or<sp>u.notifyAlways<sp>=<sp>true)"" ) ) ) . setEntity ( ""topic"" , topic ) . setComment ( ""topicWatchDAO.getUsersWaitingNotification"" ) . list ( ) ; if ( ( users . size ( ) ) > 0 ) { this . markAllAsUnread ( topic ) ; } return users ; }""" 1,0,0,4.397204875946045,-4.564648151397705,test-name-placeholder,assertFalse ( true ),"getUsersWaitingNotificationNoWatchExpectEmptyList ( ) { net . jforum . repository . TopicWatchRepository dao = this . newDao ( ) ; net . jforum . entities . Topic topic = new net . jforum . entities . Topic ( ) ; topic . setId ( 13 ) ; java . util . List < net . jforum . entities . User > users = dao . getUsersWaitingNotification ( topic ) ; ""<AssertPlaceHolder>"" ; }","getUsersWaitingNotification ( net . jforum . entities . Topic ) { java . util . List < net . jforum . entities . User > users = session . createQuery ( ( ""select<sp>u<sp>from<sp>TopicWatch<sp>tw<sp>"" + ( ""<sp>inner<sp>join<sp>tw.user<sp>u<sp>where<sp>tw.topic<sp>=<sp>:topic<sp>"" + ""<sp>and<sp>(tw.read<sp>=<sp>true<sp>or<sp>u.notifyAlways<sp>=<sp>true)"" ) ) ) . setEntity ( ""topic"" , topic ) . setComment ( ""topicWatchDAO.getUsersWaitingNotification"" ) . list ( ) ; if ( ( users . size ( ) ) > 0 ) { this . markAllAsUnread ( topic ) ; } return users ; }"""

Basically the only data I get from this is the columns "true_lbl", "pred_lbl", "logit_0", and "logit_1", all the other columns are copied straight from the input. How would I go from this output to a generated assertion I can insert into my testcase?

[gryan11]

Hi Daniel,

To answer each of your questions:

How can I execute this step without limiting myself by what I use as a dummy value? I would expect an output to somehow check the size of the returned users list but so far it would only do this if I already give it an input assertion using users.size()

You'll need to add an assert for the specific method you want to assert on, like assertEquals ( 999, users.size() ) to the assertLines.txt. Toga will then replace 999 with it's own generated value.

Basically the only data I get from this is the columns "true_lbl", "pred_lbl", "logit_0", and "logit_1", all the other columns are copied straight from the input. How would I go from this output to a generated assertion I can insert into my testcase?

For a given test, you can pick the row with the highest logit_1 value where pred_lbl is 1 (or lowest logit_0 value, in the event all pred_lbls are 0), and then use the assertion from that row.

You can copy the assert into ""<AssertPlaceHolder>"" to get a complete generated unit test.

[BlackStar7713]

Hi Gabriel,
Thank you for the answers.
Just to clarify that I understood correctly:
There is no way to run TOGA on a testsuite that does not contain existing assertions? To use this tool I need to go through every test individually and manually create an (approximate) assertion, which can then be optimized by TOGA?

[gryan11]

Hi Daniel, That's correct. Typically the problem of selecting which variables on which to generate assertions is handled by mutation testing. There are many mature tools and methods for that such as PIT, and evosuite uses mutation testing as part of its test generation. TOGA instead focuses specifically on determining what those assertions should be. For tests without assertions you may be able to generate input assertions based on method return types. Best, Gabe

…

On Tue, Aug 30, 2022 at 4:49 PM Daniel Berreiter ***@***.***> wrote: Hi Gabriel, Thank you for the answers. Just to clarify that I understood correctly: There is no way to run TOGA on a testsuite that does not contain existing assertions? To use this tool I need to go through every test individually and manually create an (approximate) assertion, which can then be optimized by TOGA? — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADFQKF2G4GUFMAIHULJQEDV3ZXWHANCNFSM5WRVLZAQ> . You are receiving this because you commented.Message ID: ***@***.***>