Skip to content

Commit 10bb9d9

Browse files
SvenKlaassenSvenKlaassen
committed
Update DID multi-tuning results and metadata
1 parent e3eb809 commit 10bb9d9

File tree

6 files changed

+106
-100
lines changed

6 files changed

+106
-100
lines changed

‎doc/did/did_pa_multi.qmd‎

Lines changed: 37 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -326,6 +326,9 @@ generate_and_show_styled_table(
326326

327327
The simulations are based on the the [make_did_CS2021](https://docs.doubleml.org/stable/api/generated/doubleml.did.datasets.make_did_CS2021.html)-DGP with $2000$ observations. Due to time constraints we only consider one learner, use in-sample normalization and the following DGPs:
328328

329+
- Type 1: Linear outcome model and treatment assignment
330+
- Type 2: Nonlinear outcome model and linear treatment assignment
331+
- Type 3: Linear outcome model and nonlinear treatment assignment
329332
- Type 4: Nonlinear outcome model and treatment assignment
330333

331334
The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals). This is only an example as the untuned version just relies on the default configuration.
@@ -350,7 +353,7 @@ df = pd.read_csv("../../results/did/did_pa_multi_tune_detailed.csv", index_col=N
350353
assert df["repetition"].nunique() == 1
351354
n_rep = df["repetition"].unique()[0]
352355
353-
display_columns = ["Learner g", "Learner m", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
356+
display_columns = ["Learner g", "Learner m", "DGP", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
354357
```
355358

356359
### Observational Score
@@ -385,6 +388,9 @@ These simulations test different types of aggregation, as described in [DiD User
385388

386389
As before, we only consider one learner, use in-sample normalization and the following DGPs:
387390

391+
- Type 1: Linear outcome model and treatment assignment
392+
- Type 2: Nonlinear outcome model and linear treatment assignment
393+
- Type 3: Linear outcome model and nonlinear treatment assignment
388394
- Type 4: Nonlinear outcome model and treatment assignment
389395

390396
The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals). This is only an example as the untuned version just relies on the default configuration.
@@ -395,23 +401,23 @@ The non-uniform results (coverage, ci length and bias) refer to averaged values
395401
#| echo: false
396402
397403
# set up data
398-
df_group = pd.read_csv("../../results/did/did_pa_multi_tune_group.csv", index_col=None)
404+
df_group_tune = pd.read_csv("../../results/did/did_pa_multi_tune_group.csv", index_col=None)
399405
400-
assert df_group["repetition"].nunique() == 1
401-
n_rep_group = df_group["repetition"].unique()[0]
406+
assert df_group_tune["repetition"].nunique() == 1
407+
n_rep_group_tune = df_group_tune["repetition"].unique()[0]
402408
403-
display_columns = ["Learner g", "Learner m", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
409+
display_columns_tune = ["Learner g", "Learner m", "DGP", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
404410
```
405411

406412
#### Observational Score
407413

408414
```{python}
409415
#| echo: false
410416
generate_and_show_styled_table(
411-
main_df=df_group,
417+
main_df=df_group_tune,
412418
filters={"level": 0.95, "Score": "observational"},
413-
display_cols=display_columns,
414-
n_rep=n_rep_group,
419+
display_cols=display_columns_tune,
420+
n_rep=n_rep_group_tune,
415421
level_col="level",
416422
coverage_highlight_cols=["Coverage", "Uniform Coverage"]
417423
)
@@ -420,10 +426,10 @@ generate_and_show_styled_table(
420426
```{python}
421427
#| echo: false
422428
generate_and_show_styled_table(
423-
main_df=df_group,
429+
main_df=df_group_tune,
424430
filters={"level": 0.9, "Score": "observational"},
425-
display_cols=display_columns,
426-
n_rep=n_rep_group,
431+
display_cols=display_columns_tune,
432+
n_rep=n_rep_group_tune,
427433
level_col="level",
428434
coverage_highlight_cols=["Coverage", "Uniform Coverage"]
429435
)
@@ -436,23 +442,23 @@ generate_and_show_styled_table(
436442
#| echo: false
437443
438444
# set up data
439-
df_time = pd.read_csv("../../results/did/did_pa_multi_tune_time.csv", index_col=None)
445+
df_time_tune = pd.read_csv("../../results/did/did_pa_multi_tune_time.csv", index_col=None)
440446
441-
assert df_time["repetition"].nunique() == 1
442-
n_rep_time = df_time["repetition"].unique()[0]
447+
assert df_time_tune["repetition"].nunique() == 1
448+
n_rep_time_tune = df_time_tune["repetition"].unique()[0]
443449
444-
display_columns = ["Learner g", "Learner m", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
450+
display_columns_tune = ["Learner g", "Learner m", "DGP", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
445451
```
446452

447453
#### Observational Score
448454

449455
```{python}
450456
#| echo: false
451457
generate_and_show_styled_table(
452-
main_df=df_time,
458+
main_df=df_time_tune,
453459
filters={"level": 0.95, "Score": "observational"},
454-
display_cols=display_columns,
455-
n_rep=n_rep_time,
460+
display_cols=display_columns_tune,
461+
n_rep=n_rep_time_tune,
456462
level_col="level",
457463
coverage_highlight_cols=["Coverage", "Uniform Coverage"]
458464
)
@@ -461,10 +467,10 @@ generate_and_show_styled_table(
461467
```{python}
462468
#| echo: false
463469
generate_and_show_styled_table(
464-
main_df=df_time,
470+
main_df=df_time_tune,
465471
filters={"level": 0.9, "Score": "observational"},
466-
display_cols=display_columns,
467-
n_rep=n_rep_time,
472+
display_cols=display_columns_tune,
473+
n_rep=n_rep_time_tune,
468474
level_col="level",
469475
coverage_highlight_cols=["Coverage", "Uniform Coverage"]
470476
)
@@ -476,23 +482,23 @@ generate_and_show_styled_table(
476482
#| echo: false
477483
478484
# set up data
479-
df_es = pd.read_csv("../../results/did/did_pa_multi_tune_eventstudy.csv", index_col=None)
485+
df_es_tune = pd.read_csv("../../results/did/did_pa_multi_tune_eventstudy.csv", index_col=None)
480486
481-
assert df_es["repetition"].nunique() == 1
482-
n_rep_es = df_es["repetition"].unique()[0]
487+
assert df_es_tune["repetition"].nunique() == 1
488+
n_rep_es_tune = df_es_tune["repetition"].unique()[0]
483489
484-
display_columns = ["Learner g", "Learner m", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
490+
display_columns_tune = ["Learner g", "Learner m", "DGP", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage", "Loss g_control", "Loss g_treated", "Loss m"]
485491
```
486492

487493
#### Observational Score
488494

489495
```{python}
490496
#| echo: false
491497
generate_and_show_styled_table(
492-
main_df=df_es,
498+
main_df=df_es_tune,
493499
filters={"level": 0.95, "Score": "observational"},
494-
display_cols=display_columns,
495-
n_rep=n_rep_es,
500+
display_cols=display_columns_tune,
501+
n_rep=n_rep_es_tune,
496502
level_col="level",
497503
coverage_highlight_cols=["Coverage", "Uniform Coverage"]
498504
)
@@ -501,10 +507,10 @@ generate_and_show_styled_table(
501507
```{python}
502508
#| echo: false
503509
generate_and_show_styled_table(
504-
main_df=df_es,
510+
main_df=df_es_tune,
505511
filters={"level": 0.9, "Score": "observational"},
506-
display_cols=display_columns,
507-
n_rep=n_rep_es,
512+
display_cols=display_columns_tune,
513+
n_rep=n_rep_es_tune,
508514
level_col="level",
509515
coverage_highlight_cols=["Coverage", "Uniform Coverage"]
510516
)

‎results/did/did_pa_multi_tune_detailed.csv‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
1-
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Tuned,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,repetition
2-
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,False,0.9041666666666667,1.1400890857194432,0.28740542972213634,0.92,1.7825586546503152,100
3-
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,True,0.8966666666666667,0.6472337011214075,0.15944046942151935,0.89,1.0113932407840978,100
4-
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,False,0.9608333333333333,1.3584999361424834,0.28740542972213634,0.98,1.9530265671281697,100
5-
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,True,0.9516666666666667,0.7712265231342353,0.15944046942151935,0.94,1.108993599863019,100
6-
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,False,0.9091666666666667,1.228317976026286,0.29714569696579457,0.87,1.9154225471875832,100
7-
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,True,0.9075,0.6999396898634899,0.16414917750405422,0.89,1.0923185146373655,100
8-
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,False,0.9625,1.4636311432990996,0.29714569696579457,0.96,2.1000879769164107,100
9-
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,True,0.955,0.8340295823313698,0.16414917750405422,0.92,1.1991852306848672,100
10-
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,False,0.9116666666666667,1.3107305270692071,0.31985432471813957,0.93,2.043307297150825,100
11-
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,True,0.8908333333333333,0.7112998258902837,0.17517537461692242,0.84,1.1042718075016527,100
12-
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,False,0.9616666666666667,1.5618317547526317,0.31985432471813957,0.96,2.2409152941225887,100
13-
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,True,0.9391666666666667,0.8475660193171083,0.17517537461692242,0.9,1.2154039059944974,100
14-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9166666666666667,1.4374818316154423,0.33276716041579246,0.94,2.2272959804052928,100
15-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.7691666666666667,0.7404367767653766,0.254447961458634,0.69,1.1489525496687707,100
16-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.9675,1.712865249668844,0.33276716041579246,0.96,2.449349686911489,100
17-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.8541666666666667,0.8822848376963073,0.254447961458634,0.79,1.2629079128831524,100
1+
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Tuned,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,Loss g_control,Loss g_treated,Loss m,repetition
2+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,False,0.8991666666666667,1.1367782872857282,0.2762362295842985,0.93,1.7739576901791148,3.661560613833878,2.9515587152603526,0.8541707478068831,100
3+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,True,0.8775,0.6396360988823492,0.1679386739873294,0.83,1.0011531338160695,3.360158161171733,2.7665447739373614,0.6798065307413697,100
4+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,False,0.9625,1.354554876482567,0.2762362295842985,0.97,1.9460309141153076,3.661560613833878,2.9515587152603526,0.8541707478068831,100
5+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,True,0.9233333333333333,0.7621734216828837,0.1679386739873294,0.89,1.0961658049458993,3.360158161171733,2.7665447739373614,0.6798065307413697,100
6+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,False,0.8983333333333333,1.2330356378884049,0.30663270016192556,0.92,1.9173958386413434,3.386477693258062,3.152253900679997,0.8480386768604923,100
7+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,True,0.8916666666666667,0.6719441811647116,0.16476409918696738,0.86,1.0513367602608807,3.1088298411208632,2.92912769411877,0.6763092050163523,100
8+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,False,0.9541666666666667,1.4692525841309676,0.30663270016192556,0.96,2.1053464077111452,3.386477693258062,3.152253900679997,0.8480386768604923,100
9+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,True,0.9425,0.8006708761952025,0.16476409918696738,0.93,1.1526329365873031,3.1088298411208632,2.92912769411877,0.6763092050163523,100
10+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,False,0.905,1.31358053465385,0.32661174597213943,0.9,2.0440203617896806,2.999802652037201,2.982841611043071,0.8507272462648376,100
11+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,True,0.865,0.7289212442976212,0.1945603364540163,0.86,1.1371526826681673,3.032548727546919,3.0240798464809977,0.6800265869273724,100
12+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,False,0.9566666666666667,1.5652277482501924,0.32661174597213943,0.96,2.242653923089107,2.999802652037201,2.982841611043071,0.8507272462648376,100
13+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,True,0.9308333333333333,0.8685632344303481,0.1945603364540163,0.88,1.2462114970320124,3.032548727546919,3.0240798464809977,0.6800265869273724,100
14+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9258333333333333,1.4012822649889498,0.33623310300936715,0.91,2.1760953165140573,2.9827080885681765,2.9977847813459078,0.8486888442599515,100
15+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.7533333333333333,0.775057031885869,0.27368317437679723,0.6,1.2076437086430896,2.996299502921045,3.0408206631075583,0.6757679707228056,100
16+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.965,1.6697308055570117,0.33623310300936715,0.94,2.3932554225940295,2.9827080885681765,2.9977847813459078,0.8486888442599515,100
17+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.8358333333333333,0.9235374160777122,0.27368317437679723,0.71,1.325967981957511,2.996299502921045,3.0408206631075583,0.6757679707228056,100

‎results/did/did_pa_multi_tune_eventstudy.csv‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
1-
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Tuned,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,repetition
2-
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,False,0.9283333333333332,1.073068168296971,0.2597502402941499,0.93,1.4984680319815546,100
3-
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,True,0.905,0.5982631043246142,0.14657170491101748,0.88,0.8348004786911729,100
4-
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,False,0.9733333333333333,1.2786395873512446,0.2597502402941499,0.98,1.677549043525822,100
5-
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,True,0.9516666666666667,0.7128744579714308,0.14657170491101748,0.97,0.9341199957423918,100
6-
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,False,0.9183333333333333,1.184163621831536,0.28342610855647676,0.94,1.6513528430347295,100
7-
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,True,0.895,0.6576942435782439,0.15332741525347127,0.9,0.9110830194425428,100
8-
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,False,0.9616666666666667,1.4110179851649454,0.28342610855647676,0.99,1.8462857744217072,100
9-
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,True,0.95,0.7836910282660079,0.15332741525347127,0.95,1.0193279652523601,100
10-
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,False,0.9183333333333333,1.2647923551028497,0.2906734347752842,0.95,1.7525858381045432,100
11-
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,True,0.885,0.6743634497620673,0.16589326251977687,0.88,0.9307320111152868,100
12-
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,False,0.9683333333333333,1.5070930466424515,0.2906734347752842,0.97,1.9677774556587133,100
13-
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,True,0.9383333333333332,0.8035536125323768,0.16589326251977687,0.93,1.0446188993963292,100
14-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9283333333333332,1.4064992969714387,0.314613746630164,0.91,1.9372254520475036,100
15-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.7133333333333333,0.7088318207664741,0.26929824646603895,0.66,0.975482632053878,100
16-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.9683333333333333,1.6759472825883586,0.314613746630164,0.98,2.1745712334757688,100
17-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.8033333333333332,0.8446252098267877,0.26929824646603895,0.76,1.0954531498790554,100
1+
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Tuned,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,Loss g_control,Loss g_treated,Loss m,repetition
2+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,False,0.9083333333333333,1.0742547796394701,0.2618988136569734,0.94,1.5022639132232276,3.661560613833878,2.9515587152603526,0.8541707478068831,100
3+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,True,0.8633333333333333,0.5878107921893911,0.15731265501369246,0.88,0.8222540013002478,3.360158161171733,2.7665447739373614,0.6798065307413697,100
4+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,False,0.9566666666666667,1.2800535219754794,0.2618988136569734,0.98,1.6776898490222303,3.661560613833878,2.9515587152603526,0.8541707478068831,100
5+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,True,0.9233333333333333,0.7004197598727457,0.15731265501369246,0.91,0.915822574094016,3.360158161171733,2.7665447739373614,0.6798065307413697,100
6+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,False,0.9016666666666667,1.1885817333189461,0.28095848380634825,0.9,1.6488463476730966,3.386477693258062,3.152253900679997,0.8480386768604923,100
7+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,True,0.91,0.6233279876475303,0.1499301158544056,0.91,0.8702171833507445,3.1088298411208632,2.92912769411877,0.6763092050163523,100
8+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,False,0.9533333333333333,1.4162824897099824,0.28095848380634825,0.95,1.844474784460219,3.386477693258062,3.152253900679997,0.8480386768604923,100
9+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,True,0.9533333333333333,0.7427411085868194,0.1499301158544056,0.94,0.9737952014197065,3.1088298411208632,2.92912769411877,0.6763092050163523,100
10+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,False,0.91,1.265646440169177,0.3020810902480612,0.96,1.7582545006922015,2.999802652037201,2.982841611043071,0.8507272462648376,100
11+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,True,0.8516666666666667,0.6864549574097271,0.18550205150287405,0.82,0.9531749334821116,3.032548727546919,3.0240798464809977,0.6800265869273724,100
12+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,False,0.9566666666666667,1.5081107517697077,0.3020810902480612,0.99,1.9733460594562338,2.999802652037201,2.982841611043071,0.8507272462648376,100
13+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,True,0.91,0.8179615325563163,0.18550205150287405,0.91,1.0675220384475739,3.032548727546919,3.0240798464809977,0.6800265869273724,100
14+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9233333333333333,1.3639352077948335,0.31325283477445254,0.92,1.8864279971533038,2.9827080885681765,2.9977847813459078,0.8486888442599515,100
15+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.675,0.7467959521377197,0.29502185426592087,0.57,1.0327936903874404,2.996299502921045,3.0408206631075583,0.6757679707228056,100
16+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.955,1.6252290420993776,0.31325283477445254,0.94,2.1144079580645485,2.9827080885681765,2.9977847813459078,0.8486888442599515,100
17+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.76,0.8898622625181539,0.29502185426592087,0.71,1.157187445369494,2.996299502921045,3.0408206631075583,0.6757679707228056,100

0 commit comments

Comments
 (0)