Skip to content

Commit 347e33d

Browse files
SvenKlaassenSvenKlaassen
committed
rerun did sim
1 parent 7393505 commit 347e33d

File tree

7 files changed

+71
-20
lines changed

7 files changed

+71
-20
lines changed

‎results/did/did_pa_multi_tune_config.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
simulation_parameters:
2-
repetitions: 50
2+
repetitions: 100
33
max_runtime: 19800
44
random_seed: 42
55
n_jobs: -2
66
dgp_parameters:
77
DGP:
8+
- 1
9+
- 2
10+
- 3
811
- 4
912
n_obs:
1013
- 2000

‎results/did/did_pa_multi_tune_detailed.csv‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Tuned,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,repetition
2-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.92,1.348642264453785,0.3052825096979891,0.92,2.098844496580318,50
3-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.7533333333333333,0.7307170319438454,0.25209824311524026,0.58,1.1328574508344817,50
4-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.965,1.6070063761582027,0.3052825096979891,0.94,2.299873978116884,50
5-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.8483333333333333,0.8707030473917022,0.25209824311524026,0.72,1.2446051133150078,50
2+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,False,0.9041666666666667,1.1400890857194432,0.28740542972213634,0.92,1.7825586546503152,100
3+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,True,0.8966666666666667,0.6472337011214075,0.15944046942151935,0.89,1.0113932407840978,100
4+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,False,0.9608333333333333,1.3584999361424834,0.28740542972213634,0.98,1.9530265671281697,100
5+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,True,0.9516666666666667,0.7712265231342353,0.15944046942151935,0.94,1.108993599863019,100
6+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,False,0.9091666666666667,1.228317976026286,0.29714569696579457,0.87,1.9154225471875832,100
7+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,True,0.9075,0.6999396898634899,0.16414917750405422,0.89,1.0923185146373655,100
8+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,False,0.9625,1.4636311432990996,0.29714569696579457,0.96,2.1000879769164107,100
9+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,True,0.955,0.8340295823313698,0.16414917750405422,0.92,1.1991852306848672,100
10+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,False,0.9116666666666667,1.3107305270692071,0.31985432471813957,0.93,2.043307297150825,100
11+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,True,0.8908333333333333,0.7112998258902837,0.17517537461692242,0.84,1.1042718075016527,100
12+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,False,0.9616666666666667,1.5618317547526317,0.31985432471813957,0.96,2.2409152941225887,100
13+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,True,0.9391666666666667,0.8475660193171083,0.17517537461692242,0.9,1.2154039059944974,100
14+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9166666666666667,1.4374818316154423,0.33276716041579246,0.94,2.2272959804052928,100
15+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.7691666666666667,0.7404367767653766,0.254447961458634,0.69,1.1489525496687707,100
16+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.9675,1.712865249668844,0.33276716041579246,0.96,2.449349686911489,100
17+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.8541666666666667,0.8822848376963073,0.254447961458634,0.79,1.2629079128831524,100

‎results/did/did_pa_multi_tune_eventstudy.csv‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Tuned,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,repetition
2-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9133333333333333,1.3059006382995975,0.28281151638077767,0.92,1.8076721639449533,50
3-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.6933333333333332,0.6935329291378785,0.27189850445260494,0.6,0.9561393211979365,50
4-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.9633333333333333,1.5560765873124054,0.28281151638077767,0.96,2.0177850919183284,50
5-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.7766666666666667,0.8263954560638327,0.27189850445260494,0.7,1.0722540700492953,50
2+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,False,0.9283333333333332,1.073068168296971,0.2597502402941499,0.93,1.4984680319815546,100
3+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,True,0.905,0.5982631043246142,0.14657170491101748,0.88,0.8348004786911729,100
4+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,False,0.9733333333333333,1.2786395873512446,0.2597502402941499,0.98,1.677549043525822,100
5+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,True,0.9516666666666667,0.7128744579714308,0.14657170491101748,0.97,0.9341199957423918,100
6+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,False,0.9183333333333333,1.184163621831536,0.28342610855647676,0.94,1.6513528430347295,100
7+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,True,0.895,0.6576942435782439,0.15332741525347127,0.9,0.9110830194425428,100
8+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,False,0.9616666666666667,1.4110179851649454,0.28342610855647676,0.99,1.8462857744217072,100
9+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,True,0.95,0.7836910282660079,0.15332741525347127,0.95,1.0193279652523601,100
10+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,False,0.9183333333333333,1.2647923551028497,0.2906734347752842,0.95,1.7525858381045432,100
11+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,True,0.885,0.6743634497620673,0.16589326251977687,0.88,0.9307320111152868,100
12+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,False,0.9683333333333333,1.5070930466424515,0.2906734347752842,0.97,1.9677774556587133,100
13+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,True,0.9383333333333332,0.8035536125323768,0.16589326251977687,0.93,1.0446188993963292,100
14+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9283333333333332,1.4064992969714387,0.314613746630164,0.91,1.9372254520475036,100
15+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.7133333333333333,0.7088318207664741,0.26929824646603895,0.66,0.975482632053878,100
16+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.9683333333333333,1.6759472825883586,0.314613746630164,0.98,2.1745712334757688,100
17+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.8033333333333332,0.8446252098267877,0.26929824646603895,0.76,1.0954531498790554,100

‎results/did/did_pa_multi_tune_group.csv‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Tuned,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,repetition
2-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.94,1.335317766310145,0.2896013491766717,0.96,1.698765271200942,50
3-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.6666666666666667,0.7417801146292107,0.28018974004255187,0.64,0.9458331628750551,50
4-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.98,1.5911292573400337,0.2896013491766717,0.96,1.9275714352433053,50
5-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.8066666666666668,0.8838855234892817,0.28018974004255187,0.74,1.0623714755405609,50
2+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,False,0.8933333333333333,1.104533138085938,0.2786392828405217,0.93,1.403347738672554,100
3+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,True,0.9166666666666667,0.6434665238707676,0.15552856896641956,0.91,0.8209959260997447,100
4+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,False,0.9666666666666667,1.3161324113633812,0.2786392828405217,0.97,1.5959470400469875,100
5+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,True,0.9533333333333333,0.7667376545725277,0.15552856896641956,0.96,0.9301867858119841,100
6+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,False,0.92,1.2001857652221994,0.2761379784298203,0.94,1.5248040252023016,100
7+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,True,0.91,0.705320543228418,0.15481408501446786,0.88,0.8960249286786839,100
8+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,False,0.96,1.430109546557577,0.2761379784298203,0.97,1.7238479263548212,100
9+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,True,0.95,0.8404412645798971,0.15481408501446786,0.93,1.0186697969238927,100
10+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,False,0.8933333333333333,1.3085128545125806,0.30695508841663705,0.92,1.6601995824415863,100
11+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,True,0.89,0.7237290131511886,0.16938601624368954,0.91,0.9191510186073212,100
12+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,False,0.9533333333333333,1.5591892349142273,0.30695508841663705,0.96,1.8814562718978545,100
13+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,True,0.9533333333333333,0.8623763094178922,0.16938601624368954,0.92,1.042629687574485,100
14+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9233333333333333,1.4575276036298743,0.3136429990461747,0.93,1.841601446334188,100
15+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.7533333333333333,0.7563191729586094,0.28371438189486503,0.66,0.9629096996466265,100
16+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.9733333333333333,1.7367512602820832,0.3136429990461747,0.97,2.0888217668673956,100
17+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.8233333333333333,0.9012098800325217,0.28371438189486503,0.81,1.087106451467499,100

‎results/did/did_pa_multi_tune_metadata.csv‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
DoubleML Version,Script,Date,Total Runtime (minutes),Python Version,Config File
2-
0.12.dev0,DIDMultiTuningCoverageSimulation,2025-11-28 10:36,6.335572187105814,3.12.9,scripts/did/did_pa_multi_tune_config.yml
2+
0.12.dev0,DIDMultiTuningCoverageSimulation,2025-12-01 10:11,45.433327130476634,3.12.9,scripts/did/did_pa_multi_tune_config.yml

‎results/did/did_pa_multi_tune_time.csv‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
Learner g,Learner m,Score,In-sample-norm.,DGP,level,Tuned,Coverage,CI Length,Bias,Uniform Coverage,Uniform CI Length,repetition
2-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9266666666666667,1.3680691335956139,0.2862973732054817,0.9,1.6857878745010273,50
3-
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.6533333333333333,0.7330060851496288,0.29002355600549995,0.54,0.9000560955775124,50
4-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.96,1.630154918512653,0.2862973732054817,0.94,1.9313064378387346,50
5-
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.74,0.8734306225196768,0.29002355600549995,0.62,1.0346551755913964,50
2+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,False,0.9,1.1113763663388696,0.2783388170117924,0.92,1.3788558172615877,100
3+
LGBM Regr.,LGBM Clas.,observational,True,1,0.9,True,0.8833333333333333,0.6148427992241794,0.15024436835827917,0.91,0.7634587423773123,100
4+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,False,0.96,1.3242866207677713,0.2783388170117924,0.97,1.5683457246716077,100
5+
LGBM Regr.,LGBM Clas.,observational,True,1,0.95,True,0.95,0.732630382964001,0.15024436835827917,0.99,0.8733603074599771,100
6+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,False,0.93,1.2340740537756638,0.2827466455736652,0.94,1.5228025759168033,100
7+
LGBM Regr.,LGBM Clas.,observational,True,2,0.9,True,0.93,0.6909121668851053,0.14601582394245094,0.91,0.8519331823086561,100
8+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,False,0.97,1.470489932978703,0.2827466455736652,0.97,1.7440031028399017,100
9+
LGBM Regr.,LGBM Clas.,observational,True,2,0.95,True,0.9533333333333333,0.8232726252275122,0.14601582394245094,0.97,0.9758565772068002,100
10+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,False,0.9433333333333332,1.2959603630389032,0.2926620156379758,0.96,1.6022423636178649,100
11+
LGBM Regr.,LGBM Clas.,observational,True,3,0.9,True,0.8933333333333333,0.7034275560361475,0.16697163725248976,0.88,0.8670910640659377,100
12+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,False,0.98,1.5442320187817185,0.2926620156379758,0.98,1.834795211566282,100
13+
LGBM Regr.,LGBM Clas.,observational,True,3,0.95,True,0.9533333333333333,0.8381856312157769,0.16697163725248976,0.96,0.9914556000230386,100
14+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,False,0.9366666666666668,1.4986674065988734,0.3275028156163638,0.95,1.8415020458286135,100
15+
LGBM Regr.,LGBM Clas.,observational,True,4,0.9,True,0.67,0.7464049203833408,0.300651425058323,0.58,0.9150101228510614,100
16+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,False,0.9833333333333333,1.785772359077211,0.3275028156163638,0.96,2.1067443351244504,100
17+
LGBM Regr.,LGBM Clas.,observational,True,4,0.95,True,0.7733333333333333,0.8893963194440494,0.300651425058323,0.67,1.046760454668414,100

‎scripts/did/did_pa_multi_tune_config.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
# Simulation parameters for DID Multi Coverage
22

33
simulation_parameters:
4-
repetitions: 50
4+
repetitions: 100
55
max_runtime: 19800 # 5.5 hours in seconds
66
random_seed: 42
77
n_jobs: -2
88

99
dgp_parameters:
10-
DGP: [4] # Different DGP specifications
10+
DGP: [1,2,3,4] # Different DGP specifications
1111
n_obs: [2000] # Sample size for each simulation (has to be a list)
1212

1313
# Define reusable learner configurations

0 commit comments

Comments
 (0)