@@ -320,3 +320,192 @@ generate_and_show_styled_table(
320320 coverage_highlight_cols=["Coverage", "Uniform Coverage"]
321321)
322322```
323+
324+
325+ ## Tuning
326+
327+ The simulations are based on the the [ make_did_CS2021] ( https://docs.doubleml.org/stable/api/generated/doubleml.did.datasets.make_did_CS2021.html ) -DGP with $2000$ observations. Due to time constraints we only consider one learner, use in-sample normalization and the following DGPs:
328+
329+ - Type 4: Nonlinear outcome model and treatment assignment
330+
331+ The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals). This is only an example as the untuned version just relies on the default configuration.
332+
333+ ::: {.callout-note title="Metadata" collapse="true"}
334+
335+ ``` {python}
336+ #| echo: false
337+ metadata_file = '../../results/did/did_pa_multi_tune_metadata.csv'
338+ metadata_df = pd.read_csv(metadata_file)
339+ print(metadata_df.T.to_string(header=False))
340+ ```
341+
342+ :::
343+
344+ ``` {python}
345+ #| echo: false
346+
347+ # set up data
348+ df = pd.read_csv("../../results/did/did_pa_multi_tune_detailed.csv", index_col=None)
349+
350+ assert df["repetition"].nunique() == 1
351+ n_rep = df["repetition"].unique()[0]
352+
353+ display_columns = ["Learner g", "Learner m", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
354+ ```
355+
356+ ### Observational Score
357+
358+ ``` {python}
359+ #| echo: false
360+ generate_and_show_styled_table(
361+ main_df=df,
362+ filters={"level": 0.95, "Score": "observational"},
363+ display_cols=display_columns,
364+ n_rep=n_rep,
365+ level_col="level",
366+ coverage_highlight_cols=["Coverage", "Uniform Coverage"]
367+ )
368+ ```
369+
370+ ``` {python}
371+ #| echo: false
372+ generate_and_show_styled_table(
373+ main_df=df,
374+ filters={"level": 0.9, "Score": "observational"},
375+ display_cols=display_columns,
376+ n_rep=n_rep,
377+ level_col="level",
378+ coverage_highlight_cols=["Coverage", "Uniform Coverage"]
379+ )
380+ ```
381+
382+ ## Tuning Aggregated Effects
383+
384+ These simulations test different types of aggregation, as described in [ DiD User Guide] ( https://docs.doubleml.org/stable/guide/models.html#difference-in-differences-models-did ) .
385+
386+ As before, we only consider one learner, use in-sample normalization and the following DGPs:
387+
388+ - Type 4: Nonlinear outcome model and treatment assignment
389+
390+ The non-uniform results (coverage, ci length and bias) refer to averaged values over all $ATTs$ (point-wise confidende intervals). This is only an example as the untuned version just relies on the default configuration.
391+
392+ ### Group Effects
393+
394+ ``` {python}
395+ #| echo: false
396+
397+ # set up data
398+ df_group = pd.read_csv("../../results/did/did_pa_multi_tune_group.csv", index_col=None)
399+
400+ assert df_group["repetition"].nunique() == 1
401+ n_rep_group = df_group["repetition"].unique()[0]
402+
403+ display_columns = ["Learner g", "Learner m", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
404+ ```
405+
406+ #### Observational Score
407+
408+ ``` {python}
409+ #| echo: false
410+ generate_and_show_styled_table(
411+ main_df=df_group,
412+ filters={"level": 0.95, "Score": "observational"},
413+ display_cols=display_columns,
414+ n_rep=n_rep_group,
415+ level_col="level",
416+ coverage_highlight_cols=["Coverage", "Uniform Coverage"]
417+ )
418+ ```
419+
420+ ``` {python}
421+ #| echo: false
422+ generate_and_show_styled_table(
423+ main_df=df_group,
424+ filters={"level": 0.9, "Score": "observational"},
425+ display_cols=display_columns,
426+ n_rep=n_rep_group,
427+ level_col="level",
428+ coverage_highlight_cols=["Coverage", "Uniform Coverage"]
429+ )
430+ ```
431+
432+
433+ ### Time Effects
434+
435+ ``` {python}
436+ #| echo: false
437+
438+ # set up data
439+ df_time = pd.read_csv("../../results/did/did_pa_multi_tune_time.csv", index_col=None)
440+
441+ assert df_time["repetition"].nunique() == 1
442+ n_rep_time = df_time["repetition"].unique()[0]
443+
444+ display_columns = ["Learner g", "Learner m", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
445+ ```
446+
447+ #### Observational Score
448+
449+ ``` {python}
450+ #| echo: false
451+ generate_and_show_styled_table(
452+ main_df=df_time,
453+ filters={"level": 0.95, "Score": "observational"},
454+ display_cols=display_columns,
455+ n_rep=n_rep_time,
456+ level_col="level",
457+ coverage_highlight_cols=["Coverage", "Uniform Coverage"]
458+ )
459+ ```
460+
461+ ``` {python}
462+ #| echo: false
463+ generate_and_show_styled_table(
464+ main_df=df_time,
465+ filters={"level": 0.9, "Score": "observational"},
466+ display_cols=display_columns,
467+ n_rep=n_rep_time,
468+ level_col="level",
469+ coverage_highlight_cols=["Coverage", "Uniform Coverage"]
470+ )
471+ ```
472+
473+ ### Event Study Aggregation
474+
475+ ``` {python}
476+ #| echo: false
477+
478+ # set up data
479+ df_es = pd.read_csv("../../results/did/did_pa_multi_tune_eventstudy.csv", index_col=None)
480+
481+ assert df_es["repetition"].nunique() == 1
482+ n_rep_es = df_es["repetition"].unique()[0]
483+
484+ display_columns = ["Learner g", "Learner m", "Tuned", "Bias", "CI Length", "Coverage", "Uniform CI Length", "Uniform Coverage"]
485+ ```
486+
487+ #### Observational Score
488+
489+ ``` {python}
490+ #| echo: false
491+ generate_and_show_styled_table(
492+ main_df=df_es,
493+ filters={"level": 0.95, "Score": "observational"},
494+ display_cols=display_columns,
495+ n_rep=n_rep_es,
496+ level_col="level",
497+ coverage_highlight_cols=["Coverage", "Uniform Coverage"]
498+ )
499+ ```
500+
501+ ``` {python}
502+ #| echo: false
503+ generate_and_show_styled_table(
504+ main_df=df_es,
505+ filters={"level": 0.9, "Score": "observational"},
506+ display_cols=display_columns,
507+ n_rep=n_rep_es,
508+ level_col="level",
509+ coverage_highlight_cols=["Coverage", "Uniform Coverage"]
510+ )
511+ ```
0 commit comments