Traffic duplication removal (CIC-IDS-2017)

[lisa-lthorrold]

This is a more refined version of the previously closed pull request by @mlavin. #1

The main improvement over the previous script is a reduction unintended consequences of applying editcap packet deduplication over the entire pcap. This is done by isolating the impacted mac addresses and ip addresses that have been confirmed to have definite duplication by prior analysis.

[Paippi]

The wait in this code doesn't work as intended. Culprit being the remove_traffic_duplication.sh:47, which spawns a subshell causing the while loop to be performed in another process, thus leading to parent process not waiting for the process to finish.

More info here: https://mywiki.wooledge.org/BashFAQ/024
and here: https://stackoverflow.com/a/63489913/17897224

One way to fix this is to use a for loop instead.

# <---snip--->
    # Find pcap files
    for line in $(find "$1" -iname "*.pcap")
    do
# <---snip--->

[sebastien-bois]

Hi,
Are the results of the preliminary analysis used to compile the list of affected MAC addresses available somewhere ?

[lisa-lt]

Hi @sebastien-bois - did you check the remove_traffic_duplication.sh in this PR?

[sebastien-bois]

Hi @sebastien-bois - did you check the remove_traffic_duplication.sh in this PR?

@lisa-lt I did but I guess my question was not explicit.
I was asking how you identified the MAC addresses and IP addresses that you are using in the filtering part of remove_traffic_duplication.sh. In your pull request, you mentioned a "prior analysis" that helped you to establish this list of addresses. This is the one I am interested about.