dodo-mlsec.org

Does the presence of a pitfall invalidate results of a paper?

Not necessarily. While we demonstrate throughout our impact analysis that ignoring the pitfalls can affect the overall outcome significantly, the presence of a pitfall (or even multiple) does not always lower the contribution of a paper. Nonetheless, we should discuss as a community how to deal with these pitfalls in future research and minimize their impact whenever possible.

How did you select the papers for the prevalence analysis?

We skimmed all papers of the top security conferences in the last 10 years and identified 30 papers that use machine learning prominently (e.g., mentioned in the abstract or introduction). Even though this selection process is not complete and entirely free from bias, the identified papers are prototypical for this research branch and often highly cited by other researchers.

Where can I find the detailed list of pitfalls assigned to each publication?

The intention behind our paper is to provide readers with an overview of common pitfalls that should be avoided when applying machine learning in computer security. We do not intend to blame or attack prior works. Therefore, we refrain from publishing the full list of assigned pitfalls.