An expert-based bibliometric for a science of security

Abstract

Security science research is often disseminated in the form of conference papers, presentations, and journal articles. Yet traditional bibliometrics, such as citation counts and journal impact factors, are inadequate for assessing the impact of these publications for a number of reasons [1]--[3]. Existing citation databases, such as ISI Web of Science Journal Citation Reports, Scopus, and Google Scholar, do not always adequately capture the conferences and workshops where many security researchers focus their publication efforts. Computer science focused databases, such as CiteSeerX and dblp, also fall short of capturing venues appropriate for disseminating research in the multidisciplinary and emerging field of security science. Further, any citation-based metric will be a lagging indicator for fields that evolve at an extraordinarily fast pace.Our objective was to develop a scalable method at the publication venue level that would address limitations of existing bibliometrics, be specifically targeted toward the Science of Security (SoS), and that could be used across a large number of publications. To this end, we developed a customized SoS bibliometric based on expert ratings of relevant publication venues. Expert ratings are a preferred alternative for faculty evaluation for tenure and promotion in computing fields[4] which may be useful in evaluating publication venues.To develop our expert based rating system, we first identified venues relevant to the SoS. Included in our list were venues where SoS Lablet papers have been published, venues nominated by Lablet researchers, National Security Agency (NSA) program directors, and SoS experts, and security venues included in other computer science publication venue ranking systems. Venues were excluded if they published less than 10 papers annually, were a one-time only event or a local meeting, a sub-conference of a larger conference already included in the list, or if no relevant information about the venue could be found online, or from the venue's editor or publisher. We validated our list by comparing it to other ranking systems for computer science and security. Ultimately, 170 publication venues were included in our list.Next, we developed a list of experts in security science. The list was compiled from reviewers for the NSA SoS best paper competition, and recommendations from NSA program directors, SoS Lablet PIs, and other SoS experts. Experts were invited to participate and were offered a small honorarium for their contribution. 1 Participating experts (n=21) were instructed to assign a rating for the quality of the publication venue and another rating based on the relevance of the publication venue to the SoS. Numeric ratings of 1 through 4 corresponded to four categories: premier, top tier, middle tier, and bottom tier venues. Ratings were analyzed based on the assumption that the distance between ratings is not consistent, in that the distance between premier venues and top tier venues may be far greater than the distance between top tier and middle tier venues. Therefore, we used the mode of all expert ratings to determine the overall quality and relevance ratings for each venue. We then assigned an overall rating based on the lower of the quality and relevance ratings.Based on expert ratings, three of the 170 (2%) venues were considered premier venues in the SoS: ACM Computer and Communications Security, USENIX Security Symposium, and IEEE Symposium on Security & Privacy. Ten percent of venues were rated in the top tier, 48% in the middle tier, and 40% in the bottom tier.As a test case, we used this expert based bibliometric to evaluate the extent to which North Carolina State University SoS Lablet publications are being disseminated through expert rated top tier venues. Of the NCSU SoS Lablet papers published from 2012--2017, approximately 3% appeared in expert rated premier SoS publication venues, 4% appeared in top tier venues, 45% in middle tier venues, and 48% in bottom tier venues.We discuss limitations to this metric, next steps in its development, implications for security researchers seeking to publish in high impact science of security venues, and provide recommendations for future use.