Towards visual analytics for digging into human rights violations data

Abstract

Proceedings of the American Society for Information Science and TechnologyVolume 51, Issue 1 p. 1-4 Computer ScienceFree Access Towards visual analytics for digging into human rights violations data First published: 24 April 2015 https://doi.org/10.1002/meet.2014.14505101069AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat ABSTRACT In Digging into Human Rights Violation project we developed an algorithm that leverages machine learning and visualization techniques to facilitate large-scale analysis of human rights violation data. To make the algorithm more accessible to human rights workers and researchers, we conducted iterative user interface design to explore usability issues of integrating the visualization output of the algorithm into their existing data analysis process. We report our lessons learnt from the user interface design process. INTRODUCTION Various technical approaches have been developed to deal with the widespread big data challenge. These data-driven scientific efforts are argued to signify the “fourth paradigm” in scientific research (Hey, Tansley, and Tolle, 2009). Digging into Human Rights Violations is developing a tool to facilitate analysis of large-scale human rights violation data. Specifically, the project has developed an algorithm that leverages machine learning and visualization techniques to process the incoming reports, extract the entities, and visualize the results. STORYGRAM Machine learning algorithms’ reliability depends on quality training data and syntactic, structural, and semantic correlation between the training data and the test corpus; frequently, entity extractions and co-relation are inaccurate. To address this issue, we developed a visual analytic tool for data cleaning called Storygram (Miller et al. 2013). Storygram is a graph-based visualization where each event with person, location and time are represented as vertices of the graph. The edges between them are labeled with the confidence values obtained based on indications of uncertainty. The confidence values are normalized between the scale of 0 and 1 and provide some measure of the veridicality of a connection. An edge with the value 1 between a person and a location would mean that the person was at that location with 100% confidence. Since the tool is interactive, it facilitates merging of nodes, querying the graph and saving the results. Merging nodes informs the system that the elements of that Storygram are identical, thus leading to better identification of poorly specified perpetrators and victims. ITERATIVE USER INTERFACE DESIGN Storygram works as a standalone software program that interacts with users through command line. To make it more accessible to human rights workers and researchers, we have been working on integrating it into the existing violation bulletin reporting program, Martus Bulletin System (https://www.martus.org/). Martus includes a bulletin software for entering and securely storing information, server software for securing encrypted data offsite, and a search engine that allows users to find bulletins stored using the Martus server. Organizations can choose whether or not to allow for public access to bulletins. Typically, only the reporting organization has access to their data. Information can be entered according to preset or customized fields and are encrypted and stored remotely at multiple locations with passwords. Recorded bulletins are stored as XML, and are searchable by fields containing information describing the violation type, location, outcome, and more. Work to integrate our algorithm into Martus began with a now completed user interface (UI) design. That design explored how to integrate the visualization output of the algorithm into the Martus UI. Low-fidelity prototypes were created in Balsamiq. Our design follows the system's basic user interface scheme, which resembles the traditional Outlook in appearance and operation, to conform to previous users' mental model of how it works. In other words, consistency in appearance and information organization is crucial in our design. For example, Figure 1 is a page of the paper prototype showing the interface that a user would encounter after logging into Martus with the visualization integrated. The information layout is exactly the same as that in Martus. The figure shows the addition of “Visualization Search” in the drop down menu as an access point to the new feature, and “Visualization Search Results” as a general folder for storing unsorted visualization search results. Figure 1Open in figure viewerPowerPoint Initial interface after log-in. New item added to Search drop down menu to include the new feature, and a new category of search results folder to keep the new type of search results However, as we focused on offering the interaction support for an iterative process of searching, filtering, and visualizing information from the data set, we have designed new interfaces for navigating the visualization feature, and conducting searches, presenting and managing search results from the visualization. The new interfaces we created reflect the new approach of database searching. (Iterative) Searching through visualization feature Figure 2 shows the new search form for conducting a visualization search. The idea for this interface was taken from common academic databases such as ProQuest. It is assumed that users would be familiar with this kind of interface given that it is a feature of major databases. The eight search fields listed on the right are categories of important phrases/labels we identified as necessary for extracting and classifying information from Bulletin documents to generate the visualization (blind review reference). Figure 2Open in figure viewerPowerPoint Search form for visualization search. It resembles some of the major academic databases Upon the completion of a visualization search, the search result is presented to the user in an interface (see Figure 3) that shows an aggregate of documents relevant to the search query in an interactive visual and a list of customizable filters for refining the search result. A tag cloud of the retrieved relevant documents is also presented indicating the prominence of certain terms/phrases in relation to one another. Figure 3Open in figure viewerPowerPoint Visualization search result. The interface that appears as a result of a visualization search As Martus is a self-contained software that keeps all related files within the software and password-protected, it is important that the user can save search results and be able to easily review them. We created interfaces to accommodate the procedure of saving a search in Martus but kept them consistent with the Martus process (see Figure 4). Figure 4Open in figure viewerPowerPoint Review of saved visualization search results FORMATIVE EVALUATION We conducted cooperative evaluation to examine usability issues of our design. This procedure was chosen because it is an effective and rapid way of detecting major usability problems in early design stage of a new software program (Monk et al., 1993). The think-aloud protocol is a major component of the cooperative evaluation. It is a common verbal protocol that requires a user to verbalize their thoughts at each moment he/she is interacting with an interface, providing real-time feedback (Nielsen, 2012). This also saves the participant from having to recall opinions later. For participants who are inexperienced with such procedure, providing a demonstration before the study session is an effective strategy to alleviate awkwardness (Ericsson & Simon, 1993). Participant The six participants in our study were recruited partially by invitations and partially by recruitment flyers. We invited three academic professionals who have had extensive experience in working with human rights violation issues. We also recruited three participants who have had experience in searching for human rights information from the general public (one participant researched the topic for an undergraduate project, and the other two did so for private purposes). Our survey results showed that the participants varied in their data analysis practices: three were experienced with qualitative data analysis; one was familiar with statistical analysis; and two had little experience in data analysis research. No one in the sample pool had any previous experience with the Martus Bulletin System prior to the study. In fact, four participants had no experience of using software programs in data analysis. Only two participants used a self-developed database to assist qualitative analysis of video interviews. The varying levels of research experiences brought forth interesting different user expectations. Procedure Three usability evaluations were conducted face-to-face and the other three were conducted online via a web meeting software. The duration of each evaluation ranged from 30 minutes to one hour. Upon giving consent, the participant filled out a short survey about his/her background. The evaluation consisted of three tasks that were assumed to be commonly performed by users and important for investigating navigations outside of the visualization. These navigations focused on: accessing the visualization, conducting and saving a visualization search, re-organizing the saved search results, reviewing and modifying the saved search results. After the tasks were completed, the participant filled out a questionnaire to provide ratings on different aspects of the design and written feedback. Findings The participants’ comments and written feedback further expanded the understanding of the design's usability. Results showed that five of six participants looked for textual search (Bulletin Search) as the start of their search although it had been clearly explained that the goal of the study was to evaluate a visualization search interface design. They were however, able to quickly change their selection to Visualization Search to continue the task. Simple and common tasks such as completing the search form, saving visualization search result, and moving visualization results into a folder were completed without error by all six participants. On the other hand, the tasks of creating a new folder and modifying previously saved visualization search result were problematic. Only participant #4 was able to complete the latter task without error. Although it was hard for most participants to make the first move to make modifications, they were able to complete the task without problem once they were presented with the Modify Visualization Search Result interface. Participants’ comments and suggestions, and implications of our findings are discussed below. Usability Issues Our questionnaire included was derived from the five quality components in the definition of usability: learnability, efficiency, memorability, errors, and satisfaction (Nielsen, 2012). The results indicate that the design is fair to good (i.e., the ratings of different design aspects were in the middle to high range). Overall, the learnability of the prototype was highly regarded by the participants, whereas the pleasantness of the interface was rated lowest. Participants were quick to comment on the appearance of the interface design. They criticized that the design looked out-dated and inconsistent with current trend of symbolic/iconic buttons seen on many web/software interfaces. One participant suggested that the use of icons instead of textualized buttons may improve accessibility of the design. Related to the texutalized buttons on the interface, we found that participants had confusions over the functions of certain buttons. For example, participants assumed the quick link menu buttons “Create” and “Modify” were universal for managing folders, Bulletins, and visualization, when they were designed only for creating and modifying Bulletins. Furthermore, we realized from participants’ actions and comments that the placement of buttons could affect the smoothness of a navigation sequence and the completion of a task. Aside from the basic navigational processes, we also gained insights regarding search results management for users in this field of research. We learnt from the evaluation that the flexibility of switching from textual results to visual results should suit the mental models of researchers in the field. As indicated by our participants, their first approach in research is textual search, therefore allowing them to scan through textual information first and look at visual supports should be a more familiar research procedure and a better approach in assisting them in their research. In our second iteration of the UI prototype, we combined the two searches into one button and used one search form for both the Martus search fields and the new visual search fields. Provided that all of the participants are more familiar and accustomed to starting with a textual search in their research, the new interface will, by default, display textual search results first and have a button to “convert” the search results into visualization should the user wish to use visual analytics for data analysis (See Figure 5 above). The user can revert back to individual Bulletin search results by clicking “Bulletin Search Results” at the top (See Figure 6). Figure 5Open in figure viewerPowerPoint Modified search result interface. By default, the software will present textual results first and the user has the option to view visualized results by clicking the “Visualize Search Results” button Figure 6Open in figure viewerPowerPoint Modified visualized search result interface DISCUSSION Our UI design and evaluation results indicate that although visual analytic functions such as visualization search are expected to present a moderate (if not deep) learning curve to the human rights researchers due to unfamiliarity with the new data analysis practices, the approach could still be positively accepted. Survey results revealed that participants were used to using textual qualitative data analysis with some statistical analysis in the forms of simple graphs. Their inexperience with visual analytics was further confirmed by their tendency of looking for textual search results even when they were explicitly asked to manage visualization results. However, we observed that with increased length of time exposed to the visualization tool, the participants demonstrated learning and were able to complete assigned tasks. Though this should not be considered a usability issue with the interface, it does implicate that better software introduction and training support are needed to draw users’ interests and acclimatize them in the new trend of data analysis. Although our study's main focus was to investigate user's expectations of navigations outside of the visualization, our participants inadvertently showed us various expectations for the visualization. This was interesting especially when our participants had not had experience with visual analytics such as the one we presented to them. For example, they hovered over the visualization placeholder, clicked on the boxes, and expected to see the visual responses of their actions. Two participants asked whether they could get more information such as individual documents from interacting with the visualization. One participant suggested that a legend should be included to explain elements of the visualization such as the numbers that appear between the boxes. Their actions and comments suggest that for users to easily understand and utilize visual analytics, the technology has to be quite responsive to their exploratory actions, i.e. giving immediate feedback such as hover-text box, and a Help function on the side in order to assist the interpretation of certain information and maintain a sleek, un-crowded visualization interface. As users have to deal with large quantities of heterogeneous data and the process of using interactive visual analytics involves much more intuition and self-initiated processes than the traditional way of information discovery, good instructional support and user navigation assistance should be designed in the tool. Also, the tool should offer management of searches for tracking research practices and convenience. CONCLUDING REMARKS In this paper, we detailed the UI design and evaluation study to integrate the visualization output from our analytic pipeline and visualization tool, Storygram, into an existing software program tailored for analyzing human rights violation data. Our study suggests that introducing such tools to the data analysts in human rights research are challenging but promising; instructional support and user navigation assistance should be considered thoroughly in the design; and features that manage search interactions and results are desired in this augmented process of digging into human rights violation data. ACKNOWLEDGMENTS This project is funded by US Natural Science Foundation (NSF) and Canada Social Sciences and Humanities Research Council (SSHRC). We thank participants of the evaluation sessions for their feedback on the UI design. REFERENCES Ericsson, K. A. & Simon, H.A. (Rev. Ed.). (1993). Protocol analysis: Verbal reports as data. Cambridge, Mass.: MIT Press. Hey, T., Tansley, S., & Tolle, K. (2009). The fourth paradigm: Data-intensive scientific discovery, Redmond, VA: Microsoft Research. Miller, B., Li, F., Shrestha, A., & Umapathy, K. (2013). Digging into Human Rights Violations: Phrase mining and trigram visualization, Digital Humanities 2013, Nebraska, US Monk, A., Wright P., Haber, J., & Davenport L. (1993). Apendix 1 – Cooperative Evaluation: A run-time guide. In: Improving your human-computer interface:a practical technique, Prentice-Hall. Nielsen, J. & Landauer, T.K. (1993) A mathematical model of the finding of usability problems, Proceedings of ACM INTERCHI'93 Conference (pp. 206– 213). The Netherlands: IOS Press Amsterdam. Nielsen, J. (2012). Thinking aloud: the #1 usability tool. Retrieved from http://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/ Volume51, Issue12014Pages 1-4 FiguresReferencesRelatedInformation