Web Archives Research Articles

Sorting URLs out: seeing the web through infrastructural inversion of archival crawling

Web archives collections have become important sources for Internet scholars by documenting the past versions of web resources. Understanding how these collections are created and curated is of increasing concern and recent web archives scholarship has studied how the artefacts stored in archives represent specific curatorial choices and collecting practices. This paper takes a novel approach in studying web archiving practice, by focusing on the challenges encountered in archival web crawling and what they reveal about the web itself. Inspired by foundational work in infrastructure studies, infrastructural inversion is applied to study how crawler interactions surface otherwise invisible, background or taken-for-granted aspects of the web. This framework is applied to study three examples selected from interviews and ethnographic fieldwork observations of web archiving practices at the Danish Royal Library, with findings demonstrating how the challenges of archival crawling illuminate the web’s varied actors, as well as their changing relationships, power differentials and politics. Ultimately, analysis through infrastructural inversion reveals how collection via crawling positions archives as active participants in web infrastructure, both shaping and shaped by the needs and motivations of other web actors.

From Stock Shots to Ghost Data: Tracking Audiovisual Archives about the European Union

This paper deals with a major challenge linked to the collection of audiovisual documents within television and web archives. Looking for repeated sequences within a corpus of thousands of videos, we faced the fact that the footage we were looking for reveals itself to be reachable only as ghost data. In fact, any audiovisual sequence reused within different contexts exists conceptually as the repetition of one single visual unit, but from the point of view of the metadata tagging its occurrences, each item is a distinct document. Like a ghost, the shot is there, scattered among different places, but the metadata cannot point us to the visual form repeated, despite its evidence to the human viewer. When facing large amounts of data, to relate a visual unit to its occurrences, data analysis techniques are needed. We describe our procedures of collection and annotation, and the solutions combining qualitive work and a computer-aided approach to face this main challenge, within the research project Crossing Borders Archives (CROBORA).

Continuity and discontinuity in web archives: a multi-level reconstruction of the firsttuesday community through persistences, continuity spaces and web cernes

Web archives are not direct traces of the web, they are direct traces of crawlers. By design, the structure of web archives limits our capacity to explore the memory of the Web. These structural issues induce temporal discontinuities such as inconsistency, redundancy and blindness. In this paper, we address the question of re-injecting continuity within large corpora of web archives. We introduce the notions of persistences (series of time-stable snapshots of archived web pages) and continuity spaces (networks of time-consistent persistences). We demonstrate how – on the basis of a quality score – persistences can be used to select subsets of web archives within which in-depth historical analysis can be conducted at scale. We next propose to make use of a new visualization approach called web cernes to reconstruct the multi-level temporal evolution of an archived community of web sites. We finally apply our framework to study the history of the firsttuesday movement: a constellation of entrepreneurial web sites that acted in the interest of the economical growth of the web in the early 2000s.

Is this news article still relevant? Ranking by contemporary relevance in archival search

Our civilization creates enormous volumes of digital data, a substantial fraction of which is preserved and made publicly available for present and future usage. Additionally, historical born-analog records are progressively being digitized and incorporated into digital document repositories. While professionals often have a clear idea of what they are looking for in document archives, average users are likely to have no precise search needs when accessing available archives (e.g., through their online interfaces). Thus, if the results are to be relevant and appealing to average people, they should include engaging and recognizable material. However, state-of-the-art document archival retrieval systems essentially use the same approaches as search engines for synchronic document collections. In this article, we develop unique ranking criteria for assessing the usefulness of archived contents based on their estimated relationship with current times, which we call contemporary relevance. Contemporary relevance may be utilized to enhance access to archival document collections, increasing the likelihood that users will discover interesting or valuable material. We next present an effective strategy for estimating contemporary relevance degrees of news articles by utilizing learning to rank approach based on a variety of diverse features, and we then successfully test it on the New York Times news collection. The incorporation of the contemporary relevance computation into archival retrieval systems should enable a new search style in which search results are meant to relate to the context of searchers’ times, and by this have the potential to engage the archive users. As a proof of concept, we develop and demonstrate a working prototype of a simplified ranking model that operates on the top of the Portuguese Web Archive portal (arquivo.pt).

Imagining permanence on the web: Tracing the meanings of long-term preservation among the subjects of web archives

This article explores how web archiving impacts the online communication practices of individuals whose personal websites have been archived by major public libraries. Drawing on interviews with website creators and analysis of their written reflections on the archiving process, it demonstrates how web archiving alters the meanings people attach to their online activity. In most cases, the preservation of their website in a national web archive sees individuals perceive their communication practices as having wider cultural and historical significance. These meanings are shaped by the distinctive interaction between archiving and archived actors and propelled by imaginaries surrounding the culture and history of the collecting institution. Based on these findings, this article argues that web archiving can be productively understood as an intervention in the dynamics of online sociality and calls for reflexive archival and research practices that attend to the short- and long-term impacts of altering the visibility of online material.

To Re-experience the Web: A Framework for the Transformation and Replay of Archived Web Pages

When replaying an archived web page, or memento , the fundamental expectation is that the page should be viewable and function exactly as it did at the archival time. However, this expectation requires web archives upon replay to modify the page and its embedded resources so that all resources and links reference the archive rather than the original server. Although these modifications necessarily change the state of the representation, it is understood that without them the replay of mementos from the archive would not be possible. The process of replaying mementos and the modifications made to the representations by web archives varies between archives. Because of this, there is no standard terminology for describing the replay and needed modifications. In this article, we propose terminology for describing the existing styles of replay and the modifications made on the part of web archives to mementos to facilitate replay. Because of issues discovered with server-side only modifications, we propose a general framework for the auto-generation of client-side rewriting libraries. Finally, we evaluate the effectiveness of using a generated client-side rewriting library to augment the existing replay systems of web archives by crawling mementos replayed from the Internet Archive’s Wayback Machine with and without the generated client-side rewriter. By using the generated client-side rewriter, we were able to decrease the cumulative number of requests blocked by the content security policy of the Wayback Machine for 577 mementos by 87.5% and increased the cumulative number of requests made by 32.8%. We were also able to replay mementos that were previously not replayable from the Internet Archive. Many of the client-side rewriting ideas described in this work have been implemented into Wombat, a client-side URL rewriting system that is used by the Webrecorder, Pywb, and Wayback Machine playback systems.

Repercussion of Using Internet Sources: Dilemma for Research Communities

Objectives – Consultation of internet sources for educational and research purposes is the new normal. As a result of information communication technology, information creation and access are more convenient. The current study was carried out to investigate the proportion of use of internet sources for research purposes by research scholars of three central universities of North East India, namely Tezpur University, Mizoram University, and Rajiv Gandhi University. Methods – The researchers collected data from 123 respondents through an online questionnaire that was distributed through different social media platforms. The study was conducted among Research Scholars (PhD and M.Phil) of Mizoram University, Rajiv Gandhi University, and Tezpur University. Results – The research results show that research communities are moving toward digital platforms for searching and consulting their required resources. Most of the respondents consult internet sources for writing their research reports, but they do not format the references properly. Some research scholars do not follow any referencing style for citing web documents, and respondents do not have much awareness about the differences between URLs and DOIs. Research communities also face problems due to the inaccessibility of online documents cited by former researchers. Conclusion – Since most of the respondents are not familiar with the use of web archives, the current study suggests that higher education institutions should arrange awareness programs on the use of web archives. Research communities should follow the proper referencing formats to acknowledge others’ works. Publishers should mandate a citation style for authors and verify the accuracy of the references before publishing articles or other works.

Hashes are not suitable to verify fixity of the public archived web.

Web archives, such as the Internet Archive, preserve the web and allow access to prior states of web pages. We implicitly trust their versions of archived pages, but as their role moves from preserving curios of the past to facilitating present day adjudication, we are concerned with verifying the fixity of archived web pages, or mementos, to ensure they have always remained unaltered. A widely used technique in digital preservation to verify the fixity of an archived resource is to periodically compute a cryptographic hash value on a resource and then compare it with a previous hash value. If the hash values generated on the same resource are identical, then the fixity of the resource is verified. We tested this process by conducting a study on 16,627 mementos from 17 public web archives. We replayed and downloaded the mementos 39 times using a headless browser over a period of 442 days and generated a hash for each memento after each download, resulting in 39 hashes per memento. The hash is calculated by including not only the content of the base HTML of a memento but also all embedded resources, such as images and style sheets. We expected to always observe the same hash for a memento regardless of the number of downloads. However, our results indicate that 88.45% of mementos produce more than one unique hash value, and about 16% (or one in six) of those mementos always produce different hash values. We identify and quantify the types of changes that cause the same memento to produce different hashes. These results point to the need for defining an archive-aware hashing function, as conventional hashing functions are not suitable for replayed archived web pages.

Engineering archive management model based on big data analysis and deep learning model

Abstract In the background of the era of big data, the information management system of engineering archives has become more comprehensive and perfect because of the application of information technology. The application of deep learning model makes the management of engineering archives more systematic, scientific and standardized, which greatly improves the quality and efficiency of engineering archives management. The progress of society and the development of the times have put forward higher requirements for digital storage technology. This paper combines the characteristics of the new technological era, analyses the characteristics of traditional information management in the context of data processing, artificial intelligence, deep learning and other data, proposes a method for developing and managing web archives based on Bootstrapping technology, introduces an information meta-evaluation mechanism to improve the quality of mining, and uses a long and short-term memory model to extract multi-type fine-grained archival information elements in the corpus. Finally, the Alex network was established to manage the archives in a categorised manner. The experimental results show that the query results of the proposed method for the target archives are 100% accurate, and the query time for individual archives is basically within 5s, which has good management effect.

Web archive analytics: Blind spots and silences in distant readings of the archived web

Abstract In this article, we discuss epistemological and methodological aspects of web archive analytics, a recent development towards more data-centred access to web archives. More specifically, we suggest understanding both the process of archiving and subsequent steps of analysis at scale as acts of observation that can be questioned for their epistemological priori. Therefore, we propose the concepts of ‘blind spots’ (features of the live web not included upon creation in the archive) and ‘silences’ (latent features present in the archive but requiring a particular method to be made articulate). In particular, we address two forms of silences playing a structural role in web archive analytics, crucial to both historians and social scientists alike: abundance (or scale) and time. We trace epistemological implications of web archive analytics across an exemplary case study workflow and suggest methodological answers to the issues raised in this process. On the data extraction side, we introduce warc2corpus (w2c), a new tool for extracting granular, structured data, especially temporal information related to the creation, modification, and publication specifically of webpages. For data analysis, we demonstrate how distant reading techniques—more specifically structural topic modelling (STM)—can contribute to providing a rich, temporally structured representation of textual web archive content that in turn can be subjected to scholarly inquiry, interpretation, and re-contextualization.

Using web archives for an explorative study of the web presence of German parties during the European election 2019

In the digital age, political science is faced with a shift of election campaigns and political discourse to digital or virtual arenas. Because the internet is a highly volatile medium and online content can become inaccessible after the campaign season, new challenges for research arise as well as the need for the preservation of online content. Moreover, the sheer volume of data researchers have to deal with has reached levels where traditional methods are being highly challenged. This paper puts forth a web harvesting workflow with a strong focus on granular extraction of unstructured information (publication dates) for automated analysis. As our approach is methodological, we would like to point out the benefits that researches in political science may draw from adapting our methodology. We demonstrate this by analysing an event-based web crawl of German parties participating in the election campaign for the European Parliamentary Election in 2019. We employ distant reading methods to generate topic models, which are subsequently evaluated by hermeneutic analysis of a subset of the data.

Feminism Emergent: Prospects of NGOs in Pakistan

Feminism is a movement in which people struggle for the rights of women and their better prominence in all domains of life. Women got recognition through waves of feminism around the world in respect of their rights and status. In this regard, non-governmental organizations played a strong role to promote and reinforce feminist approaches that focus on the basic rights of women like education, economic development, legal aid and selfawareness, protection, and political participation. Therefore, the present research main objective is based on the prospects analysis of NGOs in perspective of feminist struggle in Pakistan. This is an e-research based on an analysis of web archives of NGOs working in Pakistan on women concerns. This also involved in-depth interviews with stake holders of these NGOs. Mix method research design was applied to analyze the prospects of 144 registered NGOs from a feminist perspective. Simple random and purposive sampling techniques were used to select samples for quantitative and qualitative data. The research findings showed that NGOs in Pakistan are struggling to accomplish feminism that contributing a great effort to raise the status of women in every field of life. It is needed to present factual understanding and accountability of NGOs in the concern of feminism and women empowerment

Know(ing) Infrastructure: The Wayback Machine as object and instrument of digital research

From documenting human rights abuses to studying online advertising, web archives are increasingly positioned as critical resources for a broad range of scholarly Internet research agendas. In this article, we reflect on the motivations and methodological challenges of investigating the world’s largest web archive, the Internet Archive’s Wayback Machine (IAWM). Using a mixed methods approach, we report on a pilot project centred around documenting the inner workings of ‘Save Page Now’ (SPN) – an Internet Archive tool that allows users to initiate the creation and storage of ‘snapshots’ of web resources. By improving our understanding of SPN and its role in shaping the IAWM, this work examines how the public tool is being used to ‘save the Web’ and highlights the challenges of operationalising a study of the dynamic sociotechnical processes supporting this knowledge infrastructure. Inspired by existing Science and Technology Studies (STS) approaches, the paper charts our development of methodological interventions to support an interdisciplinary investigation of SPN, including: ethnographic methods, ‘experimental blackbox tactics’, data tracing, modelling and documentary research. We discuss the opportunities and limitations of our methodology when interfacing with issues associated with temporality, scale and visibility, as well as critically engage with our own positionality in the research process (in terms of expertise and access). We conclude with reflections on the implications of digital STS approaches for ‘knowing infrastructure’, where the use of these infrastructures is unavoidably intertwined with our ability to study the situated and material arrangements of their creation.

Belarusian Politics and Society Web Archive: Preserving the Belarusian Grassroots Protest

Abstract Belarusian Politics and Society (BPAS) is a curated web archive created under the auspices of the Ivy Plus Library Confederation to preserve online grassroots content created in, or related to, Belarus since 2020. The article describes the historical context of the creation of BPAS (focusing on the events in Belarus in 2020–2022 and particularly, the role of online media) and the web archiving context (by exploring Internet Archive’s coverage of Belarus) as well as demonstrates how both contexts guided the curatorial work on BPAS and the decision to focus the scope of collection on fragile grassroots content. Special attention is given to the issue of content loss and content shift in the Belarusian internet.

The continued problem of URL decay: an updated analysis of health care management journal citations.

This study updates a 2009 study which examined uniform resource locator (URL) decay in health care management journals and seeks to determine whether continued URL availability relates to publication date, resource type, or top-level domain. The authors also provide an analysis of differences in findings between the two study periods. The authors collected the URLs of web-based cited references in articles published in five health care management source journals from 2016 to 2018. The URLs were checked to see if they were still active and then analyzed to determine if continued availability was related to publication date, resource type, or top-level domain. Chi-square analysis was conducted to determine associations between resource type and URL availability, and top-level domain and URL availability. A Pearson's correlation was conducted to determine the relationship between publication date and URL availability. There were statistically significant differences in URL availability across publication date, resource type, and top-level domain. Domains with the highest percentage of unavailable URLs were .com and .net, and the lowest were .edu and .gov. As expected, the older the citation, the more likely it was unavailable. The overall percentage of unavailable URLs decreased from 49.3% to 36.1% between studies. URL decay in health care management journals has decreased in the last 13 years. Still, URL decay does continue to be a problem. Authors, publishers, and librarians should continue to promote the use of digital object identifiers, web archiving, and perhaps study and replicate efforts used by health services policy research journals to increase continued URL availability rates.

Open-source intelligence: a comprehensive review of the current state, applications and future perspectives in cyber security.

The volume of data generated by today's digitally connected world is enormous, and a significant portion of it is publicly available. These data sources are web archives, public databases, and social networks such as Facebook, Twitter, LinkedIn, Emails, Telegrams, etc. Open-source intelligence (OSINT) extracts information from a collection of publicly available and accessible data. OSINT can provide a solution to the challenges in extracting and gathering intelligence from various publicly available information and social networks. OSINT is currently expanding at an incredible rate, bringing new artificial intelligence-based approaches to address issues of national security, political campaign, the cyber industry, criminal profiling, and society, as well as cyber threats and crimes. In this paper, we have described the current state of OSINT tools/techniques and the state of the art for various applications of OSINT in cyber security. In addition, we have discussed the challenges and future directions to develop autonomous models. These models can provide solutions for different social network-based security, digital forensics, and cyber crime-based problems using various machine learning (ML), deep learning (DL) and artificial intelligence (AI) with OSINT.

Actualization of Evaluativity as a Qualitative Category in the Discourse of English Internet Film Review

Introduction. The article presents the results of the analysis aimed at identifying actualization methods of evaluativity as a qualitative category within the frames of the evaluative subgenres of expert and professional film reviews. The author assumes that evaluativity serves as a method of explicating subject's attitude towards the content of the film. The relevance of the study is determined by a lack of research dedicated to the analysis of the methods expressing author’s position within the evaluative genre of internet film review. The interconnection of evaluativity category, subject factor and modus as a method of explicating subjective meanings of the statement are analyzed.Methodology and sources. The methodology of the research is based on the classification of evaluative meanings developed by N. Arutyunova. The research methodology includes PoS tagging of the texts, identification of evaluative units including qualitative adjectives and adverbs, as well as description and comparison of leading evaluation types that actualize author's position. The methodology was studied using the example of a text corpora consisting of expert and professional film reviews extracted from the web archive of the journal of Film Criticism and the website metacritic.com.Results and discussion. With the implementation of PoS tagging and quantitative method of analysis leading evaluation categories within the corpora of the professional and expert film reviews were described and compared. The results have shown that the leading type of evaluation within the frames of the referred evaluative subgenres is represented by the emotional evaluation category. The results have also indicated that the leading evaluation types of expert evaluative discourse include prescriptive and ethical evaluation, whereas professional evaluative discourse is characterized by the prevalence of generalized evaluative units.Conclusion. The analysis allowed the author to conclude that the referred genres are characterized mainly by increased expressivity and emotiveness. However, author’s position actualization in the discourse of expert film review implicates detailed analysis of structural contents of the presented norms, as well as content of the film, whereas the discourse of professional film review is characterized by increased generalization tendency, which implicates low degree of authors’ competency.

All WARC and no playback: The materialities of data-centered web archives research

This paper examines the Web ARChive (WARC) file format, revealing how the format has come to play a central role in the development and standardization of interoperable tools and methods for the international web archiving community. In the context of emerging big data approaches, I consider the sociotechnical relationships between material construction of data and information infrastructures for collecting and research. Analysis is inspired by Star and Griesemer's historical case of the Museum of Vertebrate Zoology which reveals how boundary objects and methods standardization are used to enroll actors in the work of collecting for natural history. I extend these concepts by pairing them with frameworks for studying digital materiality and the representational qualities of data artifacts. Through examples drawn from fieldwork observations studying two data-centered research projects, I consider how the materiality of the WARC format influences research methods and approaches to data extraction, selection, and transformation. Findings identify three modalities researchers use to configure WARC data for researcher needs: using indexes to support search queries, constructing derivative formats designed for certain types of analysis, and generating custom-designed datasets tailored for specific research purposes. Findings additionally reveal similarities in how these distinct methods approach automated data extraction by relying upon the WARC's standardized metadata elements. By interrogating whose information needs are being met and taken into account in the design of the WARC's underlying information representation, I reveal effects on the emerging field of web history, and consider alternative approaches to knowledge production with archived web data.

Big Data Studies: The Humanities in Uncharted Waters

Abstract: This article discusses the formidable challenges that the advent of big data brings to the digital humanities broadly and proposes some ways the Korean studies community can prepare to navigate these uncharted waters. Standard digital humanities training in data mining, text analysis, mapping, network science, and machine learning will be developed and refined over the coming years, as will research concerning the ephemeral nature of new media, web archives, and the ethics of artificial intelligence. Yet I contend that established responses to the digital transformation of the humanities, while timely and necessary, will prove inadequate for handling petabyte- and exabyte-scale born-digital sources. In the Zettabyte Era, more data is processed in real time than all of the records produced from early times to the 2010s. To make sense of the current information regime, we need critical reflections and comparisons to the classical internet age of the 1990s, the personal computer revolution of the 1980s, and early modern print cultures. This exercise will allow us to situate the humanities in an age of big data as an extension of traditional humanities research and at the same as something foreign.

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

Strong encryption algorithms and reliable anonymity routing have made cybercrime investigation more challenging. Hence, one option for law enforcement agencies (LEAs) is to search through unencrypted content on the Internet or anonymous communication networks (ACNs). The capability of automatically harvesting web content from web servers enables LEAs to collect and preserve data prone to serve as potential leads, clues, or evidence in an investigation. Although scientific studies have explored the field of web crawling soon after the inception of the web, few research studies have thoroughly scrutinised web crawling on the "dark web" or via ACNs such as I2P, IPFS, Freenet, and Tor. The current paper presents a systematic literature review (SLR) that examines the prevalence and characteristics of dark web crawlers. From a selection of 58 peer-reviewed articles mentioning crawling and the dark web, 34 remained after excluding irrelevant articles. The literature review showed that most dark web crawlers were programmed in Python, using either Selenium or Scrapy as the web scraping library. The knowledge gathered from the systematic literature review was used to develop a Tor-based web crawling model into an already existing software toolset customised for ACN-based investigations. Finally, the performance of the model was examined through a set of experiments. The results indicate that the developed crawler was successful in scraping web content from both clear and dark web pages, and scraping dark marketplaces on the Tor network. The scientific contribution of this paper entails novel knowledge concerning ACN-based web crawlers. Furthermore, it presents a model for crawling and scraping clear and dark websites for the purpose of digital investigations. The conclusions include practical implications of dark web content retrieval and archival, such as investigation clues and evidence, and the related future research topics.