HTTP Header Research Articles

Cyber Intelligence and OSINT: Developing Mitigation Techniques Against Cybercrime Threats on Social Media

Open Source Intelligence (OSINT) involve the collection or processes of gathering data and profiling of publicly available private and public sector information sources about individuals and business intelligence purposes. These sources includes internet and other social media platforms such as Facebook, emails, twitters, what’s apps for. Much debate and research has been done on the threats, vulnerabilities and the impact of the use of social media sites but this study is to minimize bias. Objective: To systematic review and synthesis findings on current empirical research topic on cyber intelligence and open source intelligence profiling to identifying both the threats and vulnerabilities on online social networks for mitigation purposes. Methods: A systematic narrative review of research using rigorous searching on online databases. The results were then subjected to review using a quantitative and quality appraisal tool and a narrative synthesis methodology. A theoretical framework was developed for the synthesis using concepts from the literature ‘The Effectiveness of Neighborhood Watch’. A Campbell Systematic Review Results: The systematic search retrieved 18 original research papers investigating and exploring the effects of online social media technologies on open source intelligence concepts. The use of social media were reported as enhancing social cohesion among peers, improving business opportunities as information gets to customers quickly. Safe identity experimentations, OSINT and cyber intelligence social media gathering is especially vital in the modern war on terror. Understanding terrorist network topologies, crime data analysis and mining, countering improvised explosive devices. The study also highlighted potential negative impacts and threats and the effect of social engineering threats in SNSs, threats of social networking and identity crime. Vulnerabilities of HTTP header information and cookies being sent to third-party aggregators as well harmful effects of exposure to threats. Conclusion: The systematic review has revealed extraordinary evidences and contradictory concepts. It has also revealed the underlining research challenges impacting on open source intelligence. Due to the invincibility nature of social media technologies, social media platforms are constantly being used for social, business and intelligence gathering purposes but to ensure proper and advance mitigating circumstance, further research is required to gain situational awareness and appropriate counter measure.

Measuring web page complexity by analyzing TCP flows and HTTP headers

To understand website complexity deeply, a web page complexity measurement system is developed. The system measures the complexity of a web page at two levels: transport-level and content-level, using a packet trace-based approach rather than server or client logs. Packet traces surpass others in the amount of information contained. Quantitative analyses show that different categories of web pages have different complexity characteristics. Experimental results show that a news web page usually loads much more elements at more accessing levels from much more web servers within diverse administrative domains over much more concurrent transmission control protocol (TCP) flows. About more than half of education pages each only involve a few logical servers, where most of elements of a web page are fetched only from one or two logical servers. The number of content types for web game traffic after login is usually least. The system can help web page designers to design more efficient web pages, and help researchers or Internet users to know communication details.

Service Usage Classification with Encrypted Internet Traffic in Mobile Messaging Apps

The rapid adoption of mobile messaging Apps has enabled us to collect massive amount of encrypted Internet traffic of mobile messaging. The classification of this traffic into different types of in-App service usages can help for intelligent network management, such as managing network bandwidth budget and providing quality of services. Traditional approaches for classification of Internet traffic rely on packet inspection, such as parsing HTTP headers. However, messaging Apps are increasingly using secure protocols, such as HTTPS and SSL, to transmit data. This imposes significant challenges on the performances of service usage classification by packet inspection. To this end, in this paper, we investigate how to exploit encrypted Internet traffic for classifying in-App usages. Specifically, we develop a system, named CUMMA, for classifying service usages of mobile messaging Apps by jointly modeling user behavioral patterns, network traffic characteristics, and temporal dependencies. Along this line, we first segment Internet traffic from traffic-flows into sessions with a number of dialogs in a hierarchical way. Also, we extract the discriminative features of traffic data from two perspectives: (i) packet length and (ii) time delay. Next, we learn a service usage predictor to classify these segmented dialogs into single-type usages or outliers. In addition, we design a clustering Hidden Markov Model (HMM) based method to detect mixed dialogs from outliers and decompose mixed dialogs into sub- dialogs of single-type usage. Indeed, CUMMA enables mobile analysts to identify service usages and analyze end-user in-App behaviors even for encrypted Internet traffic. Finally, the extensive experiments on real-world messaging data demonstrate the effectiveness and efficiency of the proposed method for service usage classification.

Reducing Latency Through Page-aware Management of Web Objects by Content Delivery Networks

As popular web sites turn to content delivery networks (CDNs) for full-site delivery, there is an opportunity to improve the end-user experience by optimizing the delivery of entire web pages, rather than just individual objects. In particular, this paper explores page-structure-aware strategies for placing objects in CDN cache hierarchies. The key idea is that the objects in a web page that have the largest impact on page latency should be served out of the closest or fastest caches in the hierarchy. We present schemes for identifying these objects and develop mechanisms to ensure that they are served with higher priority by the CDN, while balancing traditional CDN concerns such as optimizing the delivery of popular objects and minimizing bandwidth costs. To establish a baseline for evaluating improvements in page latencies, we collect and analyze publicly visible HTTP headers that reveal the distribution of objects among the various levels of a major CDN's cache hierarchy. Through extensive experiments on 83 real-world web pages, we show that latency reductions of over 100 ms can be obtained for 30% of the popular pages, with even larger reductions for the less popular pages. Using anonymized server logs provided by the CDN, we show the feasibility of reducing capacity and staleness misses of critical objects by 60% with minimal increase in overall miss rates, and bandwidth overheads of under 0.02%.

HTTPS traffic analysis and client identification using passive SSL/TLS fingerprinting

The encryption of network traffic complicates legitimate network monitoring, traffic analysis, and network forensics. In this paper, we present real-time lightweight identification of HTTPS clients based on network monitoring and SSL/TLS fingerprinting. Our experiment shows that it is possible to estimate the User-Agent of a client in HTTPS communication via the analysis of the SSL/TLS handshake. The fingerprints of SSL/TLS handshakes, including a list of supported cipher suites, differ among clients and correlate to User-Agent values from a HTTP header. We built up a dictionary of SSL/TLS cipher suite lists and HTTP User-Agents and assigned the User-Agents to the observed SSL/TLS connections to identify communicating clients. The dictionary was used to classify live HTTPS network traffic. We were able to retrieve client types from 95.4 % of HTTPS network traffic. Further, we discussed host-based and network-based methods of dictionary retrieval and estimated the quality of the data.

Web Tracking Detection System (TDS): An Effective Strategy to Reduce Systematic Monitoring and Profiling of User Habits Across Websites

This thesis is about tracking online user behavior. Tracking is a granular problem, that goes beyond the phenomenon of behavioral advertising. It takes place in the context of rapid technological, ocial and political developments. The main contributions of this thesis are:(1) A working definition for web tracking.(2) An opt-out mechanism for web tracking based on regular expressions repository in combinationwith persistent opt-out cookies.(3) We show that the current do-not-track-me register, an opt-out mechanism for online behavioraladvertising (OBA) based on opt-out cookies, of IAB/EASA falls short on it’s promise. The opt-out cookie expiration dates are inconsistent, often less then five years. This is in contrast with the NAIopt-out tool, where all opt-out cookies have a minimum expiration data of five years. Added tothat, opt-out behavior is collected by third parties.(4) We show that the interconnectedness between nodes expressed in the number of links per node is an indicator (clustering coefficient) for web tracking.(5) We show that the number off iltered nodes with TDS as a percentage of the total number of nodes is an indicator for web tracking.(6) Five rules for constraint based graph mining of HTTP header information show that with use of a confirmed web tracking repository, it is possible to identify new third party tracking activity with connectivity color maps and pattern matching.

Detecting malicious activities with user‐agent‐based profiles

SummaryHypertext transfer protocol (HTTP) has become the main protocol to carry out malicious activities. Attackers typically use HTTP for communication with command‐and‐control servers, click fraud, phishing and other malicious activities, as they can easily hide among the large amount of benign HTTP traffic. The user‐agent (UA) field in the HTTP header carries information on the application, operating system (OS), device, and so on, and adversaries fake UA strings as a way to evade detection. Motivated by this, we propose a novel grammar‐guided UA string classification method in HTTP flows. We leverage the fact that a number of ‘standard’ applications, such as web browsers and iOS mobile apps, have well‐defined syntaxes that can be specified using context‐free grammars, and we extract OS, device and other relevant information from them. We develop association heuristics to classify UA strings that are generated by ‘non‐standard’ applications that do not contain OS or device information. We provide a proof‐of‐concept system that demonstrates how our approach can be used to identify malicious applications that generate fake UA strings to engage in fraudulent activities. Copyright © 2015 John Wiley & Sons, Ltd.

A Perspective of Evolution After Five Years: A Large-Scale Study of Web Spam Evolution

Identifying and detecting web spam is an ongoing battle between spam-researchers and spammers which has been going on since search engines allowed searching of web pages to the modern sharing of web links via social networks. A common challenge faced by spam-researchers is the fact that new techniques depend on requiring a corpus of legitimate and spam web pages. Although large corpora of legitimate web pages are available to researchers, the same cannot be said about web spam or spam web pages. In this paper, we introduce the Webb Spam Corpus 2011 — a corpus of approximately 330,000 spam web pages — which we make available to researchers in the fight against spam. By having a standard corpus available, researchers can collaborate better on developing and reporting results of spam filtering techniques. The corpus contains web pages crawled from links found in over 6.3 million spam emails. We analyze multiple aspects of this corpus including redirection, HTTP headers, web page content, and classification evaluation. We also provide insights into changes in web spam since the last Webb Spam Corpus was released in 2006. These insights include: (1) spammers manipulate social media in spreading spam; (2) HTTP headers and content also change over time; (3) spammers have evolved and adopted new techniques to avoid the detection based on HTTP header information.

Geocookie: A Space-Efficient Representation of Geographic Location Sets

A new approach is presented that allows a user agent to compute a compact browser cookie-like character string (called Geocookie) for a third party, whereby for any given geographic location the third party can either infer that the location is definitely not covered by the Geocookie, or it can infer that the location is probably covered by the Geocookie, depending on whether the user agent has or has not included the location beforehand. The approach extends the concept of a Bloom filter and combines it with Geohashes, thus making it possible to store information about visited geographic locations in the filter. Geocookies can be used in many different scenarios for location-based queries and location-based services, whenever a user agent wants to inform a third party about a set of visited locations such that the third party can compile a result that either favors or excludes these locations. In contrast to existing approaches such as session cookies that are mapped to server-sided stored location trajectories, Geocookies provide a compact and privacy-preserving structure which does not reveal the actual set of all visited locations, but provides a one-way check function which can be used by the third party to evaluate given locations against the Geocookie. In addition, Geocookies provide plausible deniability in case of location matches. This paper introduces a formal definition for Geocookies together with a discussion on practical applications and embedding into HTTP headers.

An Empirical study of HTML5 Websockets and their Cross Browser behavior for Mixed Content and Untrusted Certificates

allow a full duplex connection to be made over a single socket between the client and the server. Today, Websockets is a finished standard and has greatly helped modern web applications to achieve real time communication without any overhead of sending HTTP headers with every request. This research provides an overview of the Websocket protocol and API, and focuses on the state of Websocket security. The research also aims to explicate behavior of different browser implementations of Websockets when delivering mixed content (ws/https) and the browser response when an untrusted certificate is encountered while making a secure Websocket connection. The crux of this paper is to analyze at the grassroots security concerns pertaining to Websockets and discuss best practices for secure deployment. KeywordsHTTP, Mixed Content, Security, Websockets.

Behavioral classification and detection of malware through HTTP user agent anomalies

A high proportion of modern botnets uses the HTTP protocol to communicate with its command servers and to perform a wide range of malicious activities. Nonetheless, detection of HTTP botnets is still a real challenge. Botmasters currently implement multiple techniques to hide their activity within the large amount of network traffic. On the other hand, although malware HTTP headers include multiple anomalies, little are being accounted for during detection. This paper analyzes anomalies in the HTTP user agent header field within malware traffic. It presents a taxonomy of malware user agent anomalies and uses this taxonomy in order to propose an appropriate detection mechanism.We observe, within a large set of malware HTTP traffic, that almost one malware out of eight uses a suspicious user agent header in at least one HTTP request. User agent anomalies are still being manually analyzed, whereas thousands of new malware samples are collected daily. This paper shows that a deeper analysis of malware user agents can reveal valuable detection patterns. It uses these patterns to automatically classify user agent anomalies and to extract signatures for malware detection. Our experimental results show that this solution provides a new mechanism that detects yet unknown malware by the time of building the signatures, while also satisfying a very low false positives rate.

Propagation of unintentionally shared information and online tracking

Various pieces of information are being shared online while users browse the Internet. Previous studies have demonstrated that as social networking sites (SNS) became popular, the information being leaked or shared is becoming more personal (including names and e-mail addresses). Users’ information is being shared or leaked from visited sites (both SNS and non-SNS) to third party sites (such as advertisers) in a number of ways including via the HTTP header. The intent of this study is identify the privacy implications of browsing the Internet within a single browsing session of a group of commonly visited sites (both SNS and non-SNS) doing activities common among most online users. We analysed the HTTP headers resulting from the first author’s browsing and reported on the types of information being shared or leaked, and to whom. We observed that within just a single browsing session of some sites, both the user’s identifiable and non-identifiable information are being leaked to various third party sites and also propagated to more than just one level of third party site. We also found that some SNS are also able to track user’s browsing activities not only within the SNS but also beyond it -particularly among web sites that use SNS widgets.

Novel MIME Type and Extension Based Packet Classification Algorithm in WiMAX

IEEE 802.16 provides quality of service by providing five different service classes. When a packet reaches the MAC layer, the packet classifier has to classify the packet such that the packet is associated with appropriate QoS. In this paper a packet classification algorithm is proposed that exploits the HTTP header of the application layer to determine the type of data and classify the data. The extension and MIME type (content type) headers are used to classify the packets. Further the algorithm is enhanced by considering the type of user as an additional parameter during packet classification. Simulations are done on variable bit rate video data. Based on the type of video the packets are classified as RTPS or nRTPS thereby providing differentiated QoS. Simulation results reveal that high priority video classified as RTPS traffic receive higher throughput compared to video of lower priority. The delay faced by high priority videos is correspondingly less compared to low priority video.

Research on Construction Methods for Network Covert Channels Based on HTTP

Network covert channel is a steganography technology that uses network traffic data as a carrier to transfer the secret data. This paper analyzes the working principle of network covert channels, and summarizes the commonly used construction technology of network covert channels. Then it analyzes the characteristics of the HTTP header lines and gives some methods of constructing network covert channels with these characteristics. Network covert channels based on the HTTP header lines are simple, flexible, and difficult to be detected and shielded.

Breaking Tor anonymity with game theory and data mining

SUMMARYAttacking anonymous communication networks is very tempting, and many types of attacks have already been observed. In the case for Tor, a widely used anonymous overlay network is considered. Despite the deployment of several protection mechanisms, an attack originated by just one rogue exit node is proposed. The attack is composed of two elements. The first is an active tag injection scheme. The malicious exit node injects image tags into all HTTP replies, which will be cached for upcoming requests and allow different users to be distinguished. The second element is an inference attack that leverages a semi‐supervised learning algorithm to reconstruct browsing sessions. Captured traffic flows are clustered into sessions, such that one session is most probably associated to a specific user. The clustering algorithm uses HTTP headers and logical dependencies encountered in a browsing session. A prototype has been implemented and its performance evaluated on the Tor network. The article also describes several countermeasures and advanced attacks, modeled in a game theoretical framework, and their effectiveness assessed with reference to the Nash equilibrium. Copyright © 2011 John Wiley & Sons, Ltd.

Developing RESTful Web Services with Webmachine

Many Web frameworks focus on projecting programming language entities out onto the Web handling HTTP header parsing and the protocol itself but leaving it up to developers to make sure their classes and objects obey the rules of REST and HTTP RFC 2616. Webmachine is a unique Erlang framework that codifies HTTP's rules and decision flows, letting resource developers focus mostly on resource implementations and state representations. Webmachine also provides defaults for what it requires of resource implementations, enabling developers to implement only the specific framework callbacks required to realize their particular resources.

Kernel-based control of persistent web server connections

Several overload admission control architectures have been developed to protect web servers from overload. Some of these architectures base their admission decision on information found in the HTTP header. In this context, persistent connections represent a challenging problem since the HTTP header of the first request does not reveal any information about the resource consumption of the requests that might follow on the same connection. In this paper, we present an architecture that prevents uncontrollable server overload caused by persistent connections. We evaluate our approach by various experiments.

The visible Web browser

As an aid to the study of the World-Wide Web, we have developed a software application that allows a user to observe the messages passed between a Web browser and a Web server. The application is based on the Mozilla Web Browser, and displays the HTTP headers sent and received by the browser. The program could be used by students in courses studying the Web, by researchers interested in the behavior of Web servers, and by developers to debug Web-based applications.

WebExpress: a client/intercept based system for optimizing Web browsing in a wireless environment

This paper describes an application model and software technology that makes it possible to run World Wide Web applications in wide area wireless networks. Web technology in conjunction with today's mobile devices (e.g., laptops, notebooks, personal digital assistants) and the emerging wireless technologies (e.g., digital cellular, packet radio, CDPD) offer the potential for unprecedented access to data and applications by mobile workers. Yet, the limited bandwidth, high latency, high cost, and poor reliability of today's wireless wide-area networks greatly inhibits (to the point of infeasibility) supporting such applications over wireless networks. This paper presents the Client/Intercept computational model that makes it possible to run such distributed applications efficiently in wide area wireless networks. Furthermore, it presents WebExpress, a client/intercept based system for optimizing Web browsing, that reduces data volume and latency of wireless communications by intercepting the HTTP data stream and performing various optimizations including: file caching, forms differencing, protocol reduction, and the elimination of redundant HTTP header transmission. This paper describes these optimizations and presents some experimental results.