Proceedings of the 1st international workshop on System evaluation for mobile platforms

Abstract

A core aspect of any engineering systems research is evaluation: What are the metrics and methods to determine if the system is behaving as expected. In mobile systems, this task becomes more challenging due to considerations imposed by the mobile aspect of the system. This workshop aims to bring together systems researchers to explore the basic question of evaluating mobile systems. Specifically we will examine challenges and advances made in the metrics, tools, methods, and the platforms used to evaluate mobile systems effectively. The papers presented in the workshop highlight the specific areas relating to mobile systems that make proper evaluation difficult, and include discussions about how novel techniques can be used to address the various issues that arise. The focus of the workshop is not necessarily the underlying research for mobile systems, but rather the techniques and considerations necessary for evaluating and quantifying the gains of the basic research itself. Some example areas where mobile platforms and systems evaluation issues intersect include: Reproducibility and Consistency: Given the dynamic nature of mobile systems, how can one provide reproducible and consistent results usable by other researchers?Experimental Design: How can mobile experiments be designed to isolate the desired variables? How can the effects of unwanted factors be mitigated?Power Considerations: What techniques are appropriate for measuring the power consumption of mobile systems? Are these techniques accurate, and necessary?Metrics: What are the various metrics that should be used for evaluation and comparison?E.g. Power vs. energy consumption, or latency vs. bandwidth.Applications: What are appropriate application benchmarks for mobile systems? How can the needs of user evaluations be balanced with the system evaluation perspective?Dynamic environments: How can the dynamic nature of mobile environments be handled? Mobile systems move, increasing the number of variables in the system.Form factor: Since it is often desirable for mobile systems to be small, how can researchers extend a mobile platform while still maintaining a useable form factor?Platforms/Test-beds: How can a mobile system test-beds be used to make evaluations easier and more accurate?Simulation: What simulation techniques are appropriate for evaluating mobile systems? How can a simulation capture the unique aspects of mobility?.