Collaboratory For The Study Of Earthquake Predictability Research Articles

Comparison between alarm-based and probability-based earthquake forecasting methods

SUMMARY In a recent work, we applied the every earthquake a precursor according to scale (EEPAS) probabilistic model to the pseudo-prospective forecasting of shallow earthquakes with magnitude $M\ 5.0$ in the Italian region. We compared the forecasting performance of EEPAS with that of the epidemic type aftershock sequences (ETAS) forecasting model, using the most recent consistency tests developed within the collaboratory for the study of earthquake predictability (CSEP). The application of such models for the forecasting of Italian target earthquakes seems to show peculiar characteristics for each of them. In particular, the ETAS model showed higher performance for short-term forecasting, in contrast, the EEPAS model showed higher forecasting performance for the medium/long-term. In this work, we compare the performance of EEPAS and ETAS models with that obtained by a deterministic model based on the occurrence of strong foreshocks (FORE model) using an alarm-based approach. We apply the two rate-based models (ETAS and EEPAS) estimating the best probability threshold above which we issue an alarm. The model parameters and probability thresholds for issuing the alarms are calibrated on a learning data set from 1990 to 2011 during which 27 target earthquakes have occurred within the analysis region. The pseudo-prospective forecasting performance is assessed on a validation data set from 2012 to 2021, which also comprises 27 target earthquakes. Tests to assess the forecasting capability demonstrate that, even if all models outperform a purely random method, which trivially forecast earthquake proportionally to the space–time occupied by alarms, the EEPAS model exhibits lower forecasting performance than ETAS and FORE models. In addition, the relative performance comparison of the three models demonstrates that the forecasting capability of the FORE model appears slightly better than ETAS, but the difference is not statistically significant as it remains within the uncertainty level. However, truly prospective tests are necessary to validate such results, ideally using new testing procedures allowing the analysis of alarm-based models, not yet available within the CSEP.

Are Regionally Calibrated Seismicity Models More Informative than Global Models? Insights from California, New Zealand, and Italy

Abstract Earthquake forecasting models express hypotheses about seismogenesis that underpin global and regional probabilistic seismic hazard assessments (PSHAs). An implicit assumption is that the comparatively higher spatiotemporal resolution datasets from which regional models are generated lead to more informative seismicity forecasts than global models, which are however calibrated on greater datasets of large earthquakes. Here, we prospectively assess the ability of the Global Earthquake Activity Rate (GEAR1) model and 19 time-independent regional models to forecast M 4.95+ seismicity in California, New Zealand, and Italy from 2014 through 2021, using metrics developed by the Collaboratory for the Study of Earthquake Predictability (CSEP). Our results show that regional models that adaptively smooth small earthquake locations perform best in California and Italy during the evaluation period; however, GEAR1, based on global seismicity and geodesy datasets, performs surprisingly well across all testing regions, ranking first in New Zealand, second in California, and third in Italy. Furthermore, the performance of the models is highly sensitive to spatial smoothing, and the optimal smoothing likely depends on the regional tectonic setting. Acknowledging the limited prospective test data, these results provide preliminary support for using GEAR1 as a global reference M 4.95+ seismicity model that could inform eight-year regional and global PSHAs.

Statistical power of spatial earthquake forecast tests

SUMMARYThe Collaboratory for the Study of Earthquake Predictability (CSEP) is an international effort to evaluate earthquake forecast models prospectively. In CSEP, one way to express earthquake forecasts is through a grid-based format: the expected number of earthquake occurrences within 0.1° × 0.1° spatial cells. The spatial distribution of seismicity is thereby evaluated using the Spatial test (S-test). The high-resolution grid combined with sparse and inhomogeneous earthquake distributions leads to a huge number of cells causing disparity in the number of cells, and the number of earthquakes to evaluate the forecasts, thereby affecting the statistical power of the S-test. In order to explore this issue, we conducted a global earthquake forecast experiment, in which we computed the power of the S-test to reject a spatially non-informative uniform forecast model. The S-test loses its power to reject the non-informative model when the spatial resolution is so high that every earthquake of the observed catalog tends to get a separate cell. Upon analysing the statistical power of the S-test, we found, as expected, that the statistical power of the S-test depends upon the number of earthquakes available for testing, e.g. with the conventional high-resolution grid for the global region, we would need more than 32 000 earthquakes in the observed catalog for powerful testing, which would require approximately 300 yr to record M ≥ 5.95. The other factor affecting the power is more interesting and new; it is related to the spatial grid representation of the forecast model. Aggregating forecasts on multi-resolution grids can significantly increase the statistical power of the S-test. Using the recently introduced Quadtree to generate data-based multi-resolution grids, we show that the S-test reaches its maximum power in this case already for as few as eight earthquakes in the test period. Thus, we recommend for future CSEP experiments the use of Quadtree-based multi-resolution grids, where available data determine the resolution.

Multi-Resolution Grids in Earthquake Forecasting: The Quadtree Approach

ABSTRACTThe Collaboratory for the Study of Earthquake Predictability (CSEP) is an international effort to evaluate probabilistic earthquake forecasting models. CSEP provides the cyberinfrastructure and testing methods needed to evaluate earthquake forecasts. The most common way to represent a probabilistic earthquake forecast involves specifying the average rate of earthquakes within discrete spatial cells, subdivided into magnitude bins. Typically, the spatial component uses a single-resolution Cartesian grid with spatial cell dimensions of 0.1° × 0.1° in latitude and longitude, leading to 6.48 million spatial cells for the global testing region. However, the quantity of data (e.g., number of earthquakes) available to generate and test a forecast model is usually several orders of magnitude less than the millions of spatial cells, leading to a huge disparity in the number of earthquakes and the number of cells in the grid. In this study, we propose the Quadtree to create multi-resolution grid, locally adjusted mirroring the available data for forecast generation and testing, thus providing a data-driven resolution of forecasts. The Quadtree is a hierarchical tree-based data structure used in combination with the Mercator projection to generate spatial grids. It is easy to implement and has numerous scientific and technological applications. To facilitate its application to end users, we integrated codes handling Quadtrees into pyCSEP, an open-source Python package containing tools for evaluating earthquake forecasts. Using a sample model, we demonstrate how forecast model generation can be improved significantly in terms of information gain if constrained on a multi-resolution grid instead of a high-resolution uniform grid. In addition, we demonstrate that multi-resolution Quadtree grids lead to reduced computational costs. Thus, we anitcipate that Quadtree grids will be useful for developing and evaluating earthquake forecasts.

PyCSEP: A Python Toolkit for Earthquake Forecast Developers

Abstract The Collaboratory for the Study of Earthquake Predictability (CSEP) is an open and global community whose mission is to accelerate earthquake predictability research through rigorous testing of probabilistic earthquake forecast models and prediction algorithms. pyCSEP supports this mission by providing open-source implementations of useful tools for evaluating earthquake forecasts. pyCSEP is a Python package that contains the following modules: (1) earthquake catalog access and processing, (2) representations of probabilistic earthquake forecasts, (3) statistical tests for evaluating earthquake forecasts, and (4) visualization routines and various other utilities. Most significantly, pyCSEP contains several statistical tests needed to evaluate earthquake forecasts, which can be forecasts expressed as expected earthquake rates in space–magnitude bins or specified as large sets of simulated catalogs (which includes candidate models for governmental operational earthquake forecasting). To showcase how pyCSEP can be used to evaluate earthquake forecasts, we have provided a reproducibility package that contains all the components required to re-create the figures published in this article. We recommend that interested readers work through the reproducibility package alongside this article. By providing useful tools to earthquake forecast modelers and facilitating an open-source software community, we hope to broaden the impact of the CSEP and further promote earthquake forecasting research.

Prospective evaluation of multiplicative hybrid earthquake forecasting models in California

SUMMARYThe Regional Earthquake Likelihood Models (RELM) experiment, conducted within the Collaboratory for the Study of Earthquake Predictability (CSEP), showed that the smoothed seismicity (HKJ) model by Helmstetter et al. was the most informative time-independent earthquake model in California during the 2006–2010 evaluation period. The diversity of competing forecast hypotheses and geophysical data sets used in RELM was suitable for combining multiple models that could provide more informative earthquake forecasts than HKJ. Thus, Rhoades et al. created multiplicative hybrid models that involve the HKJ model as a baseline and one or more conjugate models. In retrospective evaluations, some hybrid models showed significant information gains over the HKJ forecast. Here, we prospectively assess the predictive skills of 16 hybrids and 6 original RELM forecasts at a 0.05 significance level, using a suite of traditional and new CSEP tests that rely on a Poisson and a binary likelihood function. In addition, we include consistency test results at a Bonferroni-adjusted significance level of 0.025 to address the problem of multiple tests. Furthermore, we compare the performance of each forecast to that of HKJ. The evaluation data set contains 40 target events recorded within the CSEP California testing region from 2011 January 1 to 2020 December 31, including the 2016 Hawthorne earthquake swarm in southwestern Nevada and the 2019 Ridgecrest sequence. Consistency test results show that most forecasting models overestimate the number of earthquakes and struggle to explain the spatial distribution of epicenters, especially in the case of seismicity clusters. The binary likelihood function significantly reduces the sensitivity of spatial log-likelihood scores to clustering, however; most models still fail to adequately describe spatial earthquake patterns. Contrary to retrospective analyses, our prospective test results show that none of the models are significantly more informative than the HKJ benchmark forecast, which we interpret to be due to temporal instabilities in the fit that forms hybrids. These results suggest that smoothing high-resolution, small earthquake data remains a robust method for forecasting moderate-to-large earthquakes over a period of 5–15 yr in California.

An Operational Earthquake Forecasting Experiment for Israel: Preliminary Results

Operational Earthquake Forecasting (OEF) aims to deliver timely and reliable forecasts that may help to mitigate seismic risk during earthquake sequences. In this paper, we build the first OEF system for the State of Israel, and we evaluate its reliability. This first version of the OEF system is composed of one forecasting model, which is based on a stochastic clustering Epidemic Type Earthquake Sequence (ETES) model. For every day of the forecasting time period, January 1, 2016 - November 15, 2020, the OEF-Israel system produces a weekly forecast for target earthquakes with local magnitudes greater than 4.0 and 5.5 in the entire State of Israel. Specifically, it provides space-time-dependent seismic maps of the weekly probabilities, obtained by using a fixed set of the model’s parameters, which are estimated through the maximum likelihood technique based on a learning period of about 32 years (1983–2015). According to the guidance proposed by the Collaboratory for the Study of Earthquake Predictability (CSEP), we also perform the N- and S-statistical tests to verify the reliability of the forecasts. Results show that the OEF system forecasts a number of events comparable to the observed one, and also captures quite well the spatial distribution of the real catalog with the exception of two target events that occurred in low seismicity regions.

Pseudoprospective Evaluation of UCERF3-ETAS Forecasts during the 2019 Ridgecrest Sequence

ABSTRACTThe 2019 Ridgecrest sequence provides the first opportunity to evaluate Uniform California Earthquake Rupture Forecast v.3 with epidemic-type aftershock sequences (UCERF3-ETAS) in a pseudoprospective sense. For comparison, we include a version of the model without explicit faults more closely mimicking traditional ETAS models (UCERF3-NoFaults). We evaluate the forecasts with new metrics developed within the Collaboratory for the Study of Earthquake Predictability (CSEP). The metrics consider synthetic catalogs simulated by the models rather than synoptic probability maps, thereby relaxing the Poisson assumption of previous CSEP tests. Our approach compares statistics from the synthetic catalogs directly against observations, providing a flexible approach that can account for dependencies and uncertainties encoded in the models. We find that, to the first order, both UCERF3-ETAS and UCERF3-NoFaults approximately capture the spatiotemporal evolution of the Ridgecrest sequence, adding to the growing body of evidence that ETAS models can be informative forecasting tools. However, we also find that both models mildly overpredict the seismicity rate, on average, aggregated over the evaluation period. More severe testing indicates the overpredictions occur too often for observations to be statistically indistinguishable from the model. Magnitude tests indicate that the models do not include enough variability in forecasted magnitude-number distributions to match the data. Spatial tests highlight discrepancies between the forecasts and observations, but the greatest differences between the two models appear when aftershocks occur on modeled UCERF3-ETAS faults. Therefore, any predictability associated with embedding earthquake triggering on the (modeled) fault network may only crystalize during the presumably rare sequences with aftershocks on these faults. Accounting for uncertainty in the model parameters could improve test results during future experiments.

Forecasting the Full Distribution of Earthquake Numbers Is Fair, Robust, and Better

Forecasting the full distribution of the number of earthquakes is revealed to be inherently superior to forecasting their mean. Forecasting the full distribution of earthquake numbers is also shown to yield robust projections in the presence of surprise large earthquakes, which in the past have strongly deteriorated the scores of existing models. We show this with pseudo-prospective experiments on synthetic as well as real data from the Advanced National Seismic System (ANSS) database for California, with earthquakes with magnitude larger than 2.95 that occurred between the period 1971-2016. Our results call in question the testing methodology of the Collaboratory for the study of earthquake predictability (CSEP), which amounts to assuming a Poisson distribution of earthquake numbers, which is known to be a poor representation of the heavy-tailed distribution of earthquake numbers. Using a spatially varying ETAS model, we demonstrate a remarkable stability of the forecasting performance, when using the full distribution of earthquake numbers for the forecasts, even in the presence of large earthquakes such as Mw 7.1 Hector Mine, Mw 7.2 El Mayor-Cucapah, Mw 6.6 Sam Simeon earthquakes, or in the presence of intense swarm activity in Northwest Nevada in 2014. While our results have been derived for ETAS type models, we propose that all earthquake forecasting models of any type should embrace the full distribution of earthquake numbers, such that their true forecasting potential is revealed.

Highlights from the First Ten Years of the New Zealand Earthquake Forecast Testing Center

Research Article| June 13, 2018 Highlights from the First Ten Years of the New Zealand Earthquake Forecast Testing Center David A. Rhoades; David A. Rhoades aGNS Science, 1 Fairway Drive, Avalon, Lower Hutt 5010, New Zealand, d.rhoades@gns.cri.nz Search for other works by this author on: GSW Google Scholar Annemarie Christophersen; Annemarie Christophersen aGNS Science, 1 Fairway Drive, Avalon, Lower Hutt 5010, New Zealand, d.rhoades@gns.cri.nz Search for other works by this author on: GSW Google Scholar Matthew C. Gerstenberger; Matthew C. Gerstenberger aGNS Science, 1 Fairway Drive, Avalon, Lower Hutt 5010, New Zealand, d.rhoades@gns.cri.nz Search for other works by this author on: GSW Google Scholar Maria Liukis; Maria Liukis bJet Propulsion Laboratory, 4800 Oak Grove Drive, Pasadena, California 91109 U.S.A. Search for other works by this author on: GSW Google Scholar Fabio Silva; Fabio Silva cSouthern California Earthquake Center, University of Southern California, 3651 Trousdale Parkway, Los Angeles, California 90089‐0742 U.S.A. Search for other works by this author on: GSW Google Scholar Warner Marzocchi; Warner Marzocchi dIstituto Nazionale di Geofisica e Vulcanologia, Via di Vigna Murata 605, 00143 Rome, Italy Search for other works by this author on: GSW Google Scholar Maximilian J. Werner; Maximilian J. Werner eUniversity of Bristol, Tyndall Avenue, Bristol BS8 1TH, United Kingdom Search for other works by this author on: GSW Google Scholar Thomas H. Jordan Thomas H. Jordan cSouthern California Earthquake Center, University of Southern California, 3651 Trousdale Parkway, Los Angeles, California 90089‐0742 U.S.A. Search for other works by this author on: GSW Google Scholar Author and Article Information David A. Rhoades aGNS Science, 1 Fairway Drive, Avalon, Lower Hutt 5010, New Zealand, d.rhoades@gns.cri.nz Annemarie Christophersen aGNS Science, 1 Fairway Drive, Avalon, Lower Hutt 5010, New Zealand, d.rhoades@gns.cri.nz Matthew C. Gerstenberger aGNS Science, 1 Fairway Drive, Avalon, Lower Hutt 5010, New Zealand, d.rhoades@gns.cri.nz Maria Liukis bJet Propulsion Laboratory, 4800 Oak Grove Drive, Pasadena, California 91109 U.S.A. Fabio Silva cSouthern California Earthquake Center, University of Southern California, 3651 Trousdale Parkway, Los Angeles, California 90089‐0742 U.S.A. Warner Marzocchi dIstituto Nazionale di Geofisica e Vulcanologia, Via di Vigna Murata 605, 00143 Rome, Italy Maximilian J. Werner eUniversity of Bristol, Tyndall Avenue, Bristol BS8 1TH, United Kingdom Thomas H. Jordan cSouthern California Earthquake Center, University of Southern California, 3651 Trousdale Parkway, Los Angeles, California 90089‐0742 U.S.A. Publisher: Seismological Society of America First Online: 13 Jun 2018 Online Issn: 1938-2057 Print Issn: 0895-0695 © Seismological Society of America Seismological Research Letters (2018) 89 (4): 1229–1237. https://doi.org/10.1785/0220180032 Article history First Online: 13 Jun 2018 Cite View This Citation Add to Citation Manager Share Icon Share Facebook Twitter LinkedIn MailTo Tools Icon Tools Get Permissions Search Site Citation David A. Rhoades, Annemarie Christophersen, Matthew C. Gerstenberger, Maria Liukis, Fabio Silva, Warner Marzocchi, Maximilian J. Werner, Thomas H. Jordan; Highlights from the First Ten Years of the New Zealand Earthquake Forecast Testing Center. Seismological Research Letters 2018;; 89 (4): 1229–1237. doi: https://doi.org/10.1785/0220180032 Download citation file: Ris (Zotero) Refmanager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All ContentBy SocietySeismological Research Letters Search Advanced Search ABSTRACT We present highlights from the first decade of operation of the New Zealand Earthquake Forecast Testing Center of the Collaboratory for the Study of Earthquake Predictability (CSEP). Most results are based on reprocessing using the best available catalog, because the testing center did not consistently capture the complete real‐time catalog. Tests of models with daily updating show that aftershock models incorporating Omori–Utsu decay can outperform long‐term smoothed seismicity models with probability gains up to 1000 during major aftershock sequences. Tests of models with 3‐month updating show that several models with every earthquake a precursor according to scale (EEPAS) model, incorporating the precursory scale increase phenomenon and without Omori–Utsu decay, and the double‐branching model, with both Omori–Utsu and exponential decay in time, outperformed a regularly updated smoothed seismicity model. In tests of 5‐yr models over 10 yrs without updating, a smoothed seismicity model outperformed the earthquake source model of the New Zealand National Seismic Hazard Model. The performance of 3‐month and 5‐yr models was strongly affected by the Canterbury earthquake sequence, which occurred in a region of previously low seismicity. Smoothed seismicity models were shown to perform better with more frequent updating. CSEP models were a useful resource for the development of hybrid time‐varying models for practical forecasting after major earthquakes in the Canterbury and Kaikoura regions. You do not have access to this content, please speak to your institutional administrator if you feel you should have access.

Prospective Evaluation of Global Earthquake Forecast Models: 2 Yrs of Observations Provide Preliminary Support for Merging Smoothed Seismicity with Geodetic Strain Rates

Research Article| June 13, 2018 Prospective Evaluation of Global Earthquake Forecast Models: 2 Yrs of Observations Provide Preliminary Support for Merging Smoothed Seismicity with Geodetic Strain Rates Anne Strader; Anne Strader aGeoForschungsZentrum, Helmholtzstrasse 6/7, Building H 6, Room 207, 14467 Potsdam, Germany, strader@gfz-potsdam.de Search for other works by this author on: GSW Google Scholar Maximilian Werner; Maximilian Werner bUniversity of Bristol, Office L126, Wills Memorial Building, Queens Road, Clifton, BS8 1RJ Bristol, United Kingdom Search for other works by this author on: GSW Google Scholar José Bayona; José Bayona cGeoForschungsZentrum, Helmholtzstrasse 6/7, Building H 6, Room 102, 14467 Potsdam, Germany Search for other works by this author on: GSW Google Scholar Philip Maechling; Philip Maechling dUniversity of Southern California, 3651 Trousdale Parkway, Los Angeles, California 90089 U.S.A. Search for other works by this author on: GSW Google Scholar Fabio Silva; Fabio Silva eZumberge Hall of Science (ZHS), Room 169 C, Los Angeles, California 90089‐0740 U.S.A. Search for other works by this author on: GSW Google Scholar Maria Liukis; Maria Liukis fJet Propulsion Laboratory, NASA, Navigation and Ancillary Information Facility, 4800 Oak Grove Drive, Pasadena, California 91109 U.S.A. Search for other works by this author on: GSW Google Scholar Danijel Schorlemmer Danijel Schorlemmer gGeoForschungsZentrum, Helmholtzstrasse 6/7, Building H 6, Room 101, 14467 Potsdam, Germany Search for other works by this author on: GSW Google Scholar Seismological Research Letters (2018) 89 (4): 1262–1271. https://doi.org/10.1785/0220180051 Article history first online: 13 Jun 2018 Cite View This Citation Add to Citation Manager Share Icon Share Facebook Twitter LinkedIn MailTo Tools Icon Tools Get Permissions Search Site Citation Anne Strader, Maximilian Werner, José Bayona, Philip Maechling, Fabio Silva, Maria Liukis, Danijel Schorlemmer; Prospective Evaluation of Global Earthquake Forecast Models: 2 Yrs of Observations Provide Preliminary Support for Merging Smoothed Seismicity with Geodetic Strain Rates. Seismological Research Letters 2018;; 89 (4): 1262–1271. doi: https://doi.org/10.1785/0220180051 Download citation file: Ris (Zotero) Refmanager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All ContentBy SocietySeismological Research Letters Search Advanced Search ABSTRACT The global earthquake activity rate (GEAR1) seismicity model uses an optimized combination of geodetic strain rates, hypotheses about converting strain rates to seismicity rates from plate tectonics, and earthquake‐catalog data to estimate global Mw≥5.767 shallow (⁠≤70 km⁠) seismicity rates. It comprises two parent models: a strain rate‐based model and a smoothed‐seismicity based model. The GEAR1 model was retrospectively evaluated and calibrated using earthquake data from 2005 to 2012, resulting in a preferred log–linear multiplicative combination of the parent forecasts. Since 1 October 2015, the GEAR1 model has undergone prospective evaluation within the Collaboratory for the Study of Earthquake Predictability (CSEP) testing center, forecasting Mw≥5.95 seismicity.We present initial prospective forecast test results for the GEAR1 model, its tectonic and seismicity components, and for the first iteration of the strain‐rate‐based model, during the 1 October 2015–7 September 2017 period. During the evaluation period, observed earthquakes are consistent with the GEAR1 forecast and comparative test results likewise support that GEAR1 is more informative than either of its components alone. Based on a combination of retrospective and prospective testing, the tectonic forecasts do not effectively anticipate observed spatial earthquake distribution, largely due to overlocalization of the model with respect to observed earthquake distributions. You do not have access to this content, please speak to your institutional administrator if you feel you should have access.

Preface to the Focus Section on the Collaboratory for the Study of Earthquake Predictability (CSEP): New Results and Future Directions

Preface to the Focus Section on the Collaboratory for the Study of Earthquake Predictability (CSEP): New Results and Future Directions

Assessing ‘alarm-based CN’ earthquake predictions in Italy

<p>The quantitative assessment of the performance of earthquake prediction and/or forecast models is essential for evaluating their applicability for risk reduction purposes. Here we assess the earthquake prediction performance of the CN model applied to the Italian territory. This model has been widely publicized in Italian news media, but a careful assessment of its prediction performance is still lacking. In this paper we evaluate the results obtained so far from the CN algorithm applied to the Italian territory, by adopting widely used testing procedures and under development in the Collaboratory for the Study of Earthquake Predictability (CSEP) network. Our results show that the CN prediction performance is comparable to the prediction performance of the stationary Poisson model, that is, CN predictions do not add more to what may be expected from random chance.</p>

Voronoi residuals and other residual analyses applied to CSEP earthquake forecasts

Voronoi residuals, deviances, super-thinning, and some other residual analysis methods are applied to a selection of earthquake forecast models in the Collaboratory for the Study of Earthquake Predictability (CSEP). Unlike simple numerical summaries such as the N-test, L-test, or R-test, graphical residual methods are proposed which can be useful for comparing multiple models and for highlighting when and where a given model does not agree closely with the observed seismicity. For gridded forecasts, deviances seem ideally suited for model comparison, and Voronoi residuals seem preferable to simple grid-based residuals for assessing one model individually. For models outputting an estimated conditional rate at any particular space–time location, Voronoi residuals and super-thinning can be especially useful at identifying departures from the data.

Likelihood- and residual-based evaluation of medium-term earthquake forecast models for California

Seven competing models for forecasting medium-term earthquake rates in California are quantitatively evaluated using the framework of the Collaboratory for the Study of Earthquake Predictability (CSEP). The model class consists of contrasting versions of the Every Earthquake a Precursor According to Size (EEPAS) and Proximity to Past Earthquakes (PPE) modelling approaches. Models are ranked by their performance on likelihood-based tests, which measure the consistency between a model forecast and observed earthquakes. To directly compare one model against another, we run a classical paired t-test and its non-parametric alternative on an information gain score based on the forecasts. These test scores are complemented by several residual-based methods, which offer detailed spatial information. The experiment period covers 2009 June–2012 September, when California experienced 23 earthquakes above the magnitude threshold. Though all models fail to capture seismicity during an earthquake sequence, spatio-temporal differences between models also emerge. The overall best-performing model has strong time- and magnitude-dependence, weights all earthquakes equally as medium-term precursors of larger events and has a full set of fitted parameters. Models with this time- and magnitude-dependence offer a statistically significant advantage over simpler baseline models. In addition, models that down-weight aftershocks when forecasting larger events have a desirable feature in that they do not overpredict following an observed earthquake sequence. This tendency towards overprediction differs between the simpler model, which is based on fewer parameters, and more complex models that include more parameters.

Some reasoning on the RELM-CSEP likelihood-based tests

The null hypothesis is the essence of any statistical test: this is basically a comparison of what we observe with what we would expect to see if the null hypothesis was true. In this work, I explore the suitability of the null hypothesis of likelihood-based tests (LBTs), which are often adopted by the laboratories of the Collaboratory for the Study of Earthquake Predictability (CSEP), to check earthquake forecast models. First, I discuss the LBT in the wider context of classical statistical hypothesis testing. Then, I present some cases in which the null hypothesis of LBT is not appropriate for determining the merits of earthquake forecast models. I justify these results from a theoretical point of view, within the framework of point process theory. Finally, I propose a possible upgrade of LBT to enable the correct assessment of the forecasting capability of earthquake models. This study may provide new insights to the CSEP LBT.

Assessing annual global M6+ seismicity forecasts

SUMMARY We consider a seismicity forecast experiment conducted during the last 4 yr. At the beginning of each year, three models make a 1-yr forecast of the distribution of large earthquakes everywhere on the Earth. The forecasts are generated and the observations are collected in the Collaboratory for the Study of Earthquake Predictability (CSEP). We apply CSEP likelihood measures of consistency and comparison to see how well the forecasts match the observations, and we compare results from some intuitive reference models. These results illustrate some undesirable properties of the consistency tests: the tests can be extremely sensitive to only a

Comprehensive and Topical Evaluations of Earthquake Forecasts in Terms of Number, Time, Space, and Magnitude

Among scoring methods employed to determine the performance of probability predictions, the log‐likelihood method is the most common and useful. Although the log‐likelihood score evaluates the comprehensive power of forecasts, we need to further evaluate the topical predictive powers of respective factors of seismicity, such as total numbers, occurrence times, locations, and magnitudes. For this purpose, we used the conditional‐ or marginal‐likelihood function based on the observed events. Such topical scores reveal both strengths and weaknesses of a forecasting model and suggest the necessary improvements. We applied these scores to the probability forecasts during the devastating period of March 2011, during which the M w 9.0 Off the Pacific Coast of Tohoku‐Oki earthquake struck. However, the evaluations did not suggest that any of the prospective forecast models were consistently satisfactory. Hence, we undertook two additional types of retrospective forecasting experiments to investigate the reasons, including the possibility that the seismicity rate pattern has changed after the M 9 mega‐earthquake. In addition, our experiments revealed a technical difficulty in the one‐day forecasting protocol adopted by the Collaboratory for the Study of Earthquake Predictability (CSEP). Results of further experiments lead us to recommend specific modifications to the CSEP protocols, leading to real‐time forecasts and their evaluations.

Regional Earthquake Likelihood Models I: First-Order Results

Abstract The Regional Earthquake Likelihood Models (RELM) working group designed a 5-year experiment to forecast the number, spatial distribution, and magnitude distribution of subsequent target earthquakes, defined to be those with magnitude ≥4.95 (M 4.95+) in a well-defined California testing region. Included in the experiment specification were the description of the data source, the methods for data processing, and the proposed evaluation metrics. The RELM experiment began on 1 January 2006 and involved 17 time-invariant forecasts constructed by seismicity modelers; by the end of the experiment on 1 January 2011, 31 target earthquakes had occurred. We analyze the experiment outcome by applying the proposed consistency tests based on likelihood measures and additional comparison tests based on a measure of information gain. We find that the smoothed seismicity forecast by Helmstetter et al., 2007 based on M 2+ earthquakes since 1981, is the best forecast, regardless of whether aftershocks are included in the analysis. The RELM experiment has helped to clarify ideas about testing that can be applied to more wide-ranging earthquake forecasting experiments conducted by the Collaboratory for the Study of Earthquake Predictability (CSEP). Online Material: Figures and tables showing the RELM testing region and collection region definitions, numerical results associated with the RELM experiment, and the uncorrected forecast by Ebel et al. (2007).

Completeness of the Mainland China Earthquake Catalog and Implications for the Setup of the China Earthquake Forecast Testing Center

We describe the setup of testing regions for the China Earthquake Forecast Testing Center and provide preliminary forecast results in the scope of the Collaboratory for the Study of Earthquake Predictability (CSEP) project. We investigate the spatiotemporal variations of the completeness magnitude M c by using the frequency‐magnitude distribution of the China Earthquake Networks Center (CENC) catalog. We find three periods of significantly different M c histories: (I) 1 January 1970–30 September 2001, (II) 1 October 2001–30 September 2008, and (III) 1 October 2008–31 August 2011. M c mapping provides median values ![Graphic][1] , 2.2, and 1.6 for the three periods of time, respectively, showing the improvement in catalog completeness over time. We recommend using data from periods II and III to define a baseline long enough for retrospective forecast testing. Small magnitude events from period I should be used with caution due to important fluctuations in completeness. For period III, coordinates of all national and regional seismic stations are available, and we therefore apply the Bayesian magnitude of completeness (BMC) technique, mapping M c continuously by using prior information on the relationship between M c and the density of seismic stations. We define four potential testing/collection areas for CSEP‐China: (A) All China, (B) North–South Seismic Belt (NSSB), (C) North and West Xinjiang Seismic Region, and (D) North China Seismic Region. In the current phase of CSEP‐China, only the NSSB (region B) is considered. To demonstrate the type of earthquake predictability experiment that will be performed in the Chinese Testing Center, we present a series of retrospective forecast experiments with TripleS, a smoothed seismicity model. Online Material: The CENC earthquake catalog (1 January 1970–31 August 2011, restricted to magnitudes M ≥3.0) as well as completeness magnitude M c( x , y ) spatial grids. [1]: /embed/inline-graphic-1.gif