Wednesday, June 15, 2016

p-Curve: Replicable vs. Non-Replicable Findings

Recently, Stephan Bruns published a paper with John Ioannidis in PLoS ONE critiquing the p-curve.  I've blogged about the p-curve previously. Their argument is that the p-curve cannot distinguish "true effects" from "null effects" in the presence of omitted variables bias. Simonsohn et al., the originators of the p-curve, have responded in their blog, which I have added to the blogroll here. They say, of course, the p-curve cannot distinguish between causal effects and other effects but it can distinguish between "false positives", which are non-replicable effects and "replicable effects", which include both "confounded effects" (correlation but not causation) and "causal effects". Bruns and Ioannidis have responded to this comment too.

In my previous blogpost on the p-curve, I showed that the Granger causality tests we meta-analysed in our Energy Journal paper in 2014 form a right-skewed p-curve. This would mean that there was a "true effect" according to the p-curve methodology. However, our meta-regression analysis where we regressed the test statistics on the square root of degrees of freedom in the underlying regressions showed no "genuine effect". Now I understand what is going on. The large number of highly significant results in the Granger causality meta-dataset is generated by "overfitting bias". This result is "replicable". If we fit VAR models to more such short time series we will again get large numbers of significant results. However, regression analysis shows that this result is bogus as the p-values are not negatively correlated with degrees of freedom. Therefore, the power trace meta-regression is a superior method to the p-curve. In addition, we can modify this regression model to account for omitted variables bias by adding dummy variables and interaction terms (as we do in our paper). This can help to identify a causal effect. Of course, if no researchers actually estimate the true causal model then this method too cannot identify the causal effect. But there are always limits to our ability to be sure of causality. Meta-regression can help rule out some cases of confounded effects.

So, to sum up there are the following dichotomies:
  • Replicable vs. non-replicable - can use p-curve.
  • True or genuine effect (a correlation in the data-generating process) vs. false positive - metaregression model is more likely to give correct inference.*
  • Causal vs. confounded effect - extended meta-regression model can rule out some confounded effects.
The bottom line is that you should use meta-regression analysis rather than the p-curve.

* In the case of unit root spurious regressions mentioned in Bruns and Ioannidis' response, things are a bit complicated. In the case of a bivariate spurious regression, where there is a drift in the same direction in both variables then it is likely that Stanley's FAT-PET and similar methods will show that there is a true effect. Even though there is no relationship at all between the two variables, the nature of the data-generating-process for each means that they will be correlated. Where there is no drift or the direction of drift varies randomly then there should be equal numbers of positive and negative t-statistics in underlying studies and no relationship between the value of the t-statistic and degrees of freedom, though there is a relationship between the absolute value of the t-statistic and degrees of freedom. Here meta-regression does better than the p-curve. I'm not sure if the meta-regression model in our Energy Journal paper might be fooled by Granger Causality tests in levels of unrelated unit root variables. These would likely be spuriously significant but the significance might not rise strongly with sample size?

Wednesday, June 1, 2016

Mid-Year Update

It's the first official day of winter today here in Australia, though it has felt wintry here in Canberra for about a week already. The 1st Semester finished last Friday and as I didn't teach I don't have any exams or papers to grade and the flow of admin stuff and meetings seems to have sharply declined. So, most of this week I can just dedicate to catching up and getting on with my research. It almost feels like I am on vacation :) Looking at my diary, the pace will begin to pick up again from next week.

I'm working on two main things this week. One is the Energy for Economic Growth Project that has now been funded by the UK Department for International Development. I mentioned our brainstorming meeting last July in Oxford in my 2015 Annual Report. I am the theme leader for Theme 1 in the first year of the project. In the middle of this month we have a virtual workshop for the theme to discuss the outlines for our proposed papers. I am coauthoring a survey paper with Paul Burke and Stephan Bruns on the macro-economic evidence as part of Theme 1. There are two other papers in the theme: one by Catherine Wolfram and Ted Miguel on the micro-economic evidence and one by Neil McCulloch on the binding constraints approach to the problem.

The other is my paper with Jack Pezzey on the Industrial Revolution, which we have presented at various conferences and seminars over the last couple of years. I'm ploughing through the math and tidying the presentation up. It's slow going but I think I can see the light at the end of the tunnel! This paper was supposed to be a key element in the ARC Discovery Projects grant that started in 2012.

In the meantime, work has started on our 2016 Discovery Projects grant. Zsuzsanna Csereklyei has now started work at Crawford as a research fellow funded by the grant. She has been scoping the potential sources of data for tracing the diffusion of energy efficient innovations and processing the first potential data source that we have identified. It is hard to find good data sources that are usable for our purpose.

There is a lot of change in the air at ANU as we have a new vice-chancellor on board since the beginning of the year and now a new director for the Crawford School has been appointed and will start later this year. We are also working out again how the various economics units at ANU relate to each other... I originally agreed to be director of the Crawford economic program for a year. That will certainly continue now to the end of this year. It's not clear whether I'll need to continue in the role longer than that.

Finally, here is a list of all papers published so far this year or now in press. I can't remember how many of them I mentioned on the blog, though I probably mentioned all on Twitter:

Bruns S. B. and D. I. Stern (in press) Research assessment using early citation information, Scientometrics. Working Paper Version | Blogpost

Stern D. I. and D. Zha (in press) Economic growth and particulate pollution concentrations in China, Environmental Economics and Policy Studies. Working Paper Version | Blogpost
Lu Y. and D. I. Stern (2016) Substitutability and the cost of climate mitigation policy, Environmental and Resource Economics. Working Paper Version | Blogpost

Sanchez L. F. and D. I. Stern (2016) Drivers of industrial and non-industrial greenhouse gas emissions, Ecological Economics 124, 17-24. Working Paper Version | Blogpost 1 | Blogpost 2

Costanza R., R. B. Howarth, I. Kubiszewski, S. Liu, C. Ma, G. Plumecocq, and D. I. Stern (2016) Influential publications in ecological economics revisited, Ecological Economics. Working Paper Version | Blogpost

Csereklyei Z., M. d. M. Rubio Varas, and D. I. Stern (2016) Energy and economic growth: The stylized facts, Energy Journal 37(2), 223-255. Working Paper Version | Blogpost

Halkos G. E., D. I. Stern, and N. G. Tzeremes (2016) Population, economic growth and regional environmental inefficiency: Evidence from U.S. states, Journal of Cleaner Production 112(5), 4288-4295. Blogpost