Percentage of sample above a value

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Percentage of sample above a value

Wayne
In figure 8.8 of DBDA, you have a histogram of the differences of two thetas, which mentions 10.6% < 0 < 89.4%. In the text, you say that "a difference of zero is among the credible differences" but you don't expand on it and I could see interpreting the histogram in two ways:

1. In a Frequentist-like failure-to-reject manner, which I believe is implied by the commentary, that a difference of zero is credible with 95% confidence and therefore we cannot state with confidence that theta1 is different from theta2.

2. Saying that 89.4% of the differences are greater than zero and therefore -- allowing for a small ROPE around zero -- the odds are roughly 8-1 that theta1 is greater than theta2. Which is not the 19-1 that 95% implies, but is pretty strong in every-day terms.

Is #2 a reasonable interpretation? Is it meaningful? (A 95% CI is somewhat arbitrary, but widely used outside of things like physics, I believe.)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Percentage of sample above a value

John K. Kruschke
Administrator

On Mon, Jul 8, 2013 at 8:35 PM, Wayne [via Doing Bayesian Data Analysis] <[hidden email]> wrote:
In figure 8.8 of DBDA, you have a histogram of the differences of two thetas, which mentions 10.6% < 0 < 89.4%. In the text, you say that "a difference of zero is among the credible differences" but you don't expand on it and I could see interpreting the histogram in two ways:

1. In a Frequentist-like failure-to-reject manner, which I believe is implied by the commentary, that a difference of zero is credible with 95% confidence and therefore we cannot state with confidence that theta1 is different from theta2.

Right, I advocate a decision rule that uses the HDI (and ROPE). In this case, the difference of zero falls within the 95% HDI, so cannot be "rejected." The 95% HDI is not very precise, however, and does not fall within a reasonable ROPE around zero, and therefore zero also cannot be accepted for practical purposes.

 

2. Saying that 89.4% of the differences are greater than zero and therefore -- allowing for a small ROPE around zero -- the odds are roughly 8-1 that theta1 is greater than theta2. Which is not the 19-1 that 95% implies, but is pretty strong in every-day terms.

Is #2 a reasonable interpretation? Is it meaningful? (A 95% CI is somewhat arbitrary, but widely used outside of things like physics, I believe.)

I am not an advocate of decision rule #2 because it does not reflect the distributional shape or actual density at zero. The distribution could have 89% of its mass greater than zero in many different ways; e.g., all piled up very close to zero or far away from zero. Here's another case to ponder: Consider a bimodal posterior, with 50% on each side of zero, but a deep valley at zero such that the density near zero is very small. The HDI has two subintervals, and the HDI does not include zero. But there's a 50-50 split of the posterior on each side of zero. Intuitively (for me), zero should be rejected.

It's important to keep in mind that the Bayesian part of this is the complete posterior distribution. What we are talking about here is going beyond the purely Bayesian part, namely, what sort of decision rule to use for converting a Bayesian posterior distribution to a discrete decision about accepting/rejecting a null value. It's not that any candidate decision rule is inherently correct or wrong; it's just more or less interpretable and useful in different contexts.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Percentage of sample above a value

Wayne
That helps a lot. So the answer is that we cannot say whether theta1 and theta2 are different or not with 95% confidence -- there's too much variability in the data to make a decision one way or the other? Which still leaves me with a bit of unease, since it looks fairly obvious to me that there's more evidence that theta1 is greater than theta2 than not. It feels like there should be some way of quantifying and verbalizing what the histogram suggests (to me).

I'd like to be able to say something like "there's mild evidence that theta1 is greater than theta2, though it isn't conclusive." Or "there's a 65% chance that theta1 is greater than theta2." Or "it's more likely that theta1 is greater than theta2 than not, but only slightly larger." Or something like that.

A classical sin in Frequentist statistics would be to lower the confidence interval until we got significance at some level. Perhaps the difference is insignificant at the 95% level, but not at the 50% level. Is that unique to Frequentist statistics and p-values, or would a similar maneuver here be allowable?

That is, can we look at the 85% HDI, the 75% HDI, etc, until we find an HDI that is entirely positive, and then say at that level of confidence that theta1 is likely greater than theta2? I realize that this is similar to my original decision rule #2, since it doesn't take into account the shape of the posterior density, so perhaps the ultimate answer is -- as frustrating is it may be -- "we have no odds here and can not say one way or the other. It's not 4-1 odds, it's not even a toss-up 1-1 odds, there are no odds, all we think we can see is nothing more than 'the man in the moon', which is an artifact of the viewer and not the moon."
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Percentage of sample above a value

John K. Kruschke
Administrator

In terms of clearly summarizing the posterior, I think it makes good sense and is informative to say "The mean (or modal) credible value is XX and the 95% HDI extends from XX to XX." Notice that summary indicates the magnitude of the difference.

In terms of a decision rule for accepting/rejecting the null, we can say "Although the mean (or modal) credible difference trends toward a small non-zero value, there is enough uncertainty, given these data, that a difference of zero is included among the 95% most credible values, and therefore we do not reject a difference of zero. Nor to we have sufficient precision to declare that the 95% HDI falls within a reasonable ROPE around zero, so we do not accept a different of zero" Notice that this indecisiveness isn't a problem with the decision rule, it's a problem with the data. Bayesian inference shows us clearly that there is a lot of uncertainty given these data.


On Tue, Jul 9, 2013 at 11:36 AM, Wayne [via Doing Bayesian Data Analysis] <[hidden email]> wrote:
That helps a lot. So the answer is that we cannot say whether theta1 and theta2 are different or not with 95% confidence -- there's too much variability in the data to make a decision one way or the other? Which still leaves me with a bit of unease, since it looks fairly obvious to me that there's more evidence that theta1 is greater than theta2 than not. It feels like there should be some way of quantifying and verbalizing what the histogram suggests (to me).

I'd like to be able to say something like "there's mild evidence that theta1 is greater than theta2, though it isn't conclusive." Or "there's a 65% chance that theta1 is greater than theta2." Or "it's more likely that theta1 is greater than theta2 than not, but only slightly larger." Or something like that.

A classical sin in Frequentist statistics would be to lower the confidence interval until we got significance at some level. Perhaps the difference is insignificant at the 95% level, but not at the 50% level. Is that unique to Frequentist statistics and p-values, or would a similar maneuver here be allowable?

That is, can we look at the 85% HDI, the 75% HDI, etc, until we find an HDI that is entirely positive, and then say at that level of confidence that theta1 is likely greater than theta2? I realize that this is similar to my original decision rule #2, since it doesn't take into account the shape of the posterior density, so perhaps the ultimate answer is -- as frustrating is it may be -- "we have no odds here and can not say one way or the other. It's not 4-1 odds, it's not even a toss-up 1-1 odds, there are no odds, all we think we can see is nothing more than 'the man in the moon', which is an artifact of the viewer and not the moon."


If you reply to this email, your message will be added to the discussion below:
http://doing-bayesian-data-analysis.12272.x6.nabble.com/Percentage-of-sample-above-a-value-tp5000689p5000691.html
To start a new topic under Doing Bayesian Data Analysis, email [hidden email]
To unsubscribe from Doing Bayesian Data Analysis, click here.
NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Percentage of sample above a value

Wayne
Perfect, thanks! That gives me a good idea of how much can be said while not going beyond what the data is able to say.
Loading...