The next concept we will look at is significance testing. Significance testing is a form of hypothesis testing focused on linear regression outputs.
Hypothesis testing is using a test statistic to determine the probability that a hypothesis is accepted or rejected. We determine the probability using prediction intervals. The basic format of hypothesis testing is such:
 Null Hypothesis: Value is explained by model
 Alt Hypothesis: Reject null hypothesis
 T – test statistic = TTest Formula = Coeffecient/Standard Error =
 Confidence interval = predicted value + (critical value)(standard error of forecast)
 Where predicted value could be sample mean
 Confidence Intervals:

Where:

 Confidence Interval = Critical Value at Significance Level
The critical value is determined by our desired accuracy level. A five percent, two tailed tstatistic has a critical value of 1.96. If the value created by our test statistic lies within the 1.96 confidence interval, then we say the null hypothesis is valid. Otherwise we reject the null hypothesis.
For example – lets take a ttest stat of 1.5. The 5% Confidence interval = 3 to 2. Thus we accept that the value of the test stat has a 95% chance of being explained by the model, and we accept the null hypothesis.
Taking this back to the ANOVA, we see there are two test statistics that are usually presented – the Ftest, the TTest, as well as a pvalue for each of these.
The Ftest is for the significance of the model, in other we test the significance of each coefficient at once.
The Ttest allows us to test for the significance of each individual coefficient.
The test we are conducting asks whether the coefficient values are significantly different from 0. The Ftest tests a null hypothesis that all the coefficients equal zero. If we reject the null hypothesis, this means that at least one of the coefficients is significantly different from zero, and therefore significantly contributes to the explanation of the dependent variable. The ttest is the same concept but on a coeffecient level instead of the model level.
The pvalue is a further check on our test statistics. The pvalue is a probability that indicates the likelihood that our test statistic would be generated when the null hypothesis is true.
Let’s say we reject the null hypothesis based on an Ftest value of 230, where the critical value is 160. If we have a Pvalue of .001, that means that there is a 0.001% chance that we would generate a Ftest value of 230 under circumstances where our regression coefficients did equal zero. In other words, there is a 0.001% chance that we have incorrectly rejected the null hypothesis. However, if the pvalue where .85, this would indicate an 85% chance that the value 230 could be generated when our coefficients equal zero, which would lead us to question our model’s validity.
In short, a low pvalue agrees with rejecting a null hypothesis, while a higher pvalue favors the null hypothesis.