How to Solve the Logistic Regression Equation for the Probability p.

As discussed previously, in logistic regression the log odds is modeled as a linear function of the X-variables. That is

    \[\log\left( p \over 1-p \right) = \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p\; .\]

To solve this equation for p, we first apply the exponential function to both sides of the equation:

    \[\exp\left( \log\left( p \over 1-p \right) \right) = \exp\left( \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p \right) .\]

Recall that \exp(z)=e^z so that the right hand side of the above equation is

    \[\exp\left( \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p \right) = e^{\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p} .\]

Also remember that “log” is the natural logarithm, so the exponential function is its inverse (i.e., \exp(\log(z))=z). Thus, the left hand side is

    \[\exp\left( \log\left( p \over 1-p \right) \right) = { p \over 1-p }.\]

Thus, after exponentiating both sides, logistic regression equation becomes:

    \[\begin{split} { p \over 1-p } &= \exp\left( \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p \right)\\ &= e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } \end{split}\]

Next multiply both sides by 1-p,

    \[p = (1-p)\:e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } ,\]

and then “break up” the (1-p) term,

    \[p = e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } - p\;e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } .\]

Now move the p\;e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } term (the last term on the right-hand side) over to the left-hand side by adding it to both sides:

    \[p + p\;e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } = e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p }\]

Next, factor out the p,

    \[p \left( 1+e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } \right) = e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p }\]

Finally, divide both sides by \left( 1+e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } \right) to get p:

    \[p = { e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } \over 1+e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p }\]

This is the equation for p that you will see in multiple articles on this website.

There is one other form of this equation that is commonly used. It is obtained by multiplying the top and bottom of the right hand side of the equation for p by e^{ -(\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p) }. Since e^{-z}e^z=1, this gives

    \[\begin{split} p &= { e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } \over 1+e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } } \times { e^{ -(\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p) } \over e^{ -(\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p) } }\\ &= { e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p }\times e^{ -(\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p) } \over e^{ -(\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p) } + e^{ -(\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p) } \times e^{ \alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p } }\\ & = { 1 \over e^{ -(\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p) } + 1 } \end{split}\]

The terms in the denominator are customarily written in the opposite order. So the second form of the equation for p is

    \[p = { 1 \over 1 + e^{ -(\alpha + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p) } }\]

I have probably put in too many steps in the derivation above, but I wanted it to be accessible to almost everyone, even if your algebra skills are a little rusty.

Questions or Comments?

Any questions or comments? Please feel free to comment below. I am always wanting to improve this material.

I have had to implement a very simple “captcha” field because of spam comments, so be a bit careful about that. Enter your answer as a number (not a word). Also, save your comment before you submit it in case you make a mistake. You can use cntrl-A and then cntrl-C to save a copy on the clipboard. Particularly if your comment is long, I would hate for you to lose it.

This entry was posted in TheMath and tagged , , , . Bookmark the permalink.

2 Responses to How to Solve the Logistic Regression Equation for the Probability p.

  1. Pingback: Logistic Regression Sigmoid Function – Wang Zhe

  2. Aye Moh says:

    Please, explaine how to find the coefficient Beta1,Beta2,….,Betap in above probability equation.

Leave a Reply

Your email address will not be published. Required fields are marked *


ENTER answer below as a NUMBER.
CAREFUL! Error will delete your comment. Save it first (cntrl-A then cntrl-C saves to clipboard). *
Time limit is exhausted. Please reload the CAPTCHA.