Logisitic regression is a classical method in statistical learning, which calculates the conditional probability
P(Y|X) and uses the label of the larger one as the prediction. Specifically, the binomial logistic regression model is:
b are weight and bias, respectively. For convenience, expend weight vector and bias vector, namely,
Then, the binomial logistic regression model is:
Logistic Regression Model
Logistic Regression model consists of parameters estimation, optimization algorithm and classify.
In Step-by-Step Guide To Implement Machine Learning III - Naive Bayes, we use the Maximum likelihood function to estimate the parameters in the Baysian model. Similarly, we use Maximum likelihood function to estimate the parameters in Logistic Regression Model. Denote
g(x) is also called sigmoid function. The likehood function is:
For convenience, we take the logarithm of the likehood function, namely:
Then, the problem is transformed into calculating the max of the likehood function.
Because, we cannot get an analytic solutions to the derivative of likehood function. To get the max of likehood function, we apply the gradient ascent method, namely:
calculate the derivative of likelihood function:
Let the derivative equal to zero, we can get:
Thus, we can get the optimized parameter through the above equation. The code of gradient ascent method is shown below:
if method == "GA":
weights = np.random.normal(0, 1, [feature_dim, 1])
for i in range(iterations):
pred = self.sigmoid(np.dot(train_data, weights))
errors = train_label - pred
weights = weights + alpha * np.dot(train_data.T, errors)
self.weights = weights
In logistics regression model, sigmoid function is applied to calculate the probability, which is expressed as:
When the result is larger than
0.5, the sample belongs to class 1, else it belongs to class
def sigmoid(self, x, derivative=False):
output = 1/(1 + np.exp(-x))
output = output * (1 - output)
Conclusion and Analysis
To get the parameters of the logistic regression model, we can also minimize the loss function. Finally, let's compare our logistics regression with the Sklearn's and the detection performance is displayed below:
The detection performance of both is similar.
The related code and dataset in this article can be found in MachineLearning.
- 13th May, 2019: Initial version