## Introduction

Logisitic regression is a classical method in statistical learning, which calculates the conditional probability `P(Y|X)`

and uses the label of the larger one as the prediction. Specifically, the binomial logistic regression model is:

where `w`

and `b`

are weight and bias, respectively. For convenience, expend weight vector and bias vector, namely,

Then, the binomial logistic regression model is:

## Logistic Regression Model

Logistic Regression model consists of parameters estimation, optimization algorithm and classify.

### Parameters Estimation

In Step-by-Step Guide To Implement Machine Learning III - Naive Bayes, we use the Maximum likelihood function to estimate the parameters in the Baysian model. Similarly, we use Maximum likelihood function to estimate the parameters in Logistic Regression Model. Denote

where:

`g(x)`

is also called **sigmoid function**. The likehood function is:

For convenience, we take the logarithm of the likehood function, namely:

Then, the problem is transformed into calculating the max of the likehood function.

### Optimization Algorithm

Because, **we cannot get an analytic solutions to the derivative of likehood function**. To get the max of likehood function, we apply the **gradient ascent method**, namely:

calculate the derivative of likelihood function:

Let the derivative equal to zero, we can get:

Thus, we can get the optimized parameter through the above equation. The code of gradient ascent method is shown below:

if method == "GA": weights = np.random.normal(0, 1, [feature_dim, 1]) for i in range(iterations): pred = self.sigmoid(np.dot(train_data, weights)) errors = train_label - pred # update the weights weights = weights + alpha * np.dot(train_data.T, errors) self.weights = weights return self

### Classify

In logistics regression model, sigmoid function is applied to calculate the probability, which is expressed as:

When the result is larger than `0.5`

, the sample belongs to class 1, else it belongs to class `0`

.

def sigmoid(self, x, derivative=False): output = 1/(1 + np.exp(-x)) if derivative: output = output * (1 - output) return output

## Conclusion and Analysis

To get the parameters of the logistic regression model, we can also minimize the loss function. Finally, let's compare our logistics regression with the Sklearn's and the detection performance is displayed below:

The detection performance of both is similar.

The related code and dataset in this article can be found in MachineLearning.

## History

- 13
^{th}May, 2019: Initial version