Updating Linear Regression Model To TensorFlow 2.0

Introduction

On September 30, 2019, Google announced that the final release of TensorFlow 2.0 (TF 2.0) is now available. From TensorFlow Guide, there are major changes in TF 2.0:

• Removing redundant APIs such as tf.app, tf.flags, tf.logging.
• Executing eagerly like Python (Eager execution)
• Keeping track of your variables and if you lose track of a tf.Variable, TF 2.0 gets garbage collected.
• Using tf.function.

You can discover more about new changes in TF 2.0 at TensorFlow Guide. In this post, I will represent some changes in TF 2.0 by creating linear regression models from scratch.

Background

In one of my older articles, I introduced the linear regression algorithm and how to create a simple linear regression model using TensorFlow 1.X. In this post, I will update that model (with 2 weights) to TF 2.0 and I also create a model with more weights (of course, more than 2).

Using the Code

Simple Linear Regression Model (With Two Weights)

In TensorFlow 1.X, I created a model and implemented it with the following lines of code:

...
# set up variable for weights
w0 = tf.Variable(0.0, name="w0")
w1 = tf.Variable(0.0, name="w1")
...
# Define the operation that will be called on each iteration
train_op = tf.train.GradientDescentOptimizer(learning_rate).minimize(costF)
# set up a session
sess = tf.Session()
# initialize all variables
init = tf.global_variables_initializer()
# execute the session
sess.run(init)
# Loop through the data training
for epoch in range(training_epochs):
       for (x, y) in zip(x_train, y_train):
              # execute the session
              sess.run(train_op, feed_dict={X: x, Y: y})
# get values of the final weights by executing the session
w_val_0 = sess.run(w0)
w_val_1 = sess.run(w1)

As code above, in TF 1.X, we must:

• declare variables (tf.Varialble) and initialize these variables (tf.global_variables_initializer) before using them.
• train our model using tf.train.GradientDescentOptimizer.
• set up (tf.Session) and run the session to execute operations in the graph.

In TF 2.0, we will:

• declare variables (tf.Variable) but don't need to use tf.global_variables_initializer, that means, TensorFlow 2.0 doesn’t make it mandatory to initialize variables.
• train our model using tf.GradientTape and we will use assign_sub for weight variables.
• not require the session execution.

From the main points above, we will re-create our linear regression model with two weights as follows:

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

learning_rate = 0.01

# steps of looping through all your data to update the parameters
training_epochs = 100

# the training set
x_train = np.linspace(0, 10, 100)
y_train = x_train + np.random.normal(0,1,100)

w0 = tf.Variable(0.)
w1 = tf.Variable(0.)

def h(x):
   y = w1*x + w0
   return y

def squared_error(y_pred, y_true):
   return tf.reduce_mean(tf.square(y_pred - y_true))

# train model
for epoch in range(training_epochs):
    with tf.GradientTape() as tape:
        y_predicted = h(x_train)
        costF = squared_error(y_predicted, y_train)
    # get gradients
    gradients = tape.gradient(costF, [w1,w0])
    # compute and adjust weights
    w1.assign_sub(gradients[0]*learning_rate)
    w0.assign_sub(gradients[1]*learning_rate)

plt.scatter(x_train, y_train)
# plot the best fit line
plt.plot(x_train, h(x_train), 'r')
plt.show()

If we run this script, the result can look like this:

Polynomial Model

When data points appear to form smooth curves rather than straight lines, we need to change our regression model from a straight line to something else. One such approach is to use a polynomial model. A polynomial is a generalization of a linear function. The following lines of code will create a polynomial model:

import tensorflow as tf 
import numpy as np 
import matplotlib.pyplot as plt
 
learning_rate = 0.05 
training_epochs = 100 

#the traning set
x_train = np.linspace(-1, 1, 101) 
# Set up raw output data based on a degree 5 polynomial
num_coeffs = 6 
trY_coeffs = [1, 2, 3, 4, 5, 6] 
y_train = 0 
for i in range(num_coeffs): 
   y_train += trY_coeffs[i] * np.power(x_train, i)
# Add some noise 
y_train += np.random.randn(*x_train.shape) * 1.5 

# Set up the weight vector to all zeros
w = tf.Variable([0.] * num_coeffs, name="parameters")

# our model function
def h(x):
   # h(x) = w5*x*x*x*x*x + w4*x*x*x*x + w3*x*x*x + w2*x*x + w1*x + w0
   y = 0
   for i in range(num_coeffs): 
      y += w[i]*pow(x,i)
   return y

# cost function
def squared_error(y_pred, y_true):
   return tf.reduce_mean(tf.square(y_pred - y_true))

# train model
for epoch in range(training_epochs):
   with tf.GradientTape() as tape:
      y_predicted = h(x_train)
      costF = squared_error(y_predicted, y_train)
   # get gradients
   gradients = tape.gradient(costF, w)
   # compute and adjust weights
   w.assign_sub(gradients*learning_rate)

plt.scatter(x_train, y_train)
# plot the best fit line
plt.plot(x_train, h(x_train), 'r')
plt.show()

If we run this script, the result can look like this:

Of course, we need to improve this model but I don't do that in this post.

Points of Interest

TensorFlow is a great platform for deep learning and machine learning and TF 2.0 focuses on simplicity and ease of use. In this post, I introduced some new changes in TF 2.0 by building simple linear regression models from scratch (don't use APIs such as Keras) and I hope you feel excited about it.

History

• 25^th March, 2020: Initial version