A Note on Python, Virtualenv, VSC Integration, and Miscellaneous

• Download source - 17.5 KB
• Download a_01_oo_excercise.zip

Background

Python is claimed to be an easy to learn and easy to use language, but this note is not about the Python language. It is about creating an isolated environment where we can run Python programs with certainty.

In this note, I attached a few simple Python files. I will use them to show you the benefit to have an isolated environment to run the Python programs. By the way, if you use Ubuntu flavored Linux, this link https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa is a great link for you.

How Many Pythons?

To run a Python program, we typically need three components on the computer.

1. A Python interpreter
2. An instance of Python Package Installer (pip) to install the dependency packages
3. A search path that the interpreter looks for the dependency packages to run our programs

Multiple Interpreters

One of the Python claimed advantages is that it is readily available on all the computers. This also means that we may have a version of Python interpreter without our knowledge. In my Linux Mint 17.2 Cinnamon 64-bit, I have at least 3 Python interpreters.

When we install an operating system, it comes with certain pre-installed Python interpreters. These Python interpreters are important for the operating system to function. We should not remove them without knowing exactly what we are doing.

Multiple Package Installers

To make things more complicated, we actually have more than one instance of pip on the computer.

Multiple Search Paths

When we run a Python program, we need to make sure that the correct instance of the interpreter is used. Let us take a look at the python-search-path.py file.

# https://docs.python.org/3/tutorial/modules.html#the-module-search-path
import sys
    
for item in sys.path:
  print(item)

If we run the program by the following command, we can find the package search path used by the python interpreter.

python python-search-path.py

If we run it by the python3.4 interpreter, we will see a totally different path.

python3.4 python-search-path.py 

Now we know that the Python versions on a computer are complicated. What makes it more unstable is that we may be using the same version of Python that is also used by the operating system. If we update a certain package that is also used by the operating system, we may cause a problem to the operating system. In the worst case, it may not even start. I have personally encountered problems with the OS, after playing with some versions of Python.

The Virtual Environment

We can work diligently to make sure that we are using the desired Python interpreter. We can also work diligently to install the packages into the correct search path to make a Python program run smoothly. But we can also create a virtual environment to easily run a Python program with certainty.

• A virtual environment makes sure that we use the desired Python interpreter.
• A virtual environment makes sure that we use the desired Python Package Installer (pip).
• A virtual environment makes sure that we install the packages in the environment and look for them in the environment when running a program.

Physically, a virtual environment is a folder. This folder contains almost everything to run a Python program. When we install a package, it is also installed in this folder.

Create a Virtual Environment

To create a virtual environment, we need to install the virtualenv tool. In my computer, I use the following command to install the virtualenv.

sudo python3 -m pip install virtualenv

We can then validate the successful installation.

It looks like that the virtualenv is independent from any specific version of Python. With the virtualenv installed, we can then create a virtual environment by the following command:

virtualenv -p /usr/bin/python3.4 environment-3.4

• When creating the virtual environment, we specified the execution path to a Python interpreter, it will be the Python interpreter when we use the virtual environment.
• When creating the virtual environment, we specified a folder name. It is the physical folder to hold the virtual environment.

Activate a Virtual Environment

To activate a virtual environment, we can issue the following command:

source environment-3.4/bin/activate

When a virtual environment is activated, we can then check the execution path to Python and pip.

python python-search-path.py

With the virtual environment activated, we can see that all the python, pip, and the search path are located inside the virtual environment folder. We can deactivate the virtual environment by the following command:

deactivate

If we want to completely remove the virtual environment, we can simply delete the environment-3.4 folder.

Install Packages in a Virtual Environment

For experimental purposes, I created a small flask application in the python-flask-api.py file.

from flask import Flask, request, jsonify
    
app = Flask(__name__)
    
@app.route('/')
def message():
  name = 'Song Li'
  return jsonify(
    username = name,
    email = 'song.li@email.com',
  )
    
if __name__ == '__main__':
   app.run(host='0.0.0.0', port=3000)

This application requires the flask package. With the virtual environment activated, we can use the following command to install the package:

python -m pip install flask

With flask installed, we can then run the python-flask-api.py file.

python python-flask-api.py 

We can further verify that the flask package is installed in the virtual environment by running the python-inspect-path.py file.

import inspect
from flask import Flask
    
print(inspect.getfile(Flask))

Another way to create a virtualenv is to use venv. This is a good document on venv. In Ubuntu, we need to manually install the python3.9-venv (or the corresponding version of venv).

apt-cache policy python3.9-venv

We can then create the virtualenv with a simple command.

python3.9 -m venv "virtualenv-directory-name"

We can reference the https://docs.python.org/3/library/venv.html for more advanced usages of venv.

The PIP Freeze

After working on a virtual environment for some time, you may have installed many packages. If you want to let another person run your program, you need to tell them these packages and the versions. With the virtual environment activated, you can issue the following command:

python -m pip freeze > requirements.txt 

It creates a file named requirements.txt that has all the information about the packages in the virtual environment.

Click==7.0
Flask==1.0.4
itsdangerous==1.1.0
Jinja2==2.10.3
MarkupSafe==1.1.1
Werkzeug==0.16.1

We can use the following command to install all the packages in a new virtual environment:

python -m pip install -r requirements.txt

We can also remove all the packages in the requirements.txt file by the following command:

python -m pip uninstall -r requirements.txt -y

The pip freeze command takes a --path parameter, it limits the requirements.txt file to include the packages installed in a certain path.

python3 -m pip freeze --path .  > requirements.txt

The pip install command take a -t or --target parameter to install the packages to the path specified.

python3 -m pip install -r requirements.txt -t .

Jupyter

It is very convenient to use Python. Here is the document on Jupyter. We can install Jupyter in a virtual environment.

python3.9 -m venv env-3.9
source env-3.9/bin/activate
pip install pip -U

The above command creates a virtual environment and upgrades pip in it. We can verify the virtual environment by the following command:

which python
pip --version

We can install Jupyter with the following command:

pip install jupyterlab

We can start Jupyter by the following command:

jupyter-lab

If we want to convert a ipynb file to a regular python file, we can use the following command:

python -m nbconvert --to script --no-prompt Untitled.ipynb

We can check if nbconvert is installed by the following command:

pip list | grep nbconvert

The VSC Integration

The Visual Studio Code is a nice IDE for Python programs. If you are not familiar with VSC, you can take a look at my earlier note. To support Python, I only need to install the Python extension by Microsoft.

To be able to run and debug the program with the Python interpreter by the virtual environment, we can simply create a launch.json file like the following. It points the pythonPath to the virtual environment.

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "python-search-path.py",
      "pythonPath": "${workspaceFolder}/environment-3.4/bin/python",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/python-search-path.py",
      "console": "integratedTerminal"
    },
    {
      "name": "python-inspect-path.py",
      "pythonPath": "${workspaceFolder}/environment-3.4/bin/python",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/python-inspect-path.py",
      "console": "integratedTerminal"
    },
    {
      "name": "python-flask-api.py",
      "pythonPath": "${workspaceFolder}/environment-3.4/bin/python",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/python-flask-api.py",
      "console": "integratedTerminal"
    }
  ]
}

If you like, you can actually add the path to the python executable in the settings.json file.

{
  "files.exclude": {
    "**/.git": true,
    "**/.gitignore": true,
    "**/environment-3.4": true
  },
  "python.pythonPath": "${workspaceFolder}/environment-3.4/bin/python3.4"
}

In this case, whenever you open the terminal, the virtual environment is automatically activated.

Flask & Gunicorn

The Flask and Django are the web frameworks in Python environment. The python-flask-api.py is a simple flask application.

import sys, os
from flask import Flask, jsonify
    
app = Flask(__name__)
    
@app.route('/')
def message():
  return jsonify(pid = os.getpid())
    
if __name__ == '__main__':
   app.run(host='0.0.0.0', port=3000)

We can launch the application by the following command:

python python-flask-api.py

But when we start the application, we see the following warning.

We need to find a good server to host the application. We have many options, but Gunicorn is a popular one. We can actually install Gunicorn in the virtual environment.

python -m pip install gunicorn

After installing Gunicorn, we can run it in the virtual environment.

gunicorn --bind 0.0.0.0:8000 python-flask-api:app

If everything goes well, we can now access the application through the 8000 port number. We can also add the -w option to specify the number of worker processes. Typically, the number of worker processes are 2-4 per core in the server.

gunicorn --bind 0.0.0.0:8000 -w 4 python-flask-api:app 

It is important to know that Flask can on multi-thread by itself (Reference).

if __name__ == '__main__':
    app.run(threaded=True)

or

if __name__ == '__main__':
    app.run(threaded=False, processes=3)

Load Balancing with Nginx

The Nginx is a popular load balancer. It is common to use Nginx to load balance multiple Flask instances. To install Nginx on my Linux Mint computer, we can use the following commands.

apt-cache policy nginx
sudo apt-get install nginx

After installation, we can use the following commands to manipulate the Nginx server.

sudo service nginx start
sudo service nginx restart
sudo service nginx reload
sudo service nginx stop
service nginx status

Because I am experimenting Nginx on my personal computer, I do not want it to start when I start the computer, I use the following command to disable the service.

sudo update-rc.d nginx disable

To set up the load balancing, I created a file named loadbalance.conf.

upstream pythonweb {
  server localhost:4000;
  server localhost:3000;
  server localhost:2000;
}
    
# This balances all the requests
# It also disable caching
server {
  listen 80;
    
  location / {
    proxy_pass "http://pythonweb/";
    add_header Cache-Control
                'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
  }
}
    
# nginx.conf file
# Comment out this line - include /etc/nginx/sites-enabled/*;
#include /home/song/sandbox/p-virtualenv-excercise/loadbalance.conf;

In my computer, I need to "include" this file in the /etc/nginx/nginx.conf file and comment out the default /etc/nginx/sites-enabled/*.

#include /etc/nginx/sites-enabled/*;
include /home/song/sandbox/p-virtualenv-excercise/loadbalance.conf;

With the configuration being ready, we can restart or reload the Nginx server. We need also start 3 instances of the Flask application on the port numbers 2000/3000/4000 in different terminals. The following is the command to start an instance on the port number 2000.

gunicorn --bind 0.0.0.0:2000 python-flask-api:app

If everything goes well, we can then access the application by port number 80 and the requests are balanced among the 3 Flask instances.

Python Classes & OO

Python does support classes and object orientation. In this section, we will be using a few examples to give a quick summary on Python classes & OO.

Example 1 - Python Empty Classes and Dynamic Attributes

# Python allows empty class
class EmptyClass:
  pass
    
# Create an instance of the class
obj_1 = EmptyClass()
    
# Attrubutes can be added to an object dynamically
obj_1.attribute1 = "The attribute 1"
obj_1.attribute2 = "The attribute 2"
    
print(obj_1.attribute1 + ' - ' + obj_1.attribute2)
    
# The __dict__ has all the information
print('The __dict__:')
print(obj_1.__dict__)
    
# Attributes can be deleted from an object
del obj_1.attribute2
print(obj_1.__dict__)
print()

Unlike classes in the other languages, a Python class does not need any implementation.

• The instance attributes can be added to a Python object dynamically;
• The instance attributes (including the __dict__ attribute) can be also removed from the object dynamically;
• The __dict__ attribute maintains all the information of the Python object.

Running the above program, we can see the following result:

The attribute 1 - The attribute 2
The __dict__:
{'attribute1': 'The attribute 1', 'attribute2': 'The attribute 2'}
{'attribute1': 'The attribute 1'}

Example 2 - Python Constructor and Instance Methods

# A class with an instructor and an instance method
class AClass:
  def __init__(self, attr_1, attr_2):
    self.attr_1 = attr_1
    self.attr_2 = attr_2
    
  def print_attributes(self):
    print("{} - {}".format(self.attr_1, self.attr_2))
    
# Create two instances
obj_1 = AClass('obj_1_a1', 'obj_1_a2')
obj_2 = AClass('obj_2_a1', 'obj_2_a2')
    
# Call the instance method
# Each object has its own instance attributes
obj_1.print_attributes()
obj_2.print_attributes()
print()

Although we can add dynamic attributes to Python objects, the most common way to initiate attributes is by a constructor.

• A Python constructor is named as __init__, the first parameter to the constructor is the reference of the object;
• A Python instance method also takes the first parameter as the reference of the object. By convention, this parameter is named as self.
• When the constructor and the instance methods are called, the self parameter is passed in implicitly by Python.

Running the above program, we can see the following result:

obj_1_a1 - obj_1_a2
obj_2_a1 - obj_2_a2

Example 3 - The Class Attributes

# A class with both instance and class attributes
class AClass:
  a_class_attr = 'The class attribute'
    
  def print_class_attribute_by_self_reference(self):
    print(self.a_class_attr)
    
  def print_class_attribute_by_class_name(self):
    print(AClass.a_class_attr)
    
# Create an instance of AClass
obj_1 = AClass()
    
# A class attribute can be accessed by
# both the instance reference and the class name
obj_1.print_class_attribute_by_self_reference()
obj_1.print_class_attribute_by_class_name()
print()
    
# Add a instance attribute with the same name
obj_1.a_class_attr = 'This can override the class attribute'
obj_1.print_class_attribute_by_self_reference()
obj_1.print_class_attribute_by_class_name()
print()
    
# Delete the instance attribute
del obj_1.a_class_attr
obj_1.print_class_attribute_by_self_reference()
obj_1.print_class_attribute_by_class_name()
print()

Besides the instance attributes, a Python class can have class level attributes.

• A class attribute can be access by the object reference and the class name;
• If the object has a instance attribute with the same name as the class attribute, and if the attribute is accessed by the object reference, the instance attribute overrides the class attribute.

Running the above program, we can see the following result.

The class attribute
The class attribute
    
This can override the class attribute
The class attribute
    
The class attribute
The class attribute

Example 4 - Class Methods and Static Methods

# A class with an instance method, a class method,
# and a static method
class AClass:
  a_class_attr = 'The class attribute'
    
  def __init__(self):
    self.a_instance_attr = 'The instance attribute'
    
  def an_instance_method(self):
    print("From the instance method - {}".format(self.a_instance_attr))
    
  @classmethod
  def a_class_method(cls):
    print("From the class method - {}".format(cls.a_class_attr))
      
  @staticmethod
  def a_static_mathod():
    print('Print from a static method')
    
    
# Create an instance of AClass and call the methods
obj_1 = AClass()
obj_1.an_instance_method()
obj_1.a_class_method()
obj_1.a_static_mathod()
print()
    
# Class and static methods can be accessed by the class name
AClass.a_class_method()
AClass.a_static_mathod()

A Python class can have class methods and static methods.

• Annotating a method by @classmethod makes it a class method. When the method is called, the reference to the class is implicitly passed in as the first parameter. By convention, this parameter is named cls;
• Annotating a method by @staticmethod makes it a static method. When the method is called, neither the object reference nor the class reference is passed in implicitly.

Running the above program, we can see the following result.

From the instance method - The instance attribute
From the class method - The class attribute
Print from a static method
    
From the class method - The class attribute
Print from a static method

Example 5 - Inheritance

# The parent class
class ParentClass:
  def __init__(self):
    self.parent_attr = 'Parent Attribute'
    
  def print(self):
    print("Print from parent class - {}".format(self.parent_attr))
    
# The child class
class ChildClass(ParentClass):
  def __init__(self):
    super().__init__()
    self.child_attr = 'Child Attribute'
    
  def print(self):
    print("Print from child class - {}".format(self.child_attr))
      
# Create instances for both classes
# and call the print method
parent_obj = ParentClass()
parent_obj.print()
    
child_obj = ChildClass()
child_obj.print()
print()
    
# The the polymorphism behavior, kind of ...
print('Test polymophism')
objects = [parent_obj, child_obj]
for obj in objects:
  obj.print()

Same as other OO languages, Python supports inheritance.

• The super() method can be used to access the method in the parent class;
• A method is overridden if the child class implements the same method;
• Python objects show polymorphism behavior similar to other languages.

Running the above program, we can see the following result:

Print from parent class - Parent Attribute
Print from child class - Child Attribute
    
Test polymophism
Print from parent class - Parent Attribute
Print from child class - Child Attribute

Spark & PySpark

Points of Interest

• This is a note on Python & Virtualenv & VSC & Miscellaneous.
• I hope you like my posts and I hope this note can help you one way or the other.

History

• 28^th January, 2020: Initial version