Python Brief Introduction

Brief Intro to Python

Outline

• Why Python
• Jupyter notebook
• Python starter
• Python for scientific computing

Why Python

Python is a generic scripting computer language used for many purposes

• System administration
• Web development
• game development
• multimedia, natural language
• Scientific computing
• Data science and statistics
• Machine Learning, …

Python is increasingly popular in the scientific community pretty much in all areas. Economists have been rapidly learning to use Python in research and teaching, and it is no stranger to young PhD students and researchers.

The ecosystem of Python developer and user community is huge, it is safe to say that Python is the computer language for everything.

Python is awarded the programming language of the year 2020.

Python trends

Jupyter notebook

The Jupyter Notebook is an interactive computing environment that enables users to author notebook documents that include:

• Live code
• Interactive widgets
• Plots
• Narrative text
• Equations
• Images
• Video

These documents provide a complete and self-contained record of a computation that can be converted to various formats and shared with others using email, Dropbox, version control systems (like git/GitHub) or nbviewer.jupyter.org.

The Jupyter Notebook combines three components:

• The notebook web application: An interactive web application for writing and running code interactively and authoring notebook documents.
• Kernels: Separate processes started by the notebook web application that runs users’ code in a given language and returns output back to the notebook web application.
• Notebook documents: Self-contained documents that contain a representation of all content visible in the notebook web application, including inputs and outputs of the computations, narrative text, equations, images, and rich media representations of objects. Each notebook document has its own kernel.

Notebooks consist of a sequence of cells. There are three basic cell types:

• Code cells: Input and output of live code that is run in the kernel
• Markdown cells: Narrative text with embedded LaTeX equations
• Raw cells: Unformatted text that is included, without modification, when notebooks are converted to different formats using nbconvert

Internally, notebook documents are JSON data with binary values base64 encoded. This allows them to be read and manipulated programmatically by any programming language. Because JSON is a text format, notebook documents are version control friendly.

Notebooks can be exported to different static formats including HTML, reStructeredText, LaTeX, PDF, and slide shows (reveal.js) using Jupyter’s nbconvert utility.

Furthermore, any notebook document available from a public URL or on GitHub can be shared via nbviewer. This service loads the notebook document from the URL and renders it as a static web page. The resulting web page may thus be shared with others without their needing to install the Jupyter Notebook.

Through Jupyter’s kernel and messaging architecture, the Notebook allows code to be run in a range of different programming languages. For each notebook document that a user opens, the web application starts a kernel that runs the code for that notebook. Each kernel is capable of running code in a single programming language and there are kernels available in the following languages:

• Python(https://github.com/ipython/ipython)
• Julia (https://github.com/JuliaLang/IJulia.jl)
• R (https://github.com/IRkernel/IRkernel)
• Ruby (https://github.com/minrk/iruby)
• Haskell (https://github.com/gibiansky/IHaskell)
• Scala (https://github.com/Bridgewater/scala-notebook)
• node.js (https://gist.github.com/Carreau/4279371)
• Go (https://github.com/takluyver/igo)

The default kernel runs Python code. The notebook provides a simple way for users to pick which of these kernels is used for a given notebook.

Each of these kernels communicate with the notebook web application and web browser using a JSON over ZeroMQ/WebSockets message protocol that is described here. Most users don’t need to know about these details, but it helps to understand that “kernels run code.”

Python starter

Python is generic. It is a calcultor to start with, and it is a gigantic system used to run big systems by corporates.

Data types

• boolean
• int, double, complex
• strings
• None

Operators

• mathematical
• logical
• bitwise
• membership
• identity
• assignment and in-place operators
• operator precedence

Collections

• Sequence containers - list, tuple
• Mapping containers - set, dict
• The collections module

Functions and methods

• Anatomy of a function
• Docstrings
• Class methods

Control flow

• if and the ternary operator
• Checking conditions - what evaluates as true/false?
• if-elif-else
• while
• break, continue
• pass

Loops and comprehensions

• for, range, enumerate
• lazy and eager evaluation
• list, set, dict comprehensions
• generator expression

Packages and namespace

• Modules (file)
• Package (hierarchical modules)
• Namespace and naming conflicts
• Using import
• Batteries included

3+4

7

Arithmetic Operators

The syntax for arithmetic operators in Python are:

Notice that division of integers always returns a float:

long_winded_computation = (1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 +
                           13 + 14 + 15 + 16 + 17 + 18 + 19 + 20)
print("long_winded_computation=",long_winded_computation)

long_winded_computation= 210

Comparison Operators

Comparison operators produce Boolean values as output. For example, if we have variables x and y with numeric values, we can evaluate the expression x < y and the result is a boolean value either True or False.

For example:

Boolean Operators

We combine logical expressions using boolean operators and, or and not.

For example:

math_is_scary = False
print(math_is_scary)

False

# Note the indention
names = ["Bob", "Alice", "Zack", "Tyler"]
for x in names:
    print(x)

Bob
Alice
Zack
Tyler

Reserved Words

Summarized below are the reserved words in Python 3. Python will raise an error if you try to assign a value to any of these keywords and so you must avoid these as variable names.

Built-in Function Names

There are several functions which are included in the standard Python library. Do not use the names of these functions as variable names otherwise the reference to the built-in function will be lost. For example, do not use sum, min, max, list or sorted as a variable name. See the full list of builtin functions.

"""Anonymous functions are handy"""
fib = [0,1,1,2,3,5,8,13,21,34,55]
results = list(filter(lambda x: x % 2==0, fib))
print('results:', results)

results: [0, 2, 8, 34]

Loops

for Loops

A for loop allows us to execute a block of code multiple times with some parameters updated each time through the loop. A for loop begins with the for statement:

iterable = [1,2,3]
for item in iterable:
    # code block indented 4 spaces
    print(item)

1
2
3

while Loops

What if we want to execute a block of code multiple times but we don’t know exactly how many times? We can’t write a for loop because this requires us to set the length of the loop in advance. This is a situation when a while loop is useful.

The following example illustrates a while loop:

n = 5
while n > 0:
    print(n)
    n = n - 1

5
4
3
2
1

Sequences

The main sequence types in Python are lists, tuples and range objects. Main differences between these sequence objects are:

• Lists are mutable and their elements are usually homogeneous (things of the same type making a list of similar objects)
• Tuples are immutable and their elements are usually heterogeneous (things of different types making a tuple describing a single structure)
• Range objects are efficient sequences of integers (commonly used in for loops), use a small amount of memory and yield items only when needed

Lists

Create a list using square brackets [ ... ] with items separated by commas. For example, create a list of square integers, assign it to a variable and use the built-in function print() to display the list:

fib = [0,1,1,2,3,5,8,13,21,34,55]
results2 = [x for x in fib if x % 2 == 0]
for x in results2:
    print(x)

0
2
8
34

Tuples

Tuples are similar to lists but are immutable, i.e., elements in a tuple cannot be changed.

listx = [3,4]
tupley = (3,4)
listx[1]=6
print("listx",listx)
try:
    tupley[1] = 0
except TypeError:
    print("cannot modify a tuple")

listx [3, 6]
cannot modify a tuple

Dictionaries

grades = {"Joel": 80, "Tim": 95}
x = grades.keys()
print("x",x)
y = grades.values()
print("y",y)

x dict_keys(['Joel', 'Tim'])
y dict_values([80, 95])

Functions

A code block that returns results, and can be used repeatedly.

"""
this code solve for consumer's utility maximization problem. U(C,l) is Cobb-Douglas, l is leisure, C is consumption
I worked out first-order conditions in equations, translated them to code
"""
import numpy as np

# utility function
def utilityfun(l,C,shareC, sharel, sigma):
    tmplc = (l**sharel) * (C**shareC)
    return (tmplc**(1.0-sigma) - 1.0)/(1.0-sigma)
# optimal choices

def optimalChoice(sharel,shareC,wage,hmax,profit,tax):
    lstar = (sharel/(sharel+shareC)) *(wage*hmax + profit - tax) / wage
    Cstar = lstar * wage * shareC / sharel
    return lstar,Cstar

# parameters
shareC = 0.65
sharel = 1.0 - shareC
sigma = 1.4

wage, profit, tax = 0.5, 1.8, 0.5
hmax = 5.5
#---optimal cohices: c/l = w * sharec/sharel
lstar, Cstar=optimalChoice(sharel,shareC,wage,hmax,profit,tax)
# maximized utility under wage
util0 = utilityfun(lstar,Cstar,shareC, sharel, sigma)

print('Optimal C=', Cstar, "optimal l=", lstar)
print('Maximized utility level=', util0)

Optimal C= 2.6325 optimal l= 2.8349999999999995
Maximized utility level= 0.8200869606862031

Built-in Functions

The standard Python library has a collection of built-in functions ready for us to use. We have already seen a few of these functions in previous sections such as type(), print() and sum(). The following is a list of built-in functions that we’ll use most often:

Python for scientific computing

Python is one of the core languages of scientific computing. In fact, it is the modules (libraries) written in Python that have become the essential tools for scientific computing.

Many of the Python modules are translated from numerical libraries written in C and Fortran.

Its popularity in economics has been rising rapidly, a collaborated effort is quantecon.org.

We take a bird’s-eye view on Python modules for scientific computing

In Python, a module is a file that contains python definitions and statements. We need to import them when we need to use them.

NumPy library is fundamental for many other modules. It is for matrix and array manipulation and operation.

Scipy is a comprehensive numerical library.

Pandas is a package for data processing and analysis, popular but largely because no good substitutes. It is based on R data frame.

Matplotlib is a basic (but rich) library for visualization. Some other complements are Bokeh, Plotly, etc.

Statsmodels is a library for statistical estimation and inference.

Scikit-learn, a Python machine learning package, mostly for supervised learning. Some other machine learning libraries, such as TensorFlow for neural networks.

Numpy

import numpy as np                     # convention to name numpy as np in your codes

x = np.array([[1,2,3],[4,5,6]])
print ('x=',x)
for xi in x:
    print('xi=',xi)
y = x.T
print('y=',y)

x= [[1 2 3]
 [4 5 6]]
xi= [1 2 3]
xi= [4 5 6]
y= [[1 4]
 [2 5]
 [3 6]]

a = np.linspace(-np.pi, np.pi, 10)    # Create even grid from -π to π
print('a=', a)

a= [-3.14159265 -2.44346095 -1.74532925 -1.04719755 -0.34906585  0.34906585
  1.04719755  1.74532925  2.44346095  3.14159265]

np.fromfunction(lambda x, y: x*3 + y + 1, (2,3))

array([[1., 2., 3.],
       [4., 5., 6.]])

Solving a system of linear equations: \begin{align*}3 x_0 + 2 x_1 - x_2 = 9 \ x_0 - 2 x_1 + 0.5x_2 = 2 \ 5 x_0 +0.2x_1 - 2x_2 = 10 \end{align*} In matrix, $A x = b$

from numpy import linalg
A = np.array([[3, 2, -1], [1,-2, 0.5], [5,0.2,-2]])
b = np.array([9,2,10])
x = np.linalg.solve(A, b)
print('x=',x)

x= [3.11428571 1.28571429 2.91428571]

Scipy

Scipy is built on top of Numpy, it includes modules for function interpolation, integration, optimization, linear algebra, Fourier transformation, eignevalues, and more. See here for a complete list.

Example: calculate $ \int_{-0.5}^2 \phi(x) dx $ where $ \phi(x) = \frac{1}{\sqrt{2\pi}} \cdot e^{-x^2/2}$ is the standard normal density function.

from scipy.stats import norm
from scipy.integrate import quad
phi = norm()
y, error = quad(phi.pdf, -1.2, 2)  # Integrate using Gaussian quadrature
print('y=',y)

y= 0.8621801978301126

Matplotlib

Given data in Numpy arrays, plot 2D or 3D figures, can also create animation.

First thing to look at should be the anatomy of a figure

Example: Standard normal distribution.

import numpy as np;
from scipy.stats import norm;
import matplotlib.pyplot as plt;
import seaborn;
seaborn.set(style='ticks');
x = np.linspace(norm.ppf(0.0000001),norm.ppf(0.9999999), 120);
pdfx = norm.pdf(x);
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(14, 6), sharey='all');
ax.plot(x,pdfx, color="green", linewidth=1.6, linestyle="-");
ax.fill_betweenx(pdfx,x,x2=1.96, where = x>1.96,interpolate=True);
plt.axvline(x=0.0, color='black', linewidth=1.4,linestyle='--');
plt.text(2.0,.1,'\n$\\alpha$=shaded area',fontsize=16, horizontalalignment='left');
plt.autoscale(enable=True, axis='x', tight=True);
ax.set_title('Standard normal distribution', fontsize=20);
ax.tick_params(axis='y', labelsize=16);
ax.tick_params(axis='x', labelsize=16);
ax.set_xlim([-4,4]);
ax.set_ylim([0.0,0.4]);
plt.xlabel("z", fontsize=16);
ax.set_ylabel('$\phi(z)$', fontsize=14);
#ax.set_aspect('equal')
#fig.tight_layout();

Pandas

Pandas is not so fast, not so efficient, not so flexible and not so well designed. Somehow, its syntax is confusing to me.

But Pandas has been improving.

For large data set, one should use https://dask.org.

# Example 1: use pandas to create time series
import pandas as pd
np.random.seed(1234)
data = np.random.randn(5, 2)  # 5x2 matrix of N(0, 1) random draws
dates = pd.date_range('28/12/2010', periods=5)
df = pd.DataFrame(data, columns=('price', 'weight'), index=dates)
print(df)

               price    weight
2010-12-28  0.471435 -1.190976
2010-12-29  1.432707 -0.312652
2010-12-30 -0.720589  0.887163
2010-12-31  0.859588 -0.636524
2011-01-01  0.015696 -2.242685


/tmp/ipykernel_40679/2615389923.py:5: UserWarning: Parsing dates in DD/MM/YYYY format when dayfirst=False (the default) was specified. This may lead to inconsistently parsed dates! Specify a format to ensure consistent parsing.
  dates = pd.date_range('28/12/2010', periods=5)

df.mean()

price     0.411768
weight   -0.699135
dtype: float64

# Example 2: read Canadian labor market condition data from CSV form
import pandas as pd
cansimid = '14100287'
filename = 'tbl14100287Final3.csv'
lfs = pd.read_csv(filename, index_col=0)
lfs.index = pd.to_datetime(lfs.index)
lfsQtr = lfs.resample('QS-OCT').mean()
lfsQtr = lfsQtr.round(2)
lfsQtr

#--unemployment rate, raw
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 8), sharey='all')
ax.plot(lfsQtr['unemplRateBothSex'],color="blue", linewidth=2.5, linestyle="-",label='Unemployment rate')
plt.autoscale(enable=True, axis='x', tight=True)
ax.set_title('Unemployment rate, Canada', fontsize=18)
ax.tick_params(axis='x', labelsize=14)
plt.xlabel("\nLate date: 2020 July. Data source: Staistics Canada.", fontsize=14)
ax.set_ylabel('Percentage', fontsize=16)
fig.tight_layout()
filename = 'UnemploymentRate_Canada.png'
plt.savefig(filename, dpi=200, format='png')
plt.show()

Statsmodels

We read in the Mroz’s data on wages from PSID, then do a OLS estimation of the wage equation

import pandas as pd
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
import matplotlib.pyplot as plt

mroz = pd.read_csv("mroz1987.csv")
mroz

# OLS estmation of wage equation for husbands
mroz['ones'] = 1.0
X = mroz.loc[:,['ones','huseduc']]
Y = np.log(mroz['huswage'])
#Y = mroz4['huswage']

model = sm.OLS(Y, X)
results = model.fit()
print(" ")
print(results.summary())

# prediction: results.fittedvalues
Yhat = results.params['ones'] + results.params['huseduc']*X['huseduc']

Yhat2 = results.params['ones']*0.7 + results.params['huseduc']*1.4*X['huseduc']

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                huswage   R-squared:                       0.155
Model:                            OLS   Adj. R-squared:                  0.154
Method:                 Least Squares   F-statistic:                     138.3
Date:                Thu, 12 Jan 2023   Prob (F-statistic):           2.03e-29
Time:                        09:20:08   Log-Likelihood:                -598.39
No. Observations:                 753   AIC:                             1201.
Df Residuals:                     751   BIC:                             1210.
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
ones           0.9105      0.083     10.943      0.000       0.747       1.074
huseduc        0.0761      0.006     11.758      0.000       0.063       0.089
==============================================================================
Omnibus:                      144.321   Durbin-Watson:                   1.890
Prob(Omnibus):                  0.000   Jarque-Bera (JB):              371.275
Skew:                          -0.986   Prob(JB):                     2.39e-81
Kurtosis:                       5.819   Cond. No.                         55.0
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Networks and Graphs

Disclaimer: This subsection is credited to quantecon.org

Python has many libraries for studying graphs.

One well-known example is NetworkX. Its features include, among many other things:

• standard graph algorithms for analyzing networks
• plotting routines

Here’s some example code that generates and plots a random graph, with node color determined by shortest path length from a central node.

%matplotlib inline
import networkx as nx
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (10,6)
np.random.seed(1234)

# Generate a random graph
p = dict((i, (np.random.uniform(0, 1), np.random.uniform(0, 1)))
         for i in range(200))
g = nx.random_geometric_graph(200, 0.12, pos=p)
pos = nx.get_node_attributes(g, 'pos')

# Find node nearest the center point (0.5, 0.5)
dists = [(x - 0.5)**2 + (y - 0.5)**2 for x, y in list(pos.values())]
ncenter = np.argmin(dists)

# Plot graph, coloring by path length from central node
p = nx.single_source_shortest_path_length(g, ncenter)
plt.figure()
nx.draw_networkx_edges(g, pos, alpha=0.4)
nx.draw_networkx_nodes(g,
                       pos,
                       nodelist=list(p.keys()),
                       node_size=120, alpha=0.5,
                       node_color=list(p.values()),
                       cmap=plt.cm.jet_r)
plt.show()