The System Administrator Notebook

Pages in the Python section

Built-In Objects Functions Python Idioms Django Anaconda

Notes on Programming in Python

Page Contents

Acknowledgement

Top Bottom

Python for Everybody: Exploring Data in Python, Charles Severance

Hello World

Top Bottom

The standard first program in Python:

print('Hello World!')

A more interactive version:

x = 'Hello '
name = input('Enter your name: ')
print(x + name + '!')

Getting Help

Top Bottom

Use the 'dir' function to list the methods of an object and help for the method definition:

#!/usr/bin/python3
#method_lookup.py 

print(dir(str))
print(help(str.casefold))

From a command shell you can use pydoc to explore python objects and libraries.

On-line documentaion for Python can be found here

Comparison Operators

Top Bottom

Comparison operators return a boolean value

#!/usr/bin/python3
#comparisons.py 

# True and False are booleans
print(type(True))
print(type(False))

x = 51
y = 43
z =  8
w = x

if x == y:
    print('x equals y')
else: 
    print('x does not equal y')

if x != y:
    print('x does not equal y')
else:
    print('x does equal y')

if x > y + z:
    print('51 is greater then 43 + 8')
elif x < y + x:
    print('51 is less than 43 + 8')
else:
    print('51 equals 43 + 8')

if x is not y + z:
    print('x is not the same as y plus z')
else:
    print('x is the same as y plus z')

if x is w:
    print('x is the same as w')

if True == False:
    pass    # this statement does nothing
else:
    print('True does not equal False')

if True is False:
    pass
else:
    print('True is not the same as False')

if True is False or False is True or True is True:
    print('True is False or False is True or True is True')

if True is not False and False is not True:
   print('True is not False and False is not True')

Note that 'is' compares whether two variables refer to the same object: Strings with the same value refer to the same object, but arrays with the same list of elements do not. If however you assign one list to another, then both lists will refer to the same object:

#!/usr/bin/python3

m = 5
n = 5

print(m is n) # True

a = 'Hello World'
b = 'Hello World' 

print(a is b)   # True

x = ['Hello', 'World']
y = ['Hello', 'World']

print(x is y)   # False

y = x

print(x is y) # True

Catching Errors

Top Bottom

Use Try:Except to catch errors:

#!/usr/bin/python3
prompt = 'Please enter a number:\n'
response = input(prompt)
print(type(response))

try:
    response = float(response)
    print('Converted')
    print(type(response))
except:
    print('Your response could not be converted to a float')
    print('Not converted')
    print(type(response))

Type Conversion

Top Bottom

Python provides built-in functions for type conversion:

#!/usr/bin/python3
import random

num = random.random()
print(num)
print(type(num))
my_integer = int(num)
print(type(my_integer))
my_string = str(num)
print(type(my_string))

my_int = random.randint(60, 90)
print(my_int)
my_float = float(my_int)
print(my_float)

my_choice = ['34', '56', '39', '23']
print(int(random.choice(my_choice)))

Function Definition

Top Bottom

Functions are defined with the 'def' keyword, and need to be defined in the script before they are called. Parentheses in the function name can be used to define function parameters. The logic of the function is defined in the indented lines following the line containing the function header. The indented lines are called the 'function body'. The function definition is terminated by the first non-indented line following the funtion body

#!/usr/bin/python3
import math

def get_diameter(radius):
    return radius * 2

def get_circumference(radius):
    return get_diameter(radius) * math.pi


print(get_circumference(3))

If you pass a list as a parameter the function gets a reference to the list and any changes made to the list in the function will be reflected in the list outside of the function. However be careful when trying to change a list using functions that do not change the list but instead return a new list:

#!/usr/bin/python3 

a = [1,2,3]

def remove_one(a):
    #del modifies the list
    del a[0]

remove_one(a)
print(a) # [2, 3]

def add_more(a):
    a = a + [4]
    # + returns a new list without changing the original
    print('Inside the funcion a is now: ')
    print(a) # [2, 3, 4]

add_more(a)
print('Outside the function a is still: ')
print(a) # [2, 3]

If you wish to capture the list value from the add_more() function, then you will need to include a return statement in the function and call the function with an assignment:

#!/usr/bin/python3 

a = [1,2,3]

def add_more(a):
    a = a + [4]
    return a

a = add_more(a)
print(a) # [1, 2, 3, 4]

Iteration

Top Bottom

The while statement can be used to iterate a series of statements while the test condition evaluates to 'True'

#!/usr/bin/python3

def get_factorial(base):
    factorial = 1
    try:
        base = int(base)
    except:
        return 'Not an integer: bailing out!'

    while base > 1:
        if factorial == 1:
            factorial = base
        else:
            factorial = factorial * base
        base = base - 1

    return factorial


while True: 
    base = input('Enter an integer or "Done" to Finish: ')
    if base == 'Done':
        break
    if base == '':
        continue
    factorial = get_factorial(base)
    print('Factorial for ' + str(base) + ' is: ' + str(factorial))

A 'break' statement can be included to terminate the loop: a 'continue' statement can be used to terminate the current iteration, and skip to the next iteration.

'For' loops can be used to iterate through the members of a list, and also support 'break' and 'continue' operators

#!/usr/bin/python3

bands = ['Coldplay', 'Travis', 'Bucks Fizz', 'Burning Spear']

for band in bands:
    if band == 'Bucks Fizz':
        continue
    print('Cue ' + band)

Regular Expressions

Top Bottom

Python supports regular expressions via the 're' library. 're' provides a number of functions:

re.search('regular_expression', string)
returns True if match found
re.findall('regurlar_expression', string
returns a list of matches. Parenthesees in the regular_expression, allow results to be captured.

Files

Top Bottom

Open a file in python using the 'open() function:

fhand = open('document.txt', 'r')

Assign each line of the file as an element of an array:

lines = fhand.readlines()

Use re library to skip blank lines, then remove trailing whitespace and lower-case the elements with:

import re
clean_lines = [ line.strip().lower() for line in open('file.txt', 'r') if re.search('\w', line) ]

If you just want to look at the first ten lines in a file, in interactive mode try:

open('not_like_this.txt', 'r').readlines()[:10]

If you have a data file with columns separated by whitespace you can load the data into a python dictionary:

def findrecord(string):
    found = { code : record for code, record in recordset.items() if string in record }
    print(found)

recordset = dict()
for line in open('data.txt', 'r'):
    if 'current' in line:
        fields = line.split()
        recordset[fields[0]] = fields[1:]

print("Found %i records" % len(recordset))

The findrecord function can be used to search for records loaded into the dictionary

The split function will split on whitespace by default or you can specify a delimiter. If you are processing CSV files, then the csv module will help with some of the vagaries of the CSV 'standard'

import csv

with open('Prescriptions.csv', newline='') as csvfile:
    prescriptions = csv.reader(csvfile, delimiter=':', quotechar='"')
        for row in prescriptions:
            print(', '.join(row))

The 'newline' option specified in open will enable the filehandle to correctly interpret newlines embedded in quoted text

The csv library also provides a 'DictReader' method, that will load the rows into a dictionary. The key names can be specified in the constructor or it will use the first line of the file to populate the key names:

import csv

with open('Prescriptions.csv', newline='', encoding='utf-8') as csvfile:
    prescriptions = csv.DictReader(csvfile)
    for prescription in prescriptions:
        print(prescription['start date'], prescription['drug name'])

Networking

Top Bottom

Python provides support for network connections via the sockets library.

#!/usr/bin/python3

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('py4inf.com', 80))

request = "GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\nHost: www.py4inf.com\n\n"
s.send(request.encode('utf8'))

while True:
    data = s.recv(512)
    if ( len(data) < 1):
        break
    print(data.decode('utf8'))

s.close()

The 'urllib' library makes retrieving http requests much simpler:

#!/usr/bin/python3
import urllib.request, urllib.parse, urllib.error
import re

response = urllib.request.urlopen('http://python.org/')

html = response.read()

links = re.findall(b'href="(http://.*?)"', html)

for link in links:
    print(link.decode())

urllib can also be used for FTP requests:

urllib.request.urlretrieve('ftp://ftp.example.com/readme.md', 'readme.md')

The BeautifulSoup library makes processing HTML simpler:

#!/usr/bin/python3

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
html = urllib.request.urlopen(url, context=ctx).read()

soup = BeautifulSoup(html, 'html.parser')

tags = soup('a')
for tag in tags:
    print('TAG:', tag)
    print('URL:', tag.get('href', None))
    print('Contents:', tag.contents[0])
    print('Attrs:', tag.attrs)

XML Parsing

Top Bottom

The xml library contains functions for processing XML Data:

#!/usr/bin/python3

import xml.etree.ElementTree as ET

input = '''
<COSDRecord>
  <id root="500"/>
  <Lung>
    <LungCore>
      <LungCoreLinkagePatientId>
        <NHSNumber extension='1112223333'/>
        <LocalPatientId><![CDATA[AAA111222]]></LocalPatientId>
        <NHSNumberStatusIndicator code="02"/>
        <Birthdate>2012-05-01</Birthdate>
      </LungCoreLinkagePatientId>
      <LungCoreLinkagePatientId>
        <NHSNumber extension='9998887777'/>
        <LocalPatientId><![CDATA[ZZZ999888]]></LocalPatientId>
        <NHSNumberStatusIndicator code="03"/>
        <Birthdate>1982-10-11</Birthdate>
      </LungCoreLinkagePatientId>
    </LungCore>
  </Lung>
</COSDRecord>'''

cr = ET.fromstring(input)
pt_link = cr.findall('Lung/LungCore/LungCoreLinkagePatientId')
print('Record Count: ', len(pt_link))

for link in pt_link:
    print('NHS Number: ', link.find('NHSNumber').get("extension"))
    print('Local Patient Id: ', link.find('LocalPatientId').text)
    print('NHS Number Status Indicator: ', link.find('NHSNumberStatusIndicator').get("code"))
    print('Birth Date: ', link.find('Birthdate').text)

JSON Parsing

Top Bottom

The json library contains functions for processing JSON Data:

#!/usr/bin/python3

import json

data = '''
[
    {   "type"   : "Prescription",
        "issuer" : "Pharma Corp",
        "id" :
            "4227c2cb-8f0c-49b6-bd93-345b6174324b",
        "assigner" : "ACME Inc"
    },
    {   "type"   : "Blood Exam",
        "issuer" : "Path Labs",
        "id" :
            "5481b3af-8f0c-49b6-bd93-345b6174324b",
        "assigner" : "ACME Inc"
    }
]'''

info = json.loads(data)
print('Event Count: ', len(info))

for event in info:
    print('Type: ', event['type'])
    print('Issuer: ', event['issuer'])
    print('ID: ', event['id'])
    print('Assigned by: ', event['assigner'])

API Access

Top Bottom

XML or JSON data can be combined with urllib to request data from third party applications following their published APIs

#!/usr/bin/python3

import urllib.request, urllib.parse, urllib.error
import json

serviceurl = 'http://localhost:8081/rest/v1/ehr'

while True:
    ehr_id = input('Enter EHR Id: ')
    if len(ehr_id) < 1: break

    url = serviceurl + urllib.parse.urlencode({'ehrid': ehr_id})

    print('Retrieving ', url)
    response = urllib.request.urlopen(url)
    data = response.read().decode()
    print('Retrieved ', len(data), ' characters')

    try:
        js = json.loads(date)
    except
        js = None

    if not js or 'status' not in js or js['status'] != OK:
        print('Error retrieving data')
        print(data)
        continue

    print(json.dump(js, indent=4))

Object Oriented Python

Top Bottom

The 'class' keyword is used to define a class in Python. We can store class definitions in an external file:

#!/usr/bin/python3

class Robot:
    obeyed = 0
    name = ''

    def __init__(self, name):
        self.name = name
        print('Robot %s created' % self.name)

    def obey(self): 
        self.obeyed = self.obeyed + 1
        print('(%s) Yes, master' % self.name)

    def __del__(self):
        print('Robot %s Scrapped. Obeyed %d commands' % (self.name, self.obeyed))

class CookBot(Robot):
    meals = 0

    def cook(self):
        self.meals = self.meals + 1
        self.obey()
        print('%s: Supper is served' % self.name)

This file defines two classes. The Robot class has two attributes: name and obeyed. Three methods are also defined: the '__init__' method is called at object construction. In the Robot class it sets the name attribute from the parameter provided at construction. The '__del__' method is called at object destruction. The final method of the Robot calls is obey() which simply increments the 'obeyed' attribute and prints a helpful message.

The CookBot class is defined as a subclass of Robot and therefore inherits all the properties of Robot, and defines its own additional method and attributes: cook() and meals.

We can use the classes defined in this file by importing the classes from the file:

#!/usr/bin/python3

from robot import Robot, CookBot

an = Robot('Robbie')
cb = CookBot('Jamie')
an.obey()
cb.obey()
cb.cook()
cb.obey()
cb.cook()
an.obey()

Comprehensions

Top Bottom

A comprehension can be used to iterate over an iterable object. For instance, an array of arrays can be configured to set up a matrix, and a list comprehension can be used to iterate over the matrix:

#!/usr/bin/env python3 
#comprehension.py

matrix = [ [ 1, 2, 3], [4, 5, 6], [7, 8, 9] ]

print(matrix[1][1])

middles = [row [1] for row in matrix]
print(middles)

double_middles = [row [1] * 2 for row in matrix]
print(double_middles)

evens = [row [1] for row in matrix if row[1] % 2 == 0]
print(evens)



The comprehension consists of an expression row[1] and a looping construct for row in matrix. Because the comprehension is enclosed in square brackets, the return value is a list. But comprehensions can also be used to return sets, dictionaries or generators:

#!/usr/bin/env python3
# generators.py

matrix = [ ['first', 1, 2, 3], ['secomd', 4, 5, 6], ['third', 7, 8, 9] ]

end_array = [ row[-1] for row in matrix ]
end_set = { row[-1] for row in matrix }
dict1 = { row[0] : row[1] for row in matrix }

generation = ( sum(row[1:]) for row in matrix )

print(next(generation))
print(next(generation))
print(next(generation))

Note the the generator is created by enclosing the comprehension in round brackets

You can use a comprehension to quickly transpose a dictionary:

dict1 = {
    'west ham' : 'upton park', 
    'chelsea' : 'stamford bridge', 
    'tottenham hotspur' : 'white hart lane'
}

dict2 = { dict1[key] : key for key in dict1 }

dict3 = { value : key for key, value in dict1.items() }

ORM Mapper

Top Bottom

SQL Alchemy provides ORM functionality to interface Python with Relational Databases.