Thursday, February 08, 2018

Python Important Tips

It is very much to forget the simple python programming tips. Here the blog to remember.

Collection manipulation tips

How to concatinate to to tuple

t = 'ABC', 24
t = t + ('Sydeny',)

In the second line , is the important character in the above code.

You can repeate the tuple:

t * 3
#('ABC', 24, 'Sydeny', 'ABC', 24, 'Sydeny', 'ABC', 24, 'Sydeny')

this is the simple and not need to mentions

#simple list
a = [1,2,3,4,5]
print a[1:3]
#[2, 3]

unpacking the data structure

letters = ('A', 'B'), 'a','b'
(l1,l2),l3,l4 = letters    
print (l1,l2,l3,l4) 

unpack the dictionary

d = {'a':1, 'b':2, 'c':3}
(k1,v1), (k2, v2), (k3,v3) = d.items() # ('c', 3)

Order is not gurranteed.

Create own Iterator

Two methods are mandatory:

  • __iter__() function and
  • __next__() in python 3 but in next() in pytho 2.

For example:

class MyIter:
    def __init__(self, *elements):
        self.elements = elements

    def __iter__(self):
        self.i = 0
        return self

    def next(self):
        if len(self.elements) > self.i:
            self.i = self.i + 1
            return self.elements[self.i-1]
        else:
            raise StopIteration('My iterator cannot continue because iterator reached to the end of elements: {}'.format(self.i))


test = MyIter(1,2,3,4)

for e in test:
    print(e)

Create the above iterator using yield :

def mygen():
    yield 'a'
    yield 'b'
    yield 'c'

g = mygen()

# similar effect
next(g)
g.send(None)

# support the for loop
for e in g:
    print(e)

Lazy geneorator example:

def add_series():
    l =[0,0]
    while True:
        l[0] = l[0] + 1
        l[1] = sum(l)
        yield l[-1]

for i, n in enumerate(add_series()):
    if i == 6:
        break
    print(n)

# 1
# 3
# 6
# 10
# 15
# 21

To stop the loop you need to break inside the for.

Generators can hold the state:

def keepval():
    val = []
    while True:
        data = yield
        val.append(data)
        yield val

g = keepval()
g.send(None)
g.send('a')  # ['a']
g.send(None) 
g.send('b')  # ['a', 'b']
g.send(None)
g.send('c')  # ['a', 'b', 'c']

Defauts

Default for dictionary:

from collections import defaultdict

p = {'a':1, 'b':2, 'c':3}
d = defaultdict(lambda: 0)
d.update(p)

d['a'] # 1
d['d'] # 0

or use the following way:

from collections import defaultdict
p = {'a':1, 'b':2, 'c':3}
l = lambda : 0
resilt = defaultdict(l, p)

result['a'] # 1
resilt['c'] # 0

You can create an incrementer for default using class because class methods are still first class functions in python:

from collections import defaultdict
class Counter(object):
    def __init__(self):
        self.c =0
    def __call__(self):
        self.c += 1
        return self.c

counter = Counter()
callable(counter)
d = defaultdict(counter)
d.update({'a':1,'b':2})

for i in list('abcdefgh'):
    d[i]

print(d) # defaultdict(<__main__.Counter object at 0x111768c90>, {'a': 1, 'c': 1, 'b': 2, 'e': 2, 'd': 3, 'g': 5, 'f': 4, 'h': 6})

In the above code __call__() is the callable methed.

Blend with zip

How to create dictionary from List of tuples

l = 'a','b','c'
p = 1,2,3

#blend dict ([('a',1),('b',2), ('c',3)])
dict(zip(l,p)) #{'a': 1, 'b': 2, 'c': 3}

Generator expression

For example:

p = 1,2,3

g = (i*i for i in p)
next(g) #1
next(g) #4

Here the use of add_series() generator in the generator expression:

def stop():
    raise StopIteration()

g = (i for i in add_series() if i < 6 or stop() )
[i for i in g] #[1, 3]

Use of itertool

Following will breake the it.count() to have all the number below 10.

# import itertools as it
# for i in it.count():
#     if i > 10:
#         break
#     print(i)
for i in it.takewhile(lambda x: x < 10, it.count()):
    print (i)

Group by odd and even numbers

isEven =  lambda x : not x % 2
# sorted(range(10), key=isEven) => [1, 3, 5, 7, 9, 0, 2, 4, 6, 8]

for e, g in it.groupby(sorted(range(10), key=isEven), key=isEven):
    print(e, list(g))
# => 
# (False, [1, 3, 5, 7, 9])
# (True, [0, 2, 4, 6, 8]) 

Cartician Product

for x, y in it.product(['a','b','c'], [1,2,3]):
    print (x, y)

# ('a', 1)
# ('a', 2)
# ('a', 3)
# ('b', 1)
# ('b', 2)
# ('b', 3)
# ('c', 1)
# ('c', 2)
# ('c', 3) 

Permutations as follows:

for i in it.permutations(['a','b','c']):
    print(i)

# ('a', 'b', 'c')
# ('a', 'c', 'b')
# ('b', 'a', 'c')
# ('b', 'c', 'a')
# ('c', 'a', 'b')
# ('c', 'b', 'a')   

for i in it.permutations(['a','b','c'], r=2):
    print(i)

# ('a', 'b')
# ('a', 'c')
# ('b', 'a')
# ('b', 'c')
# ('c', 'a')
# ('c', 'b')    

Combinations as follows:

for i in it.combinations(['a','b','c'], r = 2):
    print(i)

('a', 'b')
('a', 'c')
('b', 'c')  

functiontools

Here the partial function

import functools as ft

def double(x):
    return x * x

d = ft.partial(double, 3)
d() #9

To add any number of values using partial functools :

def add(*n):
    return sum(n)

# same as bellow partial
# add(1,2,3,4) # => 10

#---- use of partial

from functools import partial

one = partial(add, 1)
two = partial(one,2)
three = partial(two,3)
three(4) # => 10

Decorator

Decorator wrap the function to decorate such as validate input:

def decor(func):
    print('decoreate')
    return func

@decor
def hello(name):
    return name

print(hello('test'))

If you need to pass the argument to decorator

def outer_decor(arg):
    def inner_decor(func):
        if arg:
            print(arg)
        print('decoreate')
        return func
    return inner_decor

@outer_decor(arg='outer')
def hello(name):
    return name

print(hello('test'))

# outer
# decoreate
# test

pass dictionary as an argument list:

def unpack(*val):
    for k ,v in val:
        print('{} => {}'.format(k,v))

unpack(*{'a':1,'b':2,'c':3}.items())

# by the way this is an unordered.
# a => 1
# c => 3
# b => 2

Class Comparison

Python classes can be compared directly like c1 >= c2 implementing the method __ge__(). However, for all the comparisions, there is deidicated method to implement in the class level. The functools.total_ordering decorator can be used to simplify this process.

'''
    This examples shows the hospital domain where wards and patients. Dynamically
    wards can be filled by patients.
    Used the total_ordering decorator and __eq__ and __lt__ minimum requirments
    to implement the comparisons for the ==, >=, <=  and so on.
'''
from functools import total_ordering

class Patient(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age

@total_ordering
class Ward(object):
    def __init__(self, ward_no):
        self.ward_no = ward_no
        self.patients = list()

    def add_patient(self, patient):
        self.patients.append(patient)

    def __str__(self):
        return 'Ward {} has {} patients'.format(self.ward_no, len(self.patients))

    #current capacity is decide by the number of patients in the ward
    @property
    def capacity(self):
        return len(self.patients)

    #minimum required functions
    def __eq__(self, other):
        return self.capacity == other.capacity

    def __lt__(self, other):
        return self.capacity < other.capacity

# This is ward no# 1
ward_1 = Ward(1)
ward_1.add_patient(Patient('A',25)) # add patients to wards
ward_1.add_patient(Patient('B',26))
ward_1.add_patient(Patient('C',27))
ward_1.add_patient(Patient('D',28))

# ward no# 2
ward_2 = Ward(2)
ward_2.add_patient(Patient('X',25)) # add patients to wards
ward_2.add_patient(Patient('Y',26))
ward_2.add_patient(Patient('Z',27))

# As you see, can do all the comparisons
ward_1 < ward_2 # False
ward_1 <= ward_2 # False
ward_1 == ward_2 # False
ward_1 >= ward_2 # True
ward_1 > ward_2 # True

The last bit of the above python program shows the use of all the comparison operators.

Maybe Decorator

This is one of the good pattern found in the Functional Programming in Python by Sebastiaan Mathot.

add_str = lambda s: sum([int(i) for i in s.split('+')])
add_str('1+2') #-> 3

def lambda_exception_wrapper(func):
    def inner(*args):
        for a in args:
            if isinstance(a, Exception):
                return a

        try:
            return func(*args)
        except Exception as e:
            return e.args    

    return inner

safe_add_str = lambda_exception_wrapper(add_str)
safe_add_str('1+2')
#-> 3
safe_add_str(1+3) 
#-> ("'int' object has no attribute 'split'",)

Memoization

This is a functional caching mechanism. You can achive this desing pattern using decorators.

def decor(func):
    print('decoreate')
    cache = {}
    def inner(*args):
        if args in cache:
            print('found')
            return cache[args]
        print('not found')
        cache[args] =func(*args)
        return cache[args]
    return inner

@decor
def hello(*n):
    return sum(n)

print(hello(*[i for i in range(100)])) # =>
# not found
# 4950

Curring

Here the use of decorator and the Curring desing pattern to add numbers explained under the functools section.

from functools import partial
import inspect

def curry(func):

    def inner(arg):
        print(len(inspect.getargspec(func)))
        if len(inspect.getargspec(func).args) == 1:
            return func(arg)
        print('-> {}'.format(arg))
        return curry(partial(func,arg))

    return inner

@curry
def add(a,b,c):
    return a + b +c

Python Lambda calculus

As a first example see the identity lambda function

In Python this is written as lambda x: x.

The simplest application can be created by applying variable y to above identity function

right arrow shows the reduction with the result. In python, you can write this as (lambda x: x)(y) where ycan be any value.

It is complete valid expression to write as follows using identity function:

For example in the python you can test the validity of using identity function by (lambda x: x)(lambda y: y * y)(2).

Consider the substitution:

In the python you can test this as (lambda x: lambda t:[x,t])(3)(2) is the same result of (lambda x, t : [x, t])(1,2).

For example to find the even numbers in the list of numbers in lambda calculus is :

Python

For example, the simplest from of lambda in the Python (lambda x: x+1)(3) which returns result 4. Mathamatically this can be written as

Lambda can be nested as follows

(lambda x: lambda y:lambda z: x + y * z)(5)(2)(3) #11

As explain above, if you implements the even lambda function:

even = lambda x : x % 2 == 0

#For example, use the built in function to go through the list
filter(even, [1,2,3,4,5]) 
#[2, 4]

Above bulit in filter method can be written as :

def myfilter(a,b):
    """
    param: a : condition in lambda
    param: b : list of ints
    """
    if len(b) > 0:
        if a(b[0]):
            return [b[0]] + myfilter(a,b[1::])
        else:
            return myfilter(a,b[1::])
    else:
        return []

#for example:
myfilter(lambda x : x % 2 == 0, [1,2,3,4,6,10]) #[2, 4, 6, 10]

If you are still uncomfortable with the above myfilter method, here the simple example to built in sum. Let’s create sumlist function in functional programming:

def sumlist(l):
    if len(l)>1:
        return l[0] + sumlist(l[1::])
    return l[0]

#example
sumlist([1,2,3,4,5,6,7]) #28

The built in reduce function can be written as reduceints:

c = lambda x: lambda y: x + y
#Let's test the lambda function 
c(3)(c(2)(c(1)(0))) # result = 6

def reduceints(a,b):
    """
    param: a : lambda calculation
    param: b : list of ints
    """
    if len(b) > 2:
        return a(b[0])(myreduce(a,b[1::]))
    else:
        return a(b[0])(b[1])    

# For example    
reduceints(c, [1,2,3]) # result = 6       

Factorial can be written in lambda as follows:

f = lambda n: 1 if n == 0 else n * f(n-1)
f(3) # 6

In the above example lambda expression need to be assign to variable because factorial is recursion based and need a name to refer itself.

if expression in the lambda:

(lambda a: 'good' if a > 10 else 'ok' if a > 5 else 'bad')(4) # 'bad'

Closures

Here the example of incrementor

def count(val):
    c = [val]
    def inc(n):
        c[0] = c[0] + n
        return c[0]
    return inc

f = count(2)
f(2)
f(2)

List operations

Length example

def length (list):
    tail = list[1::]
    return 1 + length(tail) if tail != [] else 1

length([1,2,3,4,5,6,7,8,9])

element at operation

def elementAt(list,n):
    if (n> 1):
        return elementAt(list[1::], n-1)
    else:
        return list[0]

elementAt([1,2,3,4,5,6,7,8,9], 8)

reverse

def reverse(list):
    h = list[0]
    tail = list[1::]
    return reverse(tail)+[h] if (tail != []) else [h]

reverse([1,2,3])

The replacement to the above code in the parctical python is list[::-1] which will create copy of the list in the reverse order. Note: to copy the list l2 = l1[:].

Here the quick sort algo

def qsort(list):
    if (list != []):
        h = list[0]
        print('-- head ---: {}'.format(h))
        tail = list[1::]
        print('tail => {}'.format(tail))
        left = filter((lambda x: x < h),tail)
        print('left => {}'.format(left))
        right = filter((lambda x: x > h),tail)
        print('right => {}'.format(right))
        return qsort(left)+[h]+qsort(right)
    else:
        return  []

# Example
qsort([5,7,6,0,4])

contextlib

This is the item 7 got from the Effective Python by Brett SlatkinPublished by Addison-Wesley Professional, 2015.

import logging
from contextlib import contextmanager

@contextmanager
def debug_logging(level): 
    logger = logging.getLogger()
    old_level = logger.getEffectiveLevel()
    logger.setLevel(level)
    try:
        yield #immidely start to run withing the with block
    finally:
        logger.setLevel(old_level)

Uisng with statement you can run any function where logging.<level>('.....') set as follows:

with debug_logging(logging.DEBUG):
    my_function()

Above context propagation override the default degugging level as well.

Another good example is use of contextlib use for the try/catch:

from contextlib import contextmanager
import logging

@contextmanager
def my_exception(cls):
    try:
        yield
    except cls:
        logging.exception("my Exception")

v = 10
with my_exception(ZeroDivisionError):
    v /=0
#above excetion handled, therefore continue with next line    
print('continue ...............')

Because expection is handle within the my_exception, next line will be executed and program continue without stopping.

Sort with priority

In the following example, priority must go to the gorup.

num = [8,3,1,2,5,4,7,6]
group = {3,7,5,4}

def helper(x):
    print(x)
    if x in group:
        return (0,x)
    return (1,x)

num.sort(key=helper)
print(num) # [3, 4, 5, 7, 1, 2, 6, 8]

As you see group elements are in the head of the list in sorted order and the sorted element of the the rest of the list.

If you use the class, there the code for the above:

class SortHelper (object):
    def __init__(self, group):
        self.group = group

    def __call__(self, x):
        if x in self.group:
            return (0,x)
        return (1,x)

helper =  SortHelper(group)
num.sort(key=helper) # [3, 4, 5, 7, 1, 2, 6, 8]

In the above code, the helper method is __call__().

References:

Effective Python by Brett Slatkin Published by Addison-Wesley Professional, 2015

Written with StackEdit.

Thursday, November 16, 2017

Python Defensive Iteration

This is the item 12 explained in the “Effective Python”, by Brett Slatkin. Generators are the best save of out of memory problem. Here the problem,

class StringToList(object):
    def __init__(self, text):
        self.text = text

    def __iter__(self):
        for x in self.text:
            yield x

words = StringToList('Hello')

# case 1
it = iter(words)
it2 =iter(it)
print(it, it2) #(<generator object __iter__ at 0x10816d7d0>, <generator object __iter__ at 0x10816d7d0>)
next(it) # 'H'
next(it2) # 'e'

The problem is it and it2 pointing to the same instance and iteration is not as expected (‘it2’ give a next element as ‘e’ instead ‘H’). To overcome this problem, author has suggested the following solution which is applicable to set and dict as well. In the case 1, same container is used. But in the case 2, different containsers.

# case 2
it = iter(words)
it2 =iter(words)
print(it, it2) # (<generator object __iter__ at 0x10816d550>, <generator object __iter__ at 0x10816d320>)
next(it) # 'H'
next(it2) # 'H'

In the above code, diffrent instaces because two diffrent containers. To defence from the case 1, need to use the following defence and avoid:

if it is it2:
    raise TypeError('Same container error')
else:
    print(next(it)) # 'H'
    print(next(it2)) # 'H'

That will promote the case 2 style of programming.

Monday, August 28, 2017

PyPI hosting on AWS S3

Create a web hosting in the AWS S3

It is a common task to create a web hosting in the S3 bucket. However, you have to change the permission to access the bucket via http protocol. For that you can right click the dist folder and select the Make public from the drop down menu.

Create Distribution package

Create a directory for distribution package source for example ojservice. Your folder structure is as follows:

ojsevice+
        |
        +index.html
        |
        +error.html
        |
        +LICENSE.txt
        |
        +README.txt
        |
        +setup.py
        |
        +ojservice+
                  |
                  +__init__.py
                  |
                  +helloservice.py

In the above structure, index.html and error.html are two files given in the AWS S3 host configuration under the bucket’s Static web hosting : respectively index and error documents.

Here the helloservice.py:

def hello(name):
    return 'Hello {}'.format(name)

In addition to that you have to install the package pip2pi

pip install pip2pi

Create setup.py file for the distribution:

from distutils.core import setup

setup(
    name='ojservice',
    version='0.1',
    packages=['ojservice',],
    license='Coreative Commons Attribution-Noncommercial-Share Alike license',
    long_description=open('README.txt').read(),
)

Create Distribution package which target is dist folder under the parent directory

python setup.py sdist

run the following command to create the package hierarchy:

dir2pi dist

Sync the folder to the s3 bucket for example: ojpy

aws s3 sync dist/simple s3://ojpy

Install the package using pip

Setup the client application to import and test the pip distribution package. Virtual environment is the best.

mkdir ojtest
cd ojtest

#setup virtual env
virtualenv ~/.virtualenvs/ojtest

#activate the enviroment
source ~/.virtualenvs/ojtest/bin/activate

Here the command

pip install --index-url=http://<s3 host>/ojservice ojservice==0.1 --trusted-host <s3 host>

Here the example

pip install --index-url=http://ojpy.s3-website-ap-southeast-2.amazonaws.com/ojservice/ ojservice==0.1 --trusted-host ojpy.s3-website-ap-southeast-2.amazonaws.com

Here the code snippet to import from the distributed package

from ojservice import helloservice
print(helloservice.hello('OJ'))