Posts

Showing posts from 2016

Pandas Land

Introduction This is Panda exercises: Data Frames In this section, I explore how to create data frames from different ways: list dictionary Json csv file In addition to that basic Data Frame (DF) manipulations: import pandas as pd cols = { 'name' : [ 'Ted' , 'Mak' , 'Nina' , 'Leo' ] , 'age' : [ 50 , 20 , 33 , 25 ] } l_students= [{ 'name' : 'Tailor' , 'grade' : '10' , 'math' : 60 } ,{ 'name' : 'Lora' , 'grade' : '09' , 'math' : 80 } ,{ 'name' : 'Joe' , 'grade' : '11' , 'math' : 56.90 } ,{ 'name' : 'Tailor' , 'grade' : '11' , 'math' : 68.98 } ] studemtDF = pd.DataFrame(l_students) # read from the json import json json_students = json.dumps(l_students) # [ # { # "grade": "10", #

Atom as Spark Editor

Image
Recently I started to reach to integrate Atom editor with Spark pyspark. In addition to the Atom, I found how to integrate pyspark with IntelliJ Idea which I suppose to discuss later. The nice thin about Atom is hydrogen plugin which you can use for inline evaluation with python. Here the steps Install Spark Install Atom Install hydrogen plugin to atom most important to set the PYTHONPATH as follows export PYTHONPATH = /<SPARK_HOME>/python :/<SPARK_HOME>/python/lib/py4j- 0 . 9 -src.zip Now run the following code to verify. Here the testing code. from pyspark import SparkContext from pyspark import SQLContext sc = SparkContext() sqlContext = SQLContext(sc) df = sqlContext.createDataFrame([( "Ojitha" , "Kumanayaka" ),( "Mark" , "Anthony" )],( "first_name" , "last_name" )) df.show()    

Algorithm Analysis

Algorithm Analysis measure the speed of the task algorithm suppose to accomplished. Here the necessary maths back ground to analyse the algorithmic complexity. CONTENTS 12 -fold way Order of the Growth Algorithm Analysis of Sorting Algorithm Analysis of Searching Appendix There are two basic rules in the Algorithm analysis: Rule of sum: if an action is performed making A choices or B choices, then it can be performed \(A+B\) ways. Rule of product: If an action can be performed by making A choices followed by B choices, then it can be performed \(A\times B\) ways. Permutation: an ordered list where every object appears exactly once \[ _{n}P_{r} = \frac {n!} {(n - r)!} \text{ where } !0 = 1 \] combinations: When order is not important and the repetition is not allowed, the number of ways to choose k from the distinct n is as for \(n \ge k \ge 0\): \[ {n \choose k }= \frac {n!}{ k!(n-k)! } \] in general, choosing k out of n is same as not cho

Recommender System Notes.

These are the notes I've achieved certificate from the well known Recommender System course .  

Spring 3 Part 9: Spring Security LDAP integration

The expectation of this blog is to enable developer to start LDAP integration quickly and easily with Spring Security. I have used ApacheDS which is embedded in the Apache Studio. Here the steps: The DN dc=example,dc=com is a example domain controller you can easily follow the documentation. Here the ldif document #[1] create domain (distinguished name) dn: dc=example,dc=com objectclass: top objectclass: domain dc: example #[ 2 ] create people organizational unit dn: ou=people,dc=example,dc=com objectClass: organizationalUnit objectClass: top ou: people #[ 3 ] create a user dn: uid=ojitha,ou=people,dc=example,dc=com objectClass: organizationalPerson objectClass: person objectClass: inetOrgPerson objectClass: top cn: Ojitha Hewa sn: Hewa uid: ojitha #[ 4 ] create group dn: ou=groups,dc=example,dc=com

JavaScript Notes

Coming from the Java background, my main problem with JavaScript is encapsulation: global space is not good. I love OO because object is the best place to encapsulate the data. Here are the patterns I would like to follow always. In the JavaScript, function can be used as : function method constructors classes Module As a first example, will start with the Module. Module Pattern In this pattern, you can define number of privilege method which have access to the secret information such as private variables and methods . For example, setFirstName method can set the name local variable as well as the function getUserFullName has access to the private method getFullName as shown int he following code: var person = ( function (firstName, lastName, age) { //properties var a = age; var fname = firstName; var lname = lastName; //common functions function getFullName (f,l) { return fname+ " " +lname; } return { //open to outside