Ojitha Kumanayaka

In the Scala functions are first class. For example function literal can be possible to assign to object as follows: val f = (x:Int ) => x + x //f: Int => Int = <function1> f(2) // res0: Int = 4 The different between function literal and function value is that literal exists in source code and the value exists in the runtime. Compiler do the target typing if you don't specify the type for the x in the above source. To more precise, you can use the _ as a placeholder instead of x bound variable in the above code because x is a parameter to the function. For simplicity : //collection supports val l1 = for (i <- 1 to 10) yield i //l1: scala.collection.immutable.IndexedSeq[Int] = Vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) l1.filter(_ > 3 ) //res1: scala.collection.immutable.IndexedSeq[Int] = Vector(4, 5, 6, 7, 8, 9, 10) In the above source, when _ used that is called partially applied function because don't supply all of the arguments needed by the expre

Here the first example using decorator in python 2.7 import unittest import random import mock def myrandom(p): return random.random() > p class Test(unittest.TestCase): @mock.patch('random.random') def test_myrandom(self, mock_random): mock_random.return_value = 0.1 val = myrandom(0.0) assert val > 0 assert mock_random.call_count == 1 if __name__ == '__main__': unittest.main() Here the example for assert_callled_with() function import unittest import mock import example class Test(unittest.TestCase): @mock.patch('example.hello') def test1(self,mock_hello): x = 'Oj' example.hello(x) # Uses patched example.func mock_hello.assert_called_with(x) if __name__ == '__main__': unittest.main() Above test can be ran using context manager: import unittest import mock import example class Test(unittest.TestCase): def test1(self): x = &q

Coming from the Java background, my main problem with JavaScript is encapsulation: global space is not good. I love OO because object is the best place to encapsulate the data. Here are the patterns I would like to follow always. In the JavaScript, function can be used as : function method constructors classes Module As a first example, will start with the Module. Module Pattern In this pattern, you can define number of privilege method which have access to the secret information such as private variables and methods . For example, setFirstName method can set the name local variable as well as the function getUserFullName has access to the private method getFullName as shown int he following code: var person = (function (firstName, lastName, age){ //properties var a = age; var fname = firstName; var lname = lastName; //common functions function getFullName(f,l){ return fname+" "+lname; } return { //open to outside getFirstName:

As shown in the following example, you can use the @wrap (which return another wrapper) to to transfer JSON to object in the python. This blog written conjunction with the Python Algorithm to flattening JSON 1 . from functools import wraps def json_to_object(func): @wraps(func) def wrapper(self, d): for name, value in d.iteritems(): setattr(self, name,value) return func(self, d) return wrapper class Person(object): @json_to_object def __init__(self, d): pass a = Person({'firstName':'Tom', 'lastName':'Hanks', 'age':50}) a.firstName a.lastName a.age More advanced version from functools import wraps def json_to_object(func): @wraps(func) def wrapper(self, d): for name, value in d.iteritems(): if type(value) == dict: print(value) setattr(self,name,Person.fromJson(value)) else: setattr(self, nam

Algorithm to flatten the deep JSON structure: def flat(root, **args): d = {} for k, v in args.iteritems(): if type(v) == dict: d.update(flat((k if root == None else '{}.{}'.format(root,k)),**v)) elif type(v) == list: for idx, item in enumerate(v): d.update(flat('{}.{}'.format(k if root == None else '{}.{}'.format(root,k),idx), **item)) else: if root == None: d['{}'.format(k)] = v else: d['{}.{}'.format(root,k)] = v #print ('key: {}, val: {}'.format(k,v)) return d for example, if you flatten the following JSON structure: tt ={'name':{'firstName':'Tom', 'lastName':'Hanks'}, 'orderlineitems':[{'rice':{'qty':2,'price':10}},{'bread':{'qty':1,'price':2}}], 'age':20, 'location'

Scala matching functionality is compelling. This feature is very important for big data programming. This blog tries to consolidate matching example as much as possible. Basic use of Match Boolean operator Use in the exceptions With Constants With Regex With Vector With Map As an Iterator Case with guard conditions With Option With Tuples With List With Seq Wildcards Case classes Reference Basic use of Match The simplest matching var i = 2 // i: Int = 2 // i match { // res0: String = two case 1 => "one" // case 2 => "two" // } // You can match anything as follows: def getClassAsString(x: Any):String = x match { // getClassAsString: (x: Any)String case s: String => s + " is a string" // case i: Int => i +" is Int"

You can simply create a Administrators group as follows in the cli aws iam create-group --group-name Administrators aws iam list-groups aws iam list-attached-group-policies --group-name Administrators You can login using account alias: aws iam create-account-alias --account-alias ojitha-yahoo ws iam list-account-aliases cheat kinesis stream aws kinesis create-stream --stream-name AccessLogStream --shard-count 1 create EMR cluster aws emr create-cluster --name demo --instance-type m3.xlarge --instance-count 2 --release-label emr-4.6.0 --ec2-attributes KeyName=ojitha-yahoo --use-default-roles --applications Name=Hive Name=Spark create a Redshift cluster: aws redshift create-cluster --cluster-identifier demo --db-name demo --node-type dc1.large --cluster-type single-node --master-username master --master-user-password Redshift123 --publicly-accessible --port 8192 download appender wget http://emr-kinesis.s3.amazonaws.com/publisher/kinesis-log4j-appender-1.0.0.jar

To install the SBT in the Mac environment brew install sbt@1 Say you have a following run.scala file in your directory, package world object Hello extends App { print("Hello World") } now start the sbt and run the compile command first, after that runMain world.Hello Now you can have a build.sbt file as follows name := "Greetings" organization := "ojitha.blogspot.com.au" version := "1.0.0" To show resolvers show resolvers To show dependencies show libraryDependencies To publish locally publishLocal You can add the dependencies as follows to the build.sbt name := "Greetings" organization := "ojitha.blogspot.com.au" version := "1.0.0-SNAPSHOT" resolvers += "Sonatype releases repo" at "https://oss.sonatype.org/content/repositories/Releases" { val liftVersion = "3.1.0" libraryDependencies ++= List( "net.liftweb" %% "lift-util&q

The case class is one of the most important for data wrangle . Here some use of the case class using Scala comprehensions such as for comprehension. Here the implementation:  case class Member (name: String, age: Int) // defined class Member case class Family (members: Seq[Member]) // defined class Family //

I am using IntelliJ more than 5 years now and eventually became a fan of that. I used both Ultimate edition and community edition. The community edition Scala plugin is well established now compared to 2 years back. However, there are a number of important productivity topics to discuss here. Copy Scala worksheet For example, I want to copy the Scala worksheet to the text editor sublime. Here the steps you need to follow 1. in the IntelliJ, enable column selection mode by CMD+SHIFT+8 2. Select the text you want to copy (for example first the left side of the split screen where Scala source is available) 3. Paste the copied text sublime new file CMD + N 4. In the Sublime now press the CMD + OPTION + F to search and replace. 5. Enable the regex, and search the $ (end of the line) and replace <space>//<space> and click the replace all. 6. Now back to the IntelliJ Idea and selecting the other side ( right side of the split screen where the results are) after CMD+SH

To install the docker machine in the AWS: #for the default docker docker-machine create --driver amazonec2 aws01 #for the custom centos docker-machine create --driver amazonec2 --amazonc2-region ap-southeast-2 --amazonec2-ami ami-0c2aba6c -amazonec2-ssh-user ojitha aws-demo If you run the above default (in the N. virginia region), here the way to set the env ~ docker-machine env aws01 #output is : export DOCKER_TLS_VERIFY="1" export DOCKER_HOST="tcp://54.173.183.175:2376" export DOCKER_CERT_PATH="/Users/ojitha/.docker/machine/machines/aws01" export DOCKER_MACHINE_NAME="aws01" # Run this command to configure your shell: # eval $(docker-machine env aws01) #for the custom #eval $(docker-machine env aws-demo) For example, to run a simplest docker container ~ docker run --name hello-A dockerinaction/hello_world Unable to find image 'dockerinaction/hello_world:latest' locally latest: Pulling from dockerinaction/hello_world a3ed95ca