Atom as Spark Editor

Recently I started to reach to integrate Atom editor with Spark pyspark. In addition to the Atom, I found how to integrate pyspark with IntelliJ Idea which I suppose to discuss later. The nice thin about Atom is hydrogen plugin which you can use for inline evaluation with python. Here the steps
  1. Install Spark
  2. Install Atom
  3. Install hydrogen plugin to atom
  4. most important to set the PYTHONPATH as follows
    export PYTHONPATH=/<SPARK_HOME>/python:/<SPARK_HOME>/python/lib/py4j-0.9-src.zip
  5. Now run the following code to verify.
Here the testing code.
from pyspark import SparkContext
from pyspark import SQLContext

sc = SparkContext()
sqlContext = SQLContext(sc)
df = sqlContext.createDataFrame([("Ojitha", "Kumanayaka"),("Mark", "Anthony")],("first_name", "last_name"))
df.show()
 
 

Comments

Popular posts from this blog

How To: GitHub projects in Spring Tool Suite

Spring 3 Part 7: Spring with Databases

Parse the namespace based XML using Python