Atom as Spark Editor
Recently I started to reach to integrate Atom editor with Spark pyspark. In addition to the Atom, I found how to integrate pyspark with IntelliJ Idea which I suppose to discuss later. The nice thin about Atom is hydrogen plugin which you can use for inline evaluation with python. Here the steps Install Spark Install Atom Install hydrogen plugin to atom most important to set the PYTHONPATH as follows export PYTHONPATH = /<SPARK_HOME>/python :/<SPARK_HOME>/python/lib/py4j- 0 . 9 -src.zip Now run the following code to verify. Here the testing code. from pyspark import SparkContext from pyspark import SQLContext sc = SparkContext() sqlContext = SQLContext(sc) df = sqlContext.createDataFrame([( "Ojitha" , "Kumanayaka" ),( "Mark" , "Anthony" )],( "first_name" , "last_name" )) df.show()