Configure Pyspark

 Pyspark

Pre-Req:

• Install Python 3.9
• Find the location of python ($which python) and Keep it handy
• pip3 install ipython #optional
• pip3 install pyspark
• Download apache spark zip > Unzip to a Path

Steps:

• Create file ~/.bash_profile

• Add Below Contents

    export PYSPARK_PYTHON=python3
    export PYTHONPATH="Python location"
    export PYSPARK_PATH="../spark-x.x.x-bin-hadoop3/bin/pyspark"
    alias pyspark=$PYSPARK_PATH
    export PATH=$PATH:$PYSPARK_PATH
  
    #optional
    alias ipython='python3 -m IPython' 
  

  Example :

    export PYSPARK_PYTHON=python3
    export PYTHONPATH="/Users/deepakjayaprakash/Library/Python/3.9/bin/python3"
    export PYSPARK_PATH="/Users/deepakjayaprakash/Downloads/spark-3.3.2-bin-hadoop3/bin/pyspark"
    alias pyspark=$PYSPARK_PATH
    export PATH=$PATH:$PYSPARK_PATH
  
    #optional
    alias ipython='python3 -m IPython' 
  

• Save File

• source ~\.bash_profile

view raw pyspark.md hosted with ❤ by GitHub